Guidance for Selecting or Curating High-Quality Instruction…

Guidance for Selecting or Curating High-Quality Instructional Resources

Learning List, Inc.

January 2023

Guidance for Selecting or Curating High-Quality Instructional Resources

Contents

Evaluating Core and Supplemental Materials...................................................5 Communicate the Need ...............................................................................5 Define the Non-Negotiables.........................................................................6 Establish a Timeline .....................................................................................6 Document the Logistics ................................................................................7 Identify Selection Committee Makeup and Requirements............................7 Distribute a Needs Assessment ....................................................................8 Narrow the List of Materials ........................................................................9 Develop or Customize a Rubric...................................................................10 Provide Training for the Selection Committee ............................................13 Conduct the Reviews .................................................................................14 Reconvene the Selection Committee..........................................................14 Plan the Implementation ...........................................................................16 Curating High-Quality Resources....................................................................16 Document the Logistics ..............................................................................18 Identify Selection Committee Makeup .......................................................18 Develop a Rubric........................................................................................19 Provide Training for the Selection Committee ............................................19 Conduct the Reviews .................................................................................19 Create and Maintain an Online Library of High-Quality Curated Resources.20 Conclusion ..................................................................................................... 20 Bibliography ..................................................................................................20

The state’s math standards recently changed, and your district purchases new instructional materials for all math teachers. Teachers use the materials for a few months, but when students struggle to master the content, they grow skeptical about the material’s quality and suitability for their students. Increasingly, teachers turn to lesson resources they have developed themselves, borrowed from colleagues, or downloaded from the internet . The district’s materials gather dust, and district leaders are left with curriculum chaos. Sound familiar?

In a 2022 study by the RAND Corporation, one out of five teachers surveyed reported not using their district- or school-purchased materials. The most common reasons teachers gave for not using their district materials were (1) the material did not meet their studen ts’ needs , and (2) they did not have time to learn how to use or incorporate the material into

their lessons. Additionally, between one-third and one-half of the teachers surveyed indicated that they need more or better curriculum materials. (Prado Tuma et al. 2022, 9) The study recommends that school systems could improve the alignment between teachers’ and students’ needs and the instructional materials they purchase by involving teachers in the selection of new instructional materials and by clearly communicating how teacher input will be used in the selection process. Moreover, the report suggests that state education departments, districts and professional development providers could provide guidance to help teachers become better evaluators of instructional materials. (Prado Tuma et al. 2022, 13) Over the last decade, Learning List has reviewed more than 3,300 PreK-12 core and supplemental instructional materials by approximately 200 different publishers, as well as teacher-created resources. We have also assisted

hundreds of districts, both formally and informally, with their selection processes.

In this paper, we share effective strategies for reviewing and selecting instructional resources as well as lessons learned along the way to help you ensure the quality and coherence of the instructional resources being used on campuses across your district.

Evaluating Core and Supplemental Materials

Generally, core materials are selected at the district level, and supplemental materials are selected by campuses. During the last decade, districts have started implementing more formal processes for selecting core materials, while campus leaders typically rely on a variety of processes, including word of mouth, when selecting supplemental materials. (Wang, et al., 2019, 14). To ensure that teachers and students are supported with high-quality materials, a structured process for reviewing and selecting all instructional materials is needed. The following process considers the voices of educators, parents, and the community, facilitates consistent reviews, and produces data to support the selection decisions. Since supplemental materials are not as robust as core materials, the process may be pared down for the selection of supplemental materials as explained below.

Communicate the Need

The adoption of core materials is typically, though not always, driven by a change in state standards. In contrast, supplemental materials are adopted for many reasons. To ensure that district and/or campus leaders, teachers, and the community understand the need for new materials, it is important to clearly

communicate the reason(s) for and goal(s) of an adoption before beginning the review process.

Define the Non-Negotiables

District or campus leaders should define the non-negotiables for the adoption. Non- negotiables are the basic requirements for the new materials. Non-negotiables may include: the budget/price limit, the minimum alignment percentage, instructional model, required supports for students and resources for teachers, and the format of the material. Your district/campus’ experience with remote learning may suggest additional non- negotiables.

Practice Tip

Useful resources for developing the list of non-negotiables:

• Board policy

• Administrative procedures

• District/campus budget

• District curriculum / campus instructional goals and

instructional model

• Student achievement data

• District/campus LMS and other

technology requirements

The list of non-negotiables can be used to help narrow the number of materials the selection committee will review and inform the development of the rubric as explained further below.

Establish a Timeline

In order to ensure that the new material(s) will arrive in the classrooms on time and that teachers have time to learn how to use them, it is critical to establish a timeline for the review and adoption process. Working back from the board meeting where the materials will be adopted or from the deadline for getting materials into the classrooms, create a realistic timeline for the entire process. Include time for ordering any required technology components and providing product-specific professional development for all teachers who will be using the new material(s).

Document the Logistics

A well-planned and thorough adoption process will more likely lead to a successful outcome. At a minimum, think through and document answers to the following:

• When, were and how will the selection committee review the materials?

• Will a rubric be used for the adoption? If so, how will the rubric be developed? • Will the selection committee review the materials simultaneously in- person or virtually and on their own time? • If product samples are online, how will login credentials be disseminated to the reviewers? • Will publishers be permitted to present to the selection committee? If so, when and where?

• What are the voting protocols for the selection committee’s recommendations?

• How will the new materials be distributed?

• Who will be involved in planning and supporting the implementation of the new materials? • Is there a budget allocation for technology, professional learning, and ongoing support?

Identify Selection Committee Makeup and Requirements

State law and/or school board policy may dictate the positions that must be represented on the selection committee. At a minimum, the committee should include both experienced and new teachers representing the content area(s) and grade bands covered by the adoption. Teachers who provide support services for students should also be represented on the committee. If the new

material may have digital components, a technology integration specialist should be included, as well.

When adopting supplemental materials, the type of material being adopted and the number of teachers who will be using it should dictate the makeup and size of the selection committee. To secure buy-in for the new material selected, the committee should consist of a representative sample of the teachers who will be using it and staff who will be supporting its implementation. Including both experienced and inexperienced teachers will help ensure that the material selected will support teachers with a wide range of experience.

In addition to the makeup of the committee, it is important to document attendance requirements and how selection committee members will be replaced if they are unable to meet those requirements.

Distribute a Needs Assessment

Creating opportunities for all stakeholders to participate in the adoption processes fosters support for and mitigates opposition to the materials. Distributing a needs assessment is an effective way to accomplish that. A needs assessment is a survey for gathering

feedback about the features stakeholders believe are necessary (“must have”) and desirable (“nice -to- have”) in the new material(s). Distributing a needs assessment gives all stakeholders, not just the selection committee, a voice in the process.

Practice Tip

When developing the needs assessment survey, we have observed that it is helpful

to first ask stakeholders about the features they value in the current

materials before asking about features they believe the new materials must have.

For the adoption of core materials, the needs assessment should be distributed to • All teachers who will be using the newly adopted material, including teachers who provide support services

• All staff who will be supporting its implementation, such as instructional coaches and instructional technology staff • Parents In some states, districts are required to survey the community before a curriculum adoption, as well. Supplemental materials impact fewer students and teachers than core materials. When adopting supplemental materials, distributing the needs assessment to the teachers who will be using the new material(s) and to staff who will be supporting their implementation should suffice. Aggregate the feedback gathered from the needs assessment to produce a list of features that the stakeholders believe the new material must have . These features should be reflected in the district’s rubric (s) for the adoption, as explained below.

Narrow the List of Materials

In order to ensure that the selection committee members have time and capacity to review the materials being considered for adoption thoroughly, it is

important to narrow the number of materials the selection committee will review.

Practice Tip

Optimally, the selection committee should review between three and five materials.

First, put together a list of the materials available for the content area(s) and grade levels or courses covered by the adoption. If your state has a review process, the list of materials submitted for state review is a

We have observed that reviewing more than five materials overburdens the selection committee members and results in lower-quality reviews and/or the submission of fewer completed rubrics.

good starting point. Another resource is The Learning List, a free directory of thousands of K-12 materials organized by grade band, subject and course.

Then use the list of non-negotiables to narrow the number of materials the selection committee will review with a rubric. If necessary, use the needs assessment results to eliminate additional materials that obviously will not meet the district’s/campus’ needs. Third-party reviews from a reliable source can facilitate the elimination process. This process may be done by district or campus content area leaders or by the selection committee.

Develop or Customize a Rubric

Using a rubric to review materials ensures greater transparency, objectivity, and consistency in the review process. A rubric review also provides data to support the district or campus’ selection decisions. Some districts or campuses prefer to develop their own rubric; others choose to customize an existing third-party rubric. Either approach works. When it comes to rubrics, more is not necessarily better. Keeping the rubric to a manageable length can be one of the biggest challenges. We have observed, that in order for selection committee members to use a rubric with fidelity, rubrics for selecting core materials should include no more than ten (10) criteria and up to five (5) sub-criteria for each criterion. Thus, the rubric should include the features in instructional materials that your district values most .

Supplemental materials may not address all of the topics in the state standards, provide support for all students, or include robust teacher resources. A scaled-down version of the district’s rubric for the content area, or a checklist of must-have features would be sufficient for reviewing supplemental materials.

Practice Tip

At a minimum, a rubric/checklist for selecting supplemental materials should address • Standards alignment • Non-negotiables • Needs Assessment feedback • Instructional best practices for the content area

Criteria and Sub-Criteria/Guidance Statements

The rubric’s criteria and sub -criteria should reflect the district’s instructional goals, the non-negotiables and “must - have” features identified by the needs assessment feedback, the instructional topics addressed in the state standards, and features associated with high-quality instructional materials and research-based best practices for the content area.

Practice Tip

Resources for developing or customizing a

rubric:

• District curriculum /

campus instructional goals

• Needs assessment feedback

• Non-negotiables

• State standards

• Content-area best practices

• Research on high-quality materials

Criterion and sub-criterion are typically formatted as statements . Criteria and sub-criteria statements should be written using precise and unambiguous language and should describe observable and measurable features of a material (N.C. State University, 2022). Each criterion and sub-criterion statement should be distinct from the others. Terms like consistently , throughout , and repeatedly may be used to describe features that should be present throughout the material. It is not imperative to include sub-criteria or guidance statements for each criterion. If used, these statements should clarify or illustrate the criteria. For that reason, sub-criteria statements often include illustrative language, such as includes or does not include . Judgmental language such as poor , well , exceptional, or excellent should not be included in the criteria or sub-criteria statements; rather, judgments sh ould be addressed in the rubric’s rating scale.

Rating Scale

A rubric must include at least one type of rating scale. A rating scale allows users to express their level of agreement with each criterion and possibly each sub-criterion statement. There are many types of rating scales, including

• Graphic • Numerical • Descriptive • Comparative

The same or a different rating scale can be used for each criterion. If the rubric includes sub-criteria, you will need to decide whether respondents will be asked to rate the materials on each sub-criterion as well as on each criterion, or whether the sub-criteria are provided to help respondents determine the rating for each criterion. To make the rubric easy to use, the type of rating scale and wording of the rating scale’s response s must be consistent with the wording of the criteria and sub-criteria statements. For example, if a rubric used the rating scale above, the criteria and sub-criteria statements should be framed in terms of “ Rate how likely the material is to ….” If the rubric scaled responses ranged from “Not a t All” to “Very Well,” the rubric criteria should be framed in terms of “Rate how well…” The rating scale response choices should be easy to understand and distinguish, and the rating scale should include a sufficient number of response choices to allow selection committee members to communicate their level of agreement with the criteria and/or sub-criteria statements clearly. We have observed that having between three and five response choices in a rating scale works best; too many choices (e.g., 1-10) makes it difficult to discern a meaningful difference in the materials’ aggregate scores. The type and format of the rating scale also affects how the response data can be aggregated and disaggregated. Using a rating scale that assigns a numeric value for each response makes it easy to tally a score for each material from the aggregate response data.

In addition to the rating scale, a rubric can include text boxes for teacher notes and/or supporting evidence. Asking reviewers to provide text-based examples to support each criteria rating helps ensure that the materials are reviewed thoroughly and provides text-based evidence that can be used to build support for the district’s adoption decisions. Once a draft of the rubric/checklist is complete, it would be prudent to run it by a few of the content-area teachers to make sure that it is a manageable length and easy to understand.

Provide Training for the Selection Committee

Before or during the first selection committee meeting, committee members should be informed about the logistics of the review process, including the beginning and end dates of the review period, the titles that will be reviewed, and how to submit completed rubrics. The rules and policies of the review process must also be clearly explained. Committee members must also receive training on how to use the rubric in order to ensure that sub-criteria are interpreted and applied consistently. Optimally, before the reviews begin, district curriculum staff would create an evidence guide with text-based examples of the difference performance levels for each criterion and key sub-criteria. If that is not possible, the selection committee should discuss performance- level examples before beginning the reviews. Practice Tip

Given the importance of selecting standards-aligned materials, consider having the district curriculum team or lead teachers for the content area review the alignment of the materials the selection committee will review before the committee members begin their reviews.

For example, one of the criteria in any instructional materials rubric should be alignment to state standards. Selection committee members should receive training on or at least discuss what alignment means

and how to evaluate a material’s alignment to standards for purposes of the review process.

If available, links to reliable third-party reviews of the materials being considered should be provided to facilitate committee members’ completion of the rubric for each material.

Conduct the Reviews

Many districts implement a virtual review process, only bringing committee members together to discuss the resulting data. While it may be necessary to conduct the reviews virtually, we have observed that having committee members in the same room while they are reviewing the materials independently has important benefits. Spontaneous conversations occur that often prompt committee members to explore the materials more deeply and ultimately facilitate consensus building around which materials met the distric t’s needs best. When that happens, the selection committee members become strong advocates for the materials selected. If reviews are conducted virtually, we have observed that it is important for the person managing the selection process to monitor comm ittee members’ progress carefully and send reminders regularly to keep the process on track.

Reconvene the Selection Committee

Once completed rubrics have been submitted, the person managing the selection process should analyze the committee’s response s to determine for each grade level covered by the adoption which material scored (1) highest overall, and (2) highest on each criterion. That data analysis should be documented in an easy-to-read format and provided to the selection committee members.

Research suggests that to improve teacher use of district and campus instructional resources, it is important to clearly communicate how the

teachers’ participation in the process influenced the adoption decisions. (Prado Tuma et al. 2022, 13) Thus, once the results have been aggregated, it is important to reconvene the selection committee to discuss the results. Most districts want the committee members to reach a consensus about the materials to recommend for adoption. However, it is important for t he district’s voting procedures to address how a material will be selected if consensus is not reached.

Recommend Materials for Adoption

Generally, the school board has to formally adopt the district’s core materials. If that is the case in your district, it is important to inform the board about how and why the materials recommended for adoption were selected. Provide the type and amount of information board members need to feel secure in voting to adopt the recommended materials and to advocate on the district/campus’ behalf if the adoption decisions are challenged. For example, it would be prudent to explain

Practice Tip

Consider addressing the following topics in the board presentation: • Why the adoption was necessary • To whom the needs assessment was distributed • How the rubric addressed the needs assessment feedback • The selection committee members’ positions • The titles of the materials reviewed and how those titles were selected • Rubric data and any amendments made during the selection committee’s deliberations to support each of the committee’s recommendations

how teachers, parents, and community were afforded an opportunity to participate in the process and who participated on the selection committee(s). Also, provide the data that support s the selection committee’s recommendations. The rationale for adopting the new materials should also be documented clearly and communicated to the teachers who will be using the new materials. To build teacher buy-in, the communication should include how teacher input was integrated into the selection process. (Prado Tuma et al. 2022, 13) It may

also be helpful to include some of the text-based examples reviewers provided in their rubrics.

It would be prudent to maintain the rubric data and documentation for several years in case the district’s use of the selected material is challenged.

Plan the Implementation

Once the materials have been ordered, the person managing the selection process, the district curriculum department, and the district technology department should start planning for a successful implementation. For example, the material’s delivery date or beginning of the district’s subscription should be communicated to the relevant stakeholders. All necessary technology components should be ordered in a timely fashion, and professional learning must be scheduled to support teachers in using the material effectively. A common reason teachers gave for not using their district- or campus- provided curriculum materials was “a lack o f time to learn how to use the material or to incorporate new materials during class.” (Prado Tuma et al. 2022, 9). No district or campus wants to spend time or money selecting materials that go unused. Implementing a transparent, data-driven review process like the one described above will more likely result in the purchase of standards-aligned, high-quality materials that will meet teachers ’ and students ’ needs .

Curating High-Quality Resources

One of the biggest curriculum challenges districts currently face is teacher entrepreneurialism as it relates to finding and using resources. A Harvard University study of teachers across six states noted that “only seven percent (7%) of teachers studied used the ir textbooks exclusively” (Blazar et al. 2019, 15). A study by the RAND Corporation similarly revealed that teachers are more likely to use a combination of comprehensive and supplemental in their lessons than rely

on a single material. (Prado Tuma et al. 2022, 7). The challenge for district and campus leaders is validating the quality of these resources.

The Thomas B. Fordham Institute evaluated the quality of the most frequently downloaded high school English language arts resources from three popular sources: Teacher Pay Teacher, ReadWriteThink, and Share My Lesson (Polikof et al., 2021). The following criteria were used to evaluate the curated resources:

Common Reasons Teachers Search for Instructional Resources

• To fill gaps in their existing materials • To engage students • To provide scaffolds or supports to help students master grade-level content • To review content from prior grade levels that students had not mastered • To differentiate instruction for students enrolled in special education, for English learners, or for advanced learners • To provide culturally relevant instruction.

• Alignment to standards • Rigor and text complexity

• Instructional and assessment quality • Adaptions for students and supports for teachers • Usability

(Prado Tuma et al ., 2022, 10)

Validating curriculum leaders’ concerns, the Fordham study concluded that 64% of widely downloaded materials should not be used or are probably not worth using. Less than 10% of the materials on each of the three sites were rated exceptional . The Fordham study showed that the popularity of teacher-curated resources is not an indicator of high quality. So how can districts ensure that teachers are using high-quality lesson resources? Instead of trying to fight this trend, embrace it. We offer the following seven-step process for reviewing and selecting high-quality curated resources, which is consistent with the process for reviewing instructional materials described above.

Create Submission Process

Begin by surveying teachers to determine the type of materials they are currently using, what purpose each resource fulfills, and what types of resources they would want to use. Encourage them to participate in the survey by explaining

that your goal is to create a district-wide library of vetted, high-quality teacher- curated resources that is easily accessible to all district educators.

Initially, have teachers attach their curated resources as a PDF to their survey responses. Longer term, consider creating a repository where teachers can submit materials for review on an ongoing basis.

Document the Logistics

To build a robust library of the best curated resources across the district, think through and document answers to the following:

• What types of resources that will be reviewed? • How will resources be submitted?

• How and how often will the submitted resources be reviewed? • How will the reviewers access the resources to be reviewed? • Will a rubric be used for reviewing the materials? If so, how will the rubric be developed? • How will the vetted resources be accessible? Who will have access? • How often will newly vetted resources be posted? • How and how often will the vetted resources be re-reviewed for continued accuracy and relevance? In order for this initiative to be inclusive and successful, the submissions process, review process, review schedule and schedule for posting new resources to the district library must be clearly communicated to all teachers on a regular basis.

Identify Selection Committee Makeup

A district or campus curriculum leader should be designated to manage the review and selection process. District curriculum staff or lead teachers for each content area may be the most suitable reviewers. Alternatively, consider having teachers evaluate the curated resources for their content area and grade band during a certain designated Professional Learning Committee (PLC) meeting(s).

Develop a Rubric

The five criteria used in the Fordham study provide a good foundation for a rubric. The rubric should also address the district’s instructional priorities for the content area and grade band. To foster buy-in, involve the selection committee members in the development of the rubric. Make sure that the rubric is not too long, because many resources will need to be evaluated. Once the rubric is developed, determine and clearly document how a resource will be deemed “ high-quality ” for inclusion in the library. For example, must reviewers reach a consensus “high quality” rating on all rubric criteria for a material to be included in the library? Must a submitted resource receive a minimum total or average rubric score in order to be included in the library?

Provide Training for the Selection Committee

Just as in the instructional materials review process, all reviewers must receive training on how to use the rubric to review different types of curated resources and how to submit their completed rubric(s). To facilitate consistent reviews over time, it may be beneficial to develop guidance with text-based examples of the difference performance levels on the rubric for each type of curated resource.

Conduct the Reviews

The number of resources submitted and make-up of the selection committee will likely determine whether reviews should be conducted in person or virtually. For example, if teachers will be reviewing the submitted resources during a PLC meeting, the reviews will likely take place in person. If that is the case, consider whether reviewers will be allowed to discuss their rubric feedback for each resource, which may be a valuable professional learning activity, or whether reviews should be conducted individually and completed rubrics submitted confidentially to avoid hurt feelings.

Once the rubrics have been submitted, the person in charge of the process should aggregate the rubrics responses and determine the list of materials that received the requisite rating or score for inclusion in the district library.

Create and Maintain an Online Library of High-Quality Curated Resources

Develop a way for teachers across the district to access the vetted resources easily. This may be as simple as creating a Google spreadsheet with links to each curated resource. Alternatively, the resources could be cataloged and linked on the district’s w ebpage for teacher resources.

The resources in the library should be re-reviewed regularly for continued relevance.

Conclusion

The internet and the widespread adoption of the Common Core State Standards have encouraged the proliferation of both commercially-produced materials and teacher-created resources. To support high-quality instruction, school systems must strive to ensure that district- and campus-purchased materials are standards- aligned, high quality and meet their teachers’ and students’ needs . However, teachers also regularly incorporate curated resources into their lessons, and quality of those resources is suspect. The confluence of high-stakes accountability, Covid-related learning gaps, and political scrutiny of district curriculum decisions necessitates a strategy to bring order to the chaos. Establishing a structured, transparent, and consistent process for evaluating the quality of both instructional materials and curated resources will support instruction aligned with the district’s instructional goals and students’ needs .

Bibliography

Blazar, D., Heller, B., Kane, T., Polikoff, M., Staiger, D., Carrell, S., Goldhaber, D., Harris, D., Hitch, R., Holden, K., and Kurlaender, M. (2019) Learning by the Book: Comparing math achievement growth by textbook in six Common Core states .

Center for Education Policy Research, Harvard University. https://cepr.harvard.edu/curriculum

N.C. State University. (2022) Rubric Best Practices, Examples, and Templates. https://teaching-resources.delta.ncsu.edu/rubric_best-practices-examples- templates/ Polikoff, M. and Dean, J. (2019) The Supplemental Curriculum Bazaar: Is What’s Online Any Good? Thomas B. Fordham Institute. https://fordhaminstitute.org/national/research/supplemental-curriculum- bazaar. Wang, E., Prado Tuma, A., Lawrence, R., Kaufman, J., Woo, A., and Henry, D. (2021) School Leaders' Role in Selecting and Supporting Teachers' Use of Instructional Materials: An Interview Study. RAND Corporation. https://www.rand.org/pubs/research_reports/RRA134-9.html.

About Learning List

Learning List offers a suite of curriculum support services, including a subscription- based instructional materials review service. Subscribers get access to all published reviews and the ability to submit additional materials for review a t no additional cost . Over the last decade, Learning List has reviewed thousands of PreK-12 instructional materials by almost 200 publishers. Each material is reviewed for alignment to state or national standards, instructional quality, and technology compatibility. Our other services include a selection facilitation service, customized curriculum alignment services, and online professional development courses.

Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Page 9 Page 10 Page 11 Page 12 Page 13 Page 14 Page 15 Page 16 Page 17 Page 18 Page 19 Page 20