Guidance for Selecting or Curating High-Quality Instruction…

• Graphic • Numerical • Descriptive • Comparative

The same or a different rating scale can be used for each criterion. If the rubric includes sub-criteria, you will need to decide whether respondents will be asked to rate the materials on each sub-criterion as well as on each criterion, or whether the sub-criteria are provided to help respondents determine the rating for each criterion. To make the rubric easy to use, the type of rating scale and wording of the rating scale’s response s must be consistent with the wording of the criteria and sub-criteria statements. For example, if a rubric used the rating scale above, the criteria and sub-criteria statements should be framed in terms of “ Rate how likely the material is to ….” If the rubric scaled responses ranged from “Not a t All” to “Very Well,” the rubric criteria should be framed in terms of “Rate how well…” The rating scale response choices should be easy to understand and distinguish, and the rating scale should include a sufficient number of response choices to allow selection committee members to communicate their level of agreement with the criteria and/or sub-criteria statements clearly. We have observed that having between three and five response choices in a rating scale works best; too many choices (e.g., 1-10) makes it difficult to discern a meaningful difference in the materials’ aggregate scores. The type and format of the rating scale also affects how the response data can be aggregated and disaggregated. Using a rating scale that assigns a numeric value for each response makes it easy to tally a score for each material from the aggregate response data.