importance of validity and reliability in assessment

In order to be valid, a measurement must also and first be reliable. More specifically, it refers to the extent to which inferences made from an assessment tool are appropriate, meaningful, and useful (American Psychological Association and the National Council on Measurement in Education). Validity is the extent to which a measurement tool measures what it is supposed to.

Increase the number of questions on a multiple choice exam that address the same learning outcome.

Recalculate interrater reliability until consistency is achieved.

Conducting norming sessions to help raters use rubrics more consistently.

Interrater reliability = number of agreements/number of possible agreements.

Testing rubrics and calculating an interrater reliability coefficient.

Create or gather and refer to examples that exemplify differences in scoring criteria.

Revisit these often while scoring to ensure consistency. Develop well-defined scoring categories with clear differences in advance.

Providing guidelines for constructing assignments that will be used to measure SLO achievement.

Agreeing on how SLO achievement will be measured.

Issues with reliability can occur in assessment when multiple people are rating student work, even with a common rubric, or when different assignments across courses or course sections are used to assess program learning outcomes. Student learning throughout the program should be relatively stable and not depend on who conducts the assessment. It does not have to be right, just consistent.

Reliability is the extent to which a measurement tool gives consistent results.

Instead, be mindful of your assessment’s limitations, but go forward with implementing improvement plans. While you should try to take steps to improve the reliability and validity of your assessment, you should not become paralyzed in your ability to draw conclusions from your assessment results and continuously focus your efforts on redeveloping your assessment instruments rather than using the results to try and improve student learning.

Reliability and validity are important concepts in assessment, however, the demands for reliability and validity in SLO assessment are not usually as rigorous as in research.

Importance of validity and reliability in assessment