Chapter 8 Abstract

 

This chapter continues the explication that began in Chapter 7 of how to examine the trustworthiness of measurements; Chapter 7 focused on precision—are the measurements consistent over different occasions? In contrast, Chapter 8 explores the broader issue of validity. In fact, this chapter is longer and more complex than the previous chapter, due to the inherent complexities in addressing this issue. In this chapter, fairness is seen as a crucial aspect of validity, one for which the discussion has already begun in Section 3.4, while discussing the design process.

 

Thus, going beyond the focus on precision in Chapter 7, one might characterize the aim of this chapter as describing ways to develop evidence about how well the instrument accomplishes what it set out to accomplish. That is, in the jargon of the social sciences, whether there is evidence for the validity of the instrument’s usage. The focus in this chapter is on accumulating a specific set of arguments for an instrument structured according to the sources of validity evidence described in the “Standards” (AERA et al.. 2014), and each of these categories will be used, in turn, to support or argue against the planned usage of the instrument. 

 

The gathering of evidence of various types of validity, as described below, should not be seen as simply a one-time event. The purpose of investigating validity evidence in instrument development is to help the measurer make the instrument work in a way that is more consistent with the intent. As such, evidence that the instrument is not doing so is not a dead-end in this process. Instead, such evidence can be seen as an important step forward. Now that there is some evidence of a problem, the trick is to find a way to make that evidence useful in the next iteration of the process.