Validity
Validity is the utility, interpretability, generalizability, and accuracy of a given measure
Validity, as a concept, has been debated greatly in the methodological and philosophical literature. In its most basic applicable form in applied psychometrics, validity is the evidence that shows that scores from a survey can be interpreted and applied in an accurate fashion within a population of interest. The process of establishing validity is an ongoing and dynamic process that never ends. Survey instruments need to be constantly revalidated and recalibrated in shifting and changing populations.
Validity is heavily grounded in empirical reasoning. The construct must be derived from a theoretical or conceptual framework within a body of literature. The evidence of validity is obtained through a tedious process of mastering the content areas related to the construct, constructing a reliable survey instrument, developing a nomological network of conceptually similar instruments, testing the instrumentation within the nomological framework, and then establishing the consequences and utility of the survey scores.
Validity is heavily grounded in empirical reasoning. The construct must be derived from a theoretical or conceptual framework within a body of literature. The evidence of validity is obtained through a tedious process of mastering the content areas related to the construct, constructing a reliable survey instrument, developing a nomological network of conceptually similar instruments, testing the instrumentation within the nomological framework, and then establishing the consequences and utility of the survey scores.
Types of validity evidence
There are five types of validity evidence:
1. Test content
With content validity, the reasoning is that an instrument can only be interpreted if it effectively reflects the construct of interest. There are two specific threats to content validity: Construct-irrelevant content and construct under-representation. Construct-irrelevant content is spurious or redundant items that are unrelated to the construct of interest. These items can introduce statistical "noise" into the analysis and detract from the precision and accuracy of survey scores. Construct under-representation means that specific content areas that are relevant towards measuring for the construct are not accounted for in the survey.
2. Internal structure
The internal structure of the survey instrument should reflect the underlying structure of the construct of interest. The construct specification begins the process of formulating this internal structure and the "factors" yielded from exploratory and confirmatory factor analyses should reflect the construct as it exists within the literature and natural environment.
3. Processes used to respond to items
The psychological, cognitive, emotional, and affective processes that respondents use to answer questions is also pertinent when assessing validity. Test instructions, the reading level of the language used in the survey, and the survey items must be written in a fashion that respondents can understand and logically respond to in the survey.
4. Association with other scores and variables
This form of validity is focused on assessing the associations between the survey instrument and other theoretically, conceptually, and empirically similar constructs. Significant associations between survey instruments and other existing instruments provides evidence of validity.
5. Consequences of test use and interpretation
This validity evidence focuses on the intended and unintended benefits or consequences of taking a survey, interpreting the results, and utilizing the results to make informed decisions.
1. Test content
With content validity, the reasoning is that an instrument can only be interpreted if it effectively reflects the construct of interest. There are two specific threats to content validity: Construct-irrelevant content and construct under-representation. Construct-irrelevant content is spurious or redundant items that are unrelated to the construct of interest. These items can introduce statistical "noise" into the analysis and detract from the precision and accuracy of survey scores. Construct under-representation means that specific content areas that are relevant towards measuring for the construct are not accounted for in the survey.
2. Internal structure
The internal structure of the survey instrument should reflect the underlying structure of the construct of interest. The construct specification begins the process of formulating this internal structure and the "factors" yielded from exploratory and confirmatory factor analyses should reflect the construct as it exists within the literature and natural environment.
3. Processes used to respond to items
The psychological, cognitive, emotional, and affective processes that respondents use to answer questions is also pertinent when assessing validity. Test instructions, the reading level of the language used in the survey, and the survey items must be written in a fashion that respondents can understand and logically respond to in the survey.
4. Association with other scores and variables
This form of validity is focused on assessing the associations between the survey instrument and other theoretically, conceptually, and empirically similar constructs. Significant associations between survey instruments and other existing instruments provides evidence of validity.
5. Consequences of test use and interpretation
This validity evidence focuses on the intended and unintended benefits or consequences of taking a survey, interpreting the results, and utilizing the results to make informed decisions.
Research Engineer will primarily focus on the second and fourth types of validity evidence. Click below for the type of valid evidence that will answer the research question.
Types of validity
Construct validity is the aggregate form of validity evidence related to assessment.
Criterion validity is evidence that a survey instrument can predict for existing outcomes that occur at the same time or in the future.
Predictive validity provides evidence of how a survey instrument predicts for outcomes and events that will occur in the future.
Concurrent validity provides evidence of how a survey instrument predicts for another measure or construct at the same time.
Content validity relates to the representation of the content areas that exist in the body of literature.
Known-groups validity generates evidence related to the ability of a survey instrument to differentiate between existing independent groups.
Convergent validity shows evidence that a survey instrument positively correlates with other survey instruments measuring for theoretically or conceptually similar constructs.
Incremental validity generates high-level multivariate evidence that a survey instrument accounts for new and unique variance above and beyond what has been accounted for up until that present time in the body of literature.
Face validity is the weakest form of validity evidence showing that at face value, an instrument does what it is supposed to do.
Confirmatory factor analysis (CFA) validates the internal structure of the survey instrument yielded from a Principal Components Analysis.
Statistician For Hire
DO YOU NEED TO HIRE A STATISTICIAN?
Eric Heidel, Ph.D. will provide statistical consulting for your research study at $100/hour. Secure checkout is available with PayPal, Stripe, Venmo, and Zelle.
- Statistical Analysis
- Sample Size Calculations
- Diagnostic Testing and Epidemiological Calculations
- Psychometrics
