Tags

  • Published on

    Predictive validity is a powerful type of psychometric evidence

    Predictive Validity

    Correlations and regression are used to establish this kind of evidence

    Predictive validity evidence means that a survey instrument has the ability to predict some sort of occurrence in the future.  The most common application of predictive validity occurs in tests like the ACT, SAT, GRE, MCAT, LSAT, and GMAT. These tests are given before entering various phases of higher education to assess an individual's potential to graduate from either undergraduate or graduate school.  Interestingly enough, the correlation between these prevalent (and expensive) tests and graduation is only 0.3!  This means that 91% of what accounts for graduation is NOT associated with test scores on these instruments.  And we are talking a multi-BILLION dollar business...but, I digress.

    Predictive validity is calculated using simple correlation coefficients.  A correlation of 0.1 is considered weak evidence, a correlation of 0.3 denotes moderate evidence, and a correlation of 0.5 would make most social scientists jump for joy. Remember, in order to understand the amount of shared variance between two constructs, you simply "square" the correlation coefficient to yield the coefficient of determination.  Even with the highest level of predictive evidence with a predictive validity coefficient of 0.5, you are only accounting for 25% of the association between the two constructs!

    Within medicine, I believe that predictive validity plays an important role in imaging and early diagnosis.  One of the benefits of working in medicine is that the measures are more objective, concrete, observable, validated, and measurable versus the social sciences.  Correlations of 0.9 are common between various etiological, prognostic, confounding, clinical, and demographic phenomena within medicine.  If an imaging or diagnostic method can detect the earlier stages of a progressing disease state, then future outcomes can be mitigated with earlier and preventative treatment.  
  • Published on

    The role of correlations in psychometrics

    Correlations are used to generate validity evidence

    Concurrent, predictive, convergent, and divergent validity

    Correlations play a central role in applied psychometrics.

    The inter-correlations among survey instrument items play a role in calculating internal consistency reliability coefficients (Cronbach's alpha, split-half, KR-20), test-retest reliability (Spearman-Brown formula), and inter-rater reliability (Kappa, ICC). Correlation matrices also play a significant role in principal components analysis (eigenvalues, factor loadings).

    Correlations are used to generate convergent, predictive, and concurrent validity evidence. Significant correlations with theoretically or conceptually similar constructs/survey instruments denotes evidence of validity. In social sciences, a validity coefficient (or correlation coefficient) of .3 is considered evidence of validity.

    Pearson's r and Spearman's rho are the most prevalent correlation tests used to generate validity evidence. These correlations are used with survey instruments that generate ordinal or continuous outcomes.
  • Published on

    Precision and Accuracy

    Precision and Accuracy

    Cornerstones of measurement reasoning

    Precision and accuracy are terms that are debated intensely in empirical arenas. While definitions will differ from textbook to textbook and within different academic circles, here is a general definition and explanation for both terms:  

    Precision relates to the reliability, consistency, and stability of a variable or outcome, as it is measured in a given population. Commonly in research and biostatistics, precision is assessed using confidence intervals (most often, 95% confidence intervals).

    When using categorical outcome variables in bivariate and multivariate analyses, the precision of odds ratios yielded from analyses is determined by the width of the confidence interval. WIDE confidence intervals mean that there is LESS precision/reliability/consistency/stability/confidence in the measure. Wide confidence intervals are attributed to small sample sizes when using categorical outcomes.

    Analyses using continuous outcomes report the 95% confidence intervals or standard errors of means, mean differences, and unstandardized beta coefficients. Sample size also plays an important role in the width of confidence intervals when using continuous outcomes.

    Precision is often communicated as reliability in psychometrics. Survey instruments are pilot tested and then reliability coefficients are generated using test-retest, internal consistency, or inter-rater methods.  

    Accuracy pertains to the validity, utility, and interpretability of a variable or outcome, as it is measured in a given population. The accuracy or validity of a measure relies upon the methods, assessment, and evidence through which it was created using a theoretical or conceptual framework. In order for a measure to be deemed accurate, it must go through rigorous testing and application in the clinical environment.

    With clinical measures related to "gold standard" treatments, the absolute risk reduction (ARR) and the number needed to treat (NNT) or the absolute risk increase (ARI) and the number needed to harm (NNH) needs to be established using randomized controlled trials and systematic reviews. With diagnostic tests, the sensitivity, specificity, positive predictive value (PPV), negative predictive value (PPV), and total diagnostic accuracy need to be compared against a current and widely accepted "gold standard" diagnostic test.

    Finally, in psychometrics, construct validity is established by gathering many different forms of empirical evidence related to the interpretability, utility, and consequences of the measure. Researchers often use correlations, between-subjects analyses, and multivariate statistics to generate validity evidence. Predictive, concurrent, convergent, and discriminant validity evidence is generated using bivariate correlations. Known-groups validity is generated using parametric and non-parametric statistical tests.  Incremental validity is yielded using statistical regression techniques.