Tags

  • Published on

    Intraclass Correlation Coefficient and inter-rater reliability

    ICC Intraclass Correlation Coefficient One-Way Random Two-Way Mixed Two-Way Random

    Inter-rater reliability with continuous ratings

    Two or more raters giving multiple continuous ratings

    The Intraclass Correlation Coefficient (ICC) is a measure of inter-rater reliability that is used when two or more raters give ratings at a continuous level.  There are two factors that dictate what type of ICC model should be used in a given study.

    1.  Will the raters given ratings for all observations?

    2.  Are the raters a sample from the overall population or are the raters the only people in the population?

    When raters do not give ratings on all observations (i.e. three ratings are given from a random sampling of three raters out of a possible six independent raters), then the One-Way Random model is used.

    When raters give ratings for all observations (i.e. three ratings are given from three raters from the overall population for each observation), then the Two-Way Random model is used.

    When raters give ratings for all observations and the raters are the only valid members of the population (i.e. three ratings are given from the most esteemed scholars in an area), then the Two-Way Mixed model is used.
  • Published on

    Categorical measurement caveats

    95% Confidence Interval Categorical Diagnostic Testing Inter-rater Reliability Intraclass Correlation Coefficient Kappa Statistic Multivariate Statistics Negative Predictive Value Odds Ratio With 95% CI Positive Predictive Value Relative Risk Sample Size Sensitivity Specificity Statistical-power-test

    Effects of categorical measurement

    Decrease statistical power and increase sample size

    Categorical variables are very prevalent in medicine. Measures like presence of comorbidities, mortality, and test results are categorical in nature. Here are some general caveats associated with categorical measurement and sample size:  

    1. Categorical outcomes will always DECREASE statistical power and INCREASE the needed sample size. This is due to the lack of precision and accuracy in categorical measurement.

    2. The underlying algebra associated with calculating 95% confidence intervals of odds ratios and relative risk is 100% dependent upon the sample size. With smaller sample sizes, by default, wider and less precise 95% confidence intervals will be found. If one of the cells of a cross-tabulation table has fewer observations that the other cells, then the 95% confidence interval will be wider and potentially not truly interpretable. A 95% confidence interval will become narrower or more precise only with larger sample sizes.  

    3. When using categorical variables for diagnostic testing purposes, larger samples sizes will be needed to calculate precise measures of sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV). With smaller sample sizes in diagnostic studies, a change in one or two observations can have drastic effects on the diagnostic values.

    This is especially true when there is a subjective rating used for purposes of diagnosing someone as "positive" or "negative" for a given disease state (radiologist reading an X-ray). Inter-rater reliability coefficients such as Kappa or ICC should be employed to ensure consistency and reliability among subsequent ratings and raters. Sensitivity, specificity, and PPV will be affected by inter-rater reliability. Receiver Operator Characteristic (ROC) curves can be used to find a given value where sensitivity and specificity of a test is maximized. ROC curves can also be used to compare the area under the curve (AUC) between several diagnostic tests at the same time so that the best can be chosen.  

    4. For each predictor categorical parameter (or variable) that you want to include in a multivariate model, you have to increase your sample size by at least 20-40 observations of the outcome. This due to the limited precision, accuracy, and statistical power associated with categorical measurement. Researchers HAVE to collect more observations in order to detect any potential significant multivariate associations.  

    In the case that a polychotomous variable is to be used in a model, create (a-1), where a is the number of categories, dichotomous variables with "0" as not being that category and "1" as being that category. For each level, 20-40 more observations of the outcome will be needed to have enough statistical power to detect differences amongst the multiple groups.        
  • Published on

    New propensity score matching, calculators, reliability, and regression diagnostics pages in Research Engineer

    Calculator Cronbach's Alpha Intraclass Correlation Coefficient Kappa Statistic KR-20 Linearity Normal Probability Plot Propensity Score Matching Regression Analysis Residual Analysis Split-half Reliability

    New pages for Research Engineer

    Increased content validity for the website

    Propensity score matching is a statistical methodology that is used in observation research designs. It also is very useful for controlling for confounding variables in multivariate models. A new page has been added describing its use in research.

    I published all of the research calculators available in Research Engineer to one page for easier access. New pages are also available for internal consistency reliability and inter-rater reliability. 

    If you use Research Engineer for purposes of regression analysis, then you have seen the methods analyses associated with residual analysis and meeting certain statistical assumptions like linearity, normality, and equal variances. New pages are available to give deeper insights into these important statistical assumptions.

    Click on a button below to get started! Thank you!