Kappa
A very conservative measure of inter-rater reliability
The Kappa statistic is utilized to generate this estimate of reliability between two raters on a categorical or ordinal outcome. Significant Kappa statistics are harder to find as the number of ratings, number of raters, and number of potential responses increases.
The steps for conducting a Kappa statistic in SPSS
1. The data is entered in a between-subjects fashion.
2. Click Analyze.
3. Drag the cursor over the Descriptives drop-down menu.
4. Click on Crosstabs.
5. Click on the first rater's observations of the outcome to highlight it.
6. Click on the arrow to move the variable into the Row(s): box.
7. Click on the second rater's observations of the outcome to highlight it.
8. Click on the arrow to move the variable into the Column(s): box.
9. Click on the Statistics button.
10. Click on the Kappa box to select it.
11. Click Continue.
12. Click OK.
2. Click Analyze.
3. Drag the cursor over the Descriptives drop-down menu.
4. Click on Crosstabs.
5. Click on the first rater's observations of the outcome to highlight it.
6. Click on the arrow to move the variable into the Row(s): box.
7. Click on the second rater's observations of the outcome to highlight it.
8. Click on the arrow to move the variable into the Column(s): box.
9. Click on the Statistics button.
10. Click on the Kappa box to select it.
11. Click Continue.
12. Click OK.
The steps for interpreting the SPSS output for the Kappa statistic
1. Look at the Symmetric Measures table, under the Approx. Sig. column. This is the p-value that will be interpreted.
If the p-value is LESS THAN .05, then there is significant agreement between the raters.
if the p-value is MORE THAN .05, then there is not agreement between the raters.
2. When interpreting Kappa, the following criteria are used:
0.40 - 0.59 is considered moderate agreement
0.60 - 0.79 is considered substantial agreement
0.80 - 0.99 is considered outstanding agreement
If the p-value is LESS THAN .05, then there is significant agreement between the raters.
if the p-value is MORE THAN .05, then there is not agreement between the raters.
2. When interpreting Kappa, the following criteria are used:
0.40 - 0.59 is considered moderate agreement
0.60 - 0.79 is considered substantial agreement
0.80 - 0.99 is considered outstanding agreement
Click on the Validity button to continue.
Hire A Statistician
DO YOU NEED TO HIRE A STATISTICIAN?
Eric Heidel, Ph.D., PStat will provide the following statistical consulting services for undergraduate and graduate students at $100/hour. Secure checkout is available with Stripe, Venmo, Zelle, or PayPal.
- Statistical Analysis
- Research Design
- Sample Size Calculations
- Diagnostic Testing and Epidemiological Calculations
- Survey Design and Psychometrics