Kappa

A very conservative measure of inter-rater reliability

The Kappa statistic is utilized to generate this estimate of reliability between two raters on a categorical or ordinal outcome. Significant Kappa statistics are harder to find as the number of ratings, number of raters, and number of potential responses increases.

The steps for conducting a Kappa statistic in SPSS

1. The data is entered in a between-subjects fashion.

2. Click Analyze.

3. Drag the cursor over the Descriptives drop-down menu.

4. Click on Crosstabs.

5. Click on the first rater's observations of the outcome to highlight it.

6. Click on the arrow to move the variable into the Row(s): box.

7. Click on the second rater's observations of the outcome to highlight it.

8. Click on the arrow to move the variable into the Column(s): box.

9. Click on the Statistics button.

10. Click on the Kappa box to select it.

11. Click Continue.

12. Click OK.

The steps for interpreting the SPSS output for the Kappa statistic

1. Look at the Symmetric Measures table, under the Approx. Sig. column. This is the p-value that will be interpreted.

If the p-value is LESS THAN .05, then there is significant agreement between the raters.

if the p-value is MORE THAN .05, then there is not agreement between the raters.

2. When interpreting Kappa, the following criteria are used:

0.40 - 0.59 is considered moderate agreement

0.60 - 0.79 is considered substantial agreement

0.80 - 0.99 is considered outstanding agreement
Click on the Validity button to continue.