Inter-rater reliability with continuous ratings
Two or more raters giving multiple continuous ratings
The Intraclass Correlation Coefficient (ICC) is a measure of inter-rater reliability that is used when two or more raters give ratings at a continuous level. There are two factors that dictate what type of ICC model should be used in a given study.
1. Will the raters given ratings for all observations?
2. Are the raters a sample from the overall population or are the raters the only people in the population?
When raters do not give ratings on all observations (i.e. three ratings are given from a random sampling of three raters out of a possible six independent raters), then the One-Way Random model is used.
When raters give ratings for all observations (i.e. three ratings are given from three raters from the overall population for each observation), then the Two-Way Random model is used.
When raters give ratings for all observations and the raters are the only valid members of the population (i.e. three ratings are given from the most esteemed scholars in an area), then the Two-Way Mixed model is used.
1. Will the raters given ratings for all observations?
2. Are the raters a sample from the overall population or are the raters the only people in the population?
When raters do not give ratings on all observations (i.e. three ratings are given from a random sampling of three raters out of a possible six independent raters), then the One-Way Random model is used.
When raters give ratings for all observations (i.e. three ratings are given from three raters from the overall population for each observation), then the Two-Way Random model is used.
When raters give ratings for all observations and the raters are the only valid members of the population (i.e. three ratings are given from the most esteemed scholars in an area), then the Two-Way Mixed model is used.
