Interrater Agreement Coefficients Frits E. Zegers University of Groningen

the degree of agreement between two or more judges who rate a number of objects on certain characteri- stics can be expressed by means of an association coef- ficient (e. g., product moment correlation, PMC). The agre- ement among judges can rarely be perfect. If two judges produce two identical sets of scores the PMS will attain its maximum value, which equals +1. However, a PMC that equals +1 does not mean that the two sets of data are exactly equal. Different judges actually make judgments only based on the relative states of individuals, objects, or phenomena, and not based on their absolute states. This article aims to introduce association coefficients that are indicative of the degree to which there is an absolute agreement and real similarity among the ratings by different judges. Some pro- perties of the PMC in specific situations may be undesi- rable. Many association coefficients have been purposed for those situations, many of which belong to the class of Euclidian coefficients. A discussion of the desirable pro- perties of these coefficients demonstrates that how the ide- ntity coefficient and its generalizations can be used to as- sess the interrater agreements.