International Journal of Language Testing
Volume:3 Issue: 1, Mar 2013

Writing assessments often make use of analytic rating scales to describe the criteria for different performance levels. However، the use of such rating scales requires a level of interpretation by raters and if several raters are involved، the reliability of examinee scores can be significantly affected (Engelhard، 1992; McNamara 1996). Variability between raters is partly managed by rater training in the use of the rating scale and this necessarily means that the rating scale itself should be well constructed and can be accurately applied to discriminate examinee performance consistently. This paper reports on the use of the Many-facets Rasch model (MFRM، Linacre، 1989) to assess the validity of a proposed analytic rating scale. The MFRM is widely used to study examinee performance and rater behavior and is useful in rating scale validation to analyze sources of variation in tests (Schaeffer، 2008). Bias analysis allows systematic subpatterns of interactions between raters and the rating scale to be examined. In this paper، scores from a set of essays rated by a team using a revised analytic descriptor were analyzed and the indices for rater severity، rater consistency، rater bias، criteria difficulty and scale functionality were studied. The findings indicate that raters were able to use the revised rating scale to discriminate performances in a consistent manner. The MFRM can contribute to improvements in rater training and rating scale development.

Traditional analysis of test results with Item Response Theory (IRT) models, and classical test theory (CTT) provide unidimensional ability values, which allow forranking of students according to their abilities. Cognitive diagnostic assessment (CDA) is aimed at providing formative diagnostic feedback through a fine-grained reporting of learners’ skill mastery. The present study, through application of deterministic input, noisy ‘‘and’’ gate [de la Torre & Douglas, 2004 (DINA) model], investigated the degree to which the items of a high stakes reading comprehension test can provide diagnostically useful information. Through expert rating, a Qmatrix including five subskills was developed. Using DINA model, student’s skill mastery profiles were produced and issues related to the diagnostic capacity of the reading test were discussed. Meanwhile, this study pedagogically demonstrates the application of the DINA model employing R software.