Iranian EFL Raters’ Cognitive Processes in Rating IELTS Speaking Tasks: The Effect of Expertise

Variations in rating the EFL learners’ oral performance are often attributed to the variations in the raters’ cognitive processes. Han’s (2016) 4-stage processing model was used to examine what cognitive processes expert and novice raters follow to rate a recorded response to the IELTS Speaking Task Two by using the IELTS rubrics. Novice and expert raters attended the 4-phase verbal protocol sessions in order to explore the cognitive processes underlying (a) their representations of IELTS speaking rubric, (b) qualitative assessment of a recorded sample response to IELTS Speaking Task Two, (c) quantitative assignment of ratings to the input and (d) revision of the assigned ratings. Qualitative data collection was followed by transcribing, segmenting, encoding, and analyzing the contents of the recorded verbal protocol reports. After content analysis, the four categories of (1) grammatical range and accuracy, (2) fluency and coherence, (3) lexical resources, and (4) pronunciation in IELTS speaking rubric were schemed into 80 themes. NVivo 8 and SPSS 19 were used to analyze the data qualitatively and quantitatively, respectively. Both qualitative and statistical findings showed that the L2 raters with a different range of expertise widely focus on different aspects of the spoken response input, have different interpretations, and apply different criteria when judging the verbal input. The findings of the present study may carry implications for rater training and validity of ratings. Expertise, as the findings of the study show, can exert an influence on the reliability of the ratings.