Rating & reflecting: Developing rater identities in ESL oral assessment

Rater agreement when assessing L2 performance often varies across raters (e.g., McNamara, 1996). What raters believe, think, and know has implications for their assessment practice, but this cognitive dimension is not accessible from scoring data alone. Using interaction analysis as primary data, and survey responses and assessment data as secondary data, this study aims to uncover the role of raters' identities (displayed/self-reported) in L2 oral assessment.

Data were collected during a research-based training program on ESL oral assessment, encompassing three days over three months where participants (11 English teachers) received feedback on their assessment severity after Day 1. Video-recordings were made on Day 2, when teachers were asked to reflect upon their feedback in small groups. With a conversation analytic (CA) approach, we focus on participants' displays of 'rater identities' in relation to formulated assessment decisions. Self-reported survey responses and scoring data were then analyzed with the purpose of uncovering how 'rater severity' was treated across datasets. The interplay of datasets allowed for a deeper understanding of cognition among teachers-as-raters, as 'relative severity' came across as central in both self-reports and in group interaction. Findings revealed that sequential analysis of collegial assessment discussions has the potential to reveal teachers' stepwise modification of their rater cognition, and that the individual feedback was an important starting point for the collaborative negotiation of rater identities. Given that raters may understand L2 oral proficiency differently, studies uncovering raters' interpretations of scoring rubrics can extend our knowledge of how raters can be trained more effectively.