Rating quality in rater mediated language assessment: a systematic literature review - MohdNoh, Muhamad Firdaus; MohdMatore, Mohd Ewan Effendi; Faamanatu-Eteuati, Niusila; Rosman, Norhidayu (The journal of contemporary issues in business and government, 2021)
How has academic research in rating quality evolved over the last decade? Time has witnessed that previous researchers actively contributed to the development of knowledge particularly in ascertaining educational assessment to be updated with latest research-based practices. Thus, this review seeks to provide a bird’s-eye view of research development on rating quality over the last ten years focusing on factors influencing raters’ rating quality within the context of rater-mediated language assessment. This systematic literature review was conducted with the aid of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines through three stages, namely identification, screening and eligibility. Accordingly, the searching process has resulted in 43 articles to be thoroughly reviewed retrieved from two powerful database, Scopus and World of Science (WoS). Five major factors have emerged in response to the objective and they include rating experience, first language, rater training, familiarity and teaching experience. Analysis indicated that these factors lead to contradicting findings in terms of raters’ rating quality except for rater training factor. Only rater training was proven to be successful in mitigating rater effect and enhancing raters’ variability, severity and reliability. However, other factors were discovered to be inconclusive depending on whether they leave any impact on raters’ rating quality. The direction for future studies is also discussed suggesting the inclusion of more qualitative or mixed-method studies conducted to be reviewed using other possible techniques.