Reliability of assessment of five L1 groups in a high stakes test: Specific focus on Arabic L1 test takers

Abstract

Language proficiency tests are frequently employed in citizenship application processes in many countries. This article investigates a high-stakes proficiency test used in Finland – the National Certificate for Language Proficiency (NCLP) – by examining the reliability of assessments for one spoken task, in the light of test justice and fairness.

The data are drawn from project Broken Finnish: Accent perceptions in societal gatekeeping from NCLP. The data includes ratings from two assessment rounds, conducted in 2015 and 2016, based on a recorded spoken task completed by NCLP participants representing five different first language (L1) groups: Arabic, Estonian, Finland Swedish, Russian and Thai. We studied whether the repeated assessments were equivalent in terms of reliability regarding the L1 groups, with a specific focus on the Arabic L1 speakers. This focus was selected due to the pronounced discrimination experienced by Arabic L1 speakers in Finland, and the role of the NCLP gatekeeping citizenship, resulting in tangible real-life consequences for the test takers.

Quantitative statistical methods were utilized, such as Gamma Coefficient (γ) and the agreement between ratings with Cohen’s Kappa (κ) and Spearman rank-correlations (r). Results showed that the separate subjective assessments were not equivalent among different language groups. The assessments of the Arabic L1 test takers had drastic variability. This raises concerns regarding the reliability and consequential validity of the assessments, highlighting critical issues of fairness and justice in language testing.