Language Education & Assessment

Special Issue Announcement:
Embracing Ubiquitous Technology in Plurilingual Education


Related Publications

Online Language Teaching: Crises and Creativities

Online Language Teaching: Crises and Creativities

Edited by Ursula Stickler, Martina Emke Online Language Teaching: Crises and Creativities collects creative and…
Insights into Teaching and Learning Writing

Insights into Teaching and Learning Writing

Edited by Hassan Mohebbi  & Yijen Wang  (2023) Writing is one of the most challenging skills…
Insights into Autonomy and Technology in Language Teaching

Insights into Autonomy and Technology in Language Teaching

Chun Lai  (2023) Drawing on the literature and research findings from relevant research fields, including…
Insights into Flipped Classrooms

Insights into Flipped Classrooms

Adrian Leis  (2023) This book is aimed at those interested in the flipped learning model as…
Insights into Task-Based Language Teaching

Insights into Task-Based Language Teaching

Sima Khezrlou  (2022) This book aims to offer a unique contribution to the expanding literature on…
Proceedings of the XXIst International CALL Research Conference

Proceedings of the XXIst International CALL Research Conference

Edited by Jozef Colpaert , Yijen Wang , Glenn Stockwell  (2022) The theme of the XXIst International CALL…
Insights into Professional Development in Language Teaching

Insights into Professional Development in Language Teaching

Thomas S.C. Farrell  (2022) From entering a teacher education program for initial qualifications onwards, language teachers…
Smart CALL: Personalization, Contextualization, & Socialization

Smart CALL: Personalization, Contextualization, & Socialization

Edited by Jozef Colpaert , Glenn Stockwell  (2022) The contributions in this volume focus on a…

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

DOI: 10.29140/lea.v5n1.769
Open Access

Role of expert judgement in language test validation

David Coniam https://orcid.org/0000-0003-4480-1742, Tony Lee https://orcid.org/0000-0003-1222-0038, Michael Milanovic https://orcid.org/0000-0002-5722-1811, Nigel Pike https://orcid.org/0000-0002-6260-012X, Wen Zhao https://orcid.org/0000-0003-4965-0146

– The calibration of test materials generally involves the interaction between empirical anlysis and expert judgement.


Author(s)

, , , ,

Paper type

Regular Articles

Pages

18-33

DOI

https://doi.org/10.29140/lea.v5n1.769

Year



Abstract

The calibration of test materials generally involves the interaction between empirical anlysis and expert judgement. This paper explores the extent to which scale familiarity might affect expert judgement as a component of test validation in the calibration process. It forms part of a larger study that investigates the alignment of the LanguageCert suite of tests, Common European Framework of Reference (CEFR), the China Standards of English (CSE) and China’s College English Test (CET).

In the larger study, Year 1 students at a prestigious university in China were administered two tests – one with items based on China’s College English Test (CET), and the other a CEFR-aligned test developed by LanguageCert. Comparable sections of the CET and the LTE involved sets of discrete items targeting lexico-grammatical competence.

In order to ascertain whether expert judges were equally comfortable placing test items on either scale (CET or CEFR), a group of professors from the university in China who set the CET-based test, were asked to expert judge the CET items against the nine CSE levels with which they were very familiar. They were then asked to judge the LTE items against the six CEFR levels, with which they were less familiar. Both sets of expert ratings and the test taker responses on both tests were then calibrated within a single frame of reference and located on the LanguageCert scale

In the analysis of the expert ratings, the CSE-familiar raters exhibited higher levels of agreement with the empirically-derived score levels for the CET items than they did with the equivalent LTE items. This supports the proposition that expert judgement may be used in the calibration process where the experts in question have a strong knowledge of both the test material and the standards against which the test material is to be judged.

Suggested citation

David Coniam, Michael Milanovic, Nigel Pike, Tony Lee, Wen Zhao. (2022). Role of expert judgement in language test validation. Language Education & Assessment, 5(1), 18–33. https://doi.org/10.29140/lea.v5n1.769

Please wait while flipbook is loading. For more related info, FAQs and issues please refer to documentation.

 

%d bloggers like this: