Machine Scoring

Machine scoring evaluates language performance against predefined proficiency criteria. Generative AI creates content. Unlike generative AI, which predicts and produces language, machine scoring operates within tightly defined parameters. In assessments such as the Spanish AAPPL, scoring models are trained on validated responses aligned to recognized standards, including the ACTFL Proficiency Guidelines and ACTFL Performance Descriptors. The goal is not to improvise; it is to replicate trained human judgment consistently and at scale.

Machine scoring models learn from large collections of authentic responses that have already been evaluated by ACTFL-certified human raters. They cannot simply be turned on and expected to rate writing or speaking accurately. The model learns the language features and performance characteristics associated with different proficiency levels and uses those patterns to evaluate new responses.

In that sense, machine scoring is not inventing scores; it is learning to mirror expert human ratings based on extensive evidence.

During development, machine-generated scores are compared with a validation dataset that is composed of test scores assigned by ACTFL-certified human raters and unseen by the machine scoring system. Statistical analyses are used to measure agreement and identify where the model performs well or needs improvement. The Spanish AAPPL machine scoring was validated against ACTFL-certified ratings over multiple years of administration. Ongoing monitoring and validation are necessary to confirm that the system continues to perform as intended over time.

Fairness begins with representative training data and continues through ongoing review, validation, and monitoring. ACTFL’s and LTI's approach is informed by established language assessment ethics principles and supported by independent expert oversight.

Only when there is enough high-quality, language-specific evidence behind it. Then, the machine needs to be trained on this language-specific evidence. Machine scoring depends on sufficient representative training data, validation against human raters, and ongoing monitoring. Without that evidence, scores may be less consistent, less accurate, or less fair.

Because testing volume varies by language, not every language has the sufficient data needed to support responsible machine scoring. Spanish AAPPL does, which is why machine scoring is currently available only for Spanish AAPPL.

The key principle is simple: efficiency should never come before evidence.

Disagreement between raters can happen in any scoring system, including systems that solely use human raters. If a disagreement occurs, an additional human rating is used to resolve the score. Human judgment remains the benchmark for evaluating machine scoring performance.

Schools should ask:

• What data was used to train the system?
• Was the data representative?
• What proficiency standards were used?
• Has the system been validated against human raters?
• How is the machine’s performance monitored?
• How often is the scoring model reviewed and recalibrated?

Voss, E., Sallee, K., Son, Y. A., Malone, M. E., Marshall, C., & Chomon-Zamora, C. (2026). An Automated Scoring System for the AAPPL Spanish Presentational Writing Tasks. Foreign Language Annals,59(1), 63-81.

Voss, E. (2025). Comparison of Traditional Machine Learning and Neural Network Approaches for Automated Scoring of Second Language English Essays. Language Testing, 42(4), 369-396.

Voss, E. (2024). Artificial Intelligence in Language Assessment. In A. J. Kunnan (Ed.), The Concise Companion to Language Assessment(pp. 112-125). Wiley-Blackwell.

Voss, E., Cushing, S. T., Ockey, G. J., & Yan, X. (2023). The Use of Assistive Technologies Including Generative AI by Test Takers in Language Assessment: A Debate of Theory and Practice. Language Assessment Quarterly, 20(4-5), 520-532.

Chapelle, C. A., & Voss, E. (2008). Utilizing Technology in Language Assessment. In N.H. Hornberger (Ed.), Encyclopedia of Language and Education(2nd ed., Vol. 7, Language Testing and Assessment, pp. 123-134). Springer.