An FAQ for Educators, Administrators, and Assessment Leaders about the Machine Scoring System for the AAPPL Spanish Presentational Writing and Interpersonal Listening and Speaking
As AI becomes more visible in education, machine scoring is gaining attention in language assessment. But one fact is often overlooked: not all automated scoring systems are built with the same level of evidence, oversight, or language-specific validation.
In world language assessment, credibility depends on more than technology. It depends on whether a scoring system has been trained on enough representative responses, aligned to recognized proficiency standards, validated against certified human raters, and continuously monitored after launch. The questions and answers below explain what that means in practice and why it matters for schools and programs making assessment decisions.
Q: Is machine scoring the same as generative AI, such as ChatGPT?
A: No. That distinction is essential. Machine scoring systems used in language assessment are not designed to generate content or make open-ended judgments. They are built for a specific purpose: evaluating language performance against established scoring criteria.
Unlike generative AI, which predicts and produces language, machine scoring operates within tightly defined parameters. In assessments such as the Spanish AAPPL, scoring models are trained on validated responses aligned to recognized standards, including the ACTFL Proficiency Guidelines and ACTFL Performance Descriptors. The goal is not to improvise; it is to replicate trained human judgment consistently and at scale.
Q: How does a machine learn to score language?
A: It learns from large volumes of human-scored responses. A machine-scoring model cannot simply be turned on and expected to rate writing or speaking accurately.
Instead, it must be trained on thousands—often hundreds of thousands—of responses that have already been scored by certified human raters using established proficiency criteria. The model learns from ACTFL-certified human raters about which language features and performance patterns correspond to a specific score. In that sense, machine scoring is not inventing scores; it is learning to mirror expert human ratings based on extensive evidence.
Q: Why did machine scoring for AAPPL take so long to develop?
A: Because building a trustworthy system takes years, not months. For Spanish AAPPL Presentational Writing (PW) and Interpersonal Listening and Speaking (ILS), the work has been the culmination of nearly a decade of research and development.
That timeline reflects two realities of responsible machine scoring:
Sufficient data: Reliable machine scoring depends on a very large pool of scored responses. Spanish, as one of the most widely tested AAPPL languages, generated enough data to support model development. The AAPPL itself is a standards-based assessment across the three modes of communication, with tasks informed by the ACTFL Proficiency Guidelines.
Alignment to standards: The machine must score consistently against recognized proficiency criteria. For the AAPPL, ratings are assigned according to ACTFL performance descriptors and proficiency guidelines, giving the model a defensible benchmark for learning.
In other words, the long timeline signals rigor. It reflects the time required to build a system ACTFL stands by and which educators can trust.
Q: How do we know machine scoring is accurate?
A: Because it was validated against human scoring before it could be used operationally.
During development, machine-generated scores are compared with a validation dataset that is composed of test scores assigned by ACTFL-certified human raters and unseen by the machine scoring system. Statistical analyses are used to measure agreement and identify where the model performs well or needs improvement. ACTFL and LTI state that the Spanish AAPPL machine scoring was validated against ACTFL-certified ratings over multiple years of administration.
Accuracy is not a one-time milestone. Ongoing monitoring and validation are necessary to confirm that the system continues to perform as intended over time.
Q: Can machine scoring work equally well in all languages?
A: Only when there is enough high-quality, language-specific evidence behind it. Without enough language-specific data, there is a risk of inconsistency in scoring and the potential for scores to inaccurately reflect a learner’s ability.
Machine scoring models learn from examples. If the training set is too small, too narrow, or not representative of the range of learners and proficiency levels, the model may miss important language patterns. The result can be weaker alignment with human raters, inconsistent scoring, and fairness concerns.
This challenge is especially important in world language assessment because testing volume is not evenly distributed across languages. A language with fewer scored responses may not yet have the evidence base needed to develop automated scoring. Spanish, by contrast, has had the scale to support this work. That’s one reason why it is the only AAPPL language for which the machine scoring system is currently available.
Responsible assessment organizations should resist broad claims about scoring every language equally well unless they can show language-specific validation. Efficiency is not the same as accuracy.
The principle is straightforward: machine scoring should be introduced only when the data, validation studies and ongoing monitoring are in place. In language assessment, evidence remains the deciding factor.
Q: How ethical is the automated scoring system used for the Spanish AAPPL PW and ILS?
A: Ethical responsibility is not removed from the machine scoring design; it is part of the design. In language assessment, that means addressing fairness, transparency, validity, and the risk of bias from the start.
Educators are right to ask whether an automated system can evaluate performance fairly across diverse test-taker populations, especially when speaking and writing scores may be used for placement, progress monitoring, or program decisions. Those concerns should not be dismissed; they should be answered with evidence.
ACTFL and LTI’s work is guided by the International Language Testing Association (ILTA) Code of Ethics, including the principle of technological responsibility. In practice, that means technological innovation must be managed with diligence and foresight so that fairness and assessment integrity are protected.
One important safeguard is representative training data. The Spanish AAPPL machine scoring system was trained on authentic responses by a variety of language learners and scored by ACTFL-certified raters, with the goal of aligning machine results closely to validated human judgment.
Ethical use also requires continued scrutiny after launch. Monitoring, research, and transparent communication about appropriate use are all part of responsible implementation.
Q: What additional approaches have ACTFL and LTI adopted to support the ethical management and use of AI in machine scoring?
A: ACTFL and LTI established an Expert Review Committee (ERC) that operates independently of both organizations. The committee provides impartial guidance on the use of machine scoring and includes experts in machine learning for language assessment, K-12 language education, and applied linguistics. It advises when and how machine scoring should be used, how its performance should be monitored, and how often reviews should occur. The ERC also helps ensure that these systems meet relevant legal standards and ethical expectations in language testing.
Q: What happens when the machine and a human rater disagree?
A: Disagreement between raters can happen in any scoring system, including systems that solely use human raters. When that happens, we apply the same finalization logic as we do when two human raters disagree – an additional human rating occurs.
Human ratings remain the benchmark for evaluating machine scoring performance. Before the scoring system is used operationally, it must demonstrate that it performs at a level comparable to, or better than, the rating agreement typically seen between qualified human raters. If ongoing monitoring shows that performance is not meeting expectations, the model can be refined and improved.
Q: Is machine scoring left on its own once it is implemented?
A: No. A credible system must be actively monitored.
Operational scoring systems should be audited regularly for consistency and fairness. New human-rated responses can be used to recalibrate the model, and anomalies must be investigated and addressed rather than ignored.
In short, machine scoring is not a set-it-and-forget-it tool. It is a system that requires ongoing oversight and management.
Q: Some tests claim to use AI scoring across multiple languages. Should schools trust those claims?
A: Schools should ask for evidence before they offer trust.
The right questions include:
- How large and representative of a sample was used to train the machine?
- What recognized standard or rating protocol was used for those samples?
- Has the machine scoring been validated against human raters?
- What does the ongoing monitoring and recalibration look like?
If an assessment provider cannot answer those questions clearly, caution is warranted. Automated scoring is only as reliable as the evidence behind it.
Q: What makes the AAPPL’s machine scoring approach different?
A: Its strength lies in the combination of standards, human benchmarking, long-term development, and ongoing monitoring and recalibration.
Standards-based training: the AAPPL tasks are informed by ACTFL proficiency expectations, and scoring is tied to ACTFL performance descriptors rather than vague criteria.
Human benchmarking: The machine scoring system for the AAPPL has been trained and validated against thousands of samples rated by ACTFL-certified human raters, which gives the model a defensible reference point for operational scoring.
Long-term development and monitoring: The development of the machine scoring system for the AAPPL took nearly a decade of research and development, followed by ongoing quality assurance in operational use.
Together, those factors make the approach not just technologically sophisticated, but assessment ready.
Q: Why does the development of machine scoring for AAPPL writing and speaking matter?
A: World language assessment has not benefited from the same depth of machine scoring research as English-language testing. Developing automated scoring of productive skills responsibly for languages other than English (LOTE), especially for school-age learners, requires substantial language-specific data, validated benchmarks, and careful psychometric work.
That is what makes the work around the Spanish AAPPL notable. The AAPPL is a K–12 assessment designed in real-world communicative contexts with interpersonal, interpretive, and presentational tasks, and its scoring is grounded in ACTFL standards. Extending machine scoring into that context is more complex than applying it to a generic writing task or an English-only test.
ACTFL and LTI have positioned this work as part of a broader effort to advance machine scoring in non-English language assessments, beginning with the Spanish AAPPL Presentational Writing (PW) and expanding into the Spanish AAPPL Interpersonal Listening and Speaking (ILS). Their public materials, such as research papers and academic conference presentations, emphasize validation against ACTFL-certified human raters and multi-year research as the foundation for operational use.
For schools and programs, the significance is practical: it shows that automated scoring in world languages can be developed responsibly, but only when the evidence is strong enough to support it.
Q: What is the bottom line for educators and decision-makers?
A: Not all AI scoring is accurate and deserves the same level of confidence.
Reliable machine scoring of speaking and writing is possible but only when it is built on substantial data, recognized standards, validation against human raters, and ongoing oversight. The AAPPL provides an example of what that level of rigor can look like in practice.
When those conditions are missing, the result may be efficient scoring without reliable measurement.
Before adopting any automated scoring solution, schools should ask the questions to know what they’re really getting. The quality of the evidence—not the appeal of the technology—should drive the decision.
