Volume 29 Number 1, 2025 A protocol for evaluating AI chatbots’ capabilities for low-resource language teachers

A protocol for evaluating AI chatbots’ capabilities for low-resource language teachers

Repository record updated on:
Dec. 5, 2025, 1:08 p.m.
Database record updated on:
Dec. 9, 2025, 7:38 p.m.
Last Harvested on:
Dec. 9, 2025, 7:38 p.m.
access-status:
open.access
bitstream:
[['https://scholarspace.manoa.hawaii.edu/bitstreams/64af4349-2b61-4638-9939-aaa24afd9ea4/download', '29_01_10125-73661.pdf']]
bitstream_txt:
[['https://scholarspace.manoa.hawaii.edu/bitstreams/162041e0-a3a9-4af3-a562-842dde2c167a/download', 'full_text']]
collection:
Volume 29 Number 1, 2025
contributor.author:
Swineheart, Nicholas Nguyen, Phuong Yeh, Ellen
date.accessioned:
2025-12-05T01:35:38Z
date.available:
2025-12-05T01:35:38Z
date.copyright:
2025
date.issued:
2025-12-08
description.abstract:
One constraint on the potential of generative AI (genAI) in language teaching is the variability in performance across languages. Large language models (LLMs) typically perform better in higher-resource languages in which they have received more training data (Godwin-Jones, 2025; Kern, 2024), but it is difficult to know how well an LLM performs in a given language. Since most users interact with an LLM through a chatbot, this paper presents a protocol that language instructors can use to evaluate a genAI chatbot’s ability to generate and analyze their target language in ways related to language teaching, along with results from piloting this protocol with 46 instructors across 26 languages. Within this protocol, instructors submitted a series of language teaching-related prompts to Microsoft Copilot and evaluated output in terms of linguistic quality, task completion, and usefulness for their teaching using a four-point Likert scale. Analyses of responses to Likert-scale items and open-ended questions indicated that while performance was perceived to be better in higher-resource languages, instructors of low- and extremely low-resource languages generally found the output to be potentially useful. The paper concludes with implications for evaluating genAI chatbots in language teaching and recommendations for categorizing languages based on availability of resources.
endingpage:
21
format.extent:
21
identifier.citation:
Swinehart, N., Nguyen, P., & Yeh, E. (2025). A protocol for evaluating AI chatbots’ capabilities for low-resource language teachers. Language Learning & Technology, 29(1), 1–21. https://doi.org/10.64152/10125/73661
identifier.doi:
https://doi.org/10.64152/10125/73661
identifier.issn:
1094-3501
identifier.uri:
https://hdl.handle.net/10125/73661
language:
eng
number:
1
publicationname:
Language Learning & Technology
publisher:
University of Hawaii National Foreign Language Resource Center Center for Language & Technology
rights.license:
Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License
rights.uri:
https://creativecommons.org/licenses/by-nc-nd/4.0/
site_url:
/item/10125-73661/
startingpage:
1
subject:
generative AI, low-resource languages, CALL evaluation
title:
A protocol for evaluating AI chatbots’ capabilities for low-resource language teachers
type:
Article Text
volume:
29