Volume 29 Number 1, 2025 The potential advantages of using an LLM-based chatbot for automated writing evaluation for English teaching practitioners

The potential advantages of using an LLM-based chatbot for automated writing evaluation for English teaching practitioners

Repository record updated on:
May 16, 2025, 1:28 a.m.
Database record updated on:
May 19, 2025, 9:54 p.m.
Last Harvested on:
May 19, 2025, 9:54 p.m.
bitstream:
[['https://scholarspace.manoa.hawaii.edu/bitstreams/628654d2-ba05-4382-8fa2-9e6558c4a180/download', '29_01_10125-73628.pdf']]
bitstream_txt:
[['https://scholarspace.manoa.hawaii.edu/bitstreams/57e3c240-dfae-47ae-82b0-b89ffcda9050/download', 'full_text']]
collection:
Volume 29 Number 1, 2025
contributor.author:
Kim, Kyungmin Lee, Jang Ho Shin, Dongkwang
contributor.editor:
Mimi Li
date.accessioned:
2025-05-03T02:20:59Z
date.available:
2025-05-03T02:20:59Z
date.copyright:
2025
date.issued:
2025-05-05
description.abstract:
With the ever-increasing demand for assessing large amounts of student writing, Automated Writing Evaluation (AWE) has emerged as an efficient system to satisfy this demand. However, it has also been suggested that applying it in teachers’ writing classes may offer limited results. Given this, the present study developed an AWE chatbot based on ChatGPT 4.0-turbo, designed as an automated rater. A total of 465 narrative essays written by Korean high school EFL students were scored according to three criteria by the developed tool; these were then compared with the scores administered by two professional raters using various analytic measures. The results showed that the AWE chatbot’s scoring was strongly correlated with that of the human raters. Meanwhile, the many-facet Rasch model’s result indicated that the two human raters’ statistics demonstrated an excellent fit, whereas those of the developed AWE chatbot were slightly lower. The Coh-Metrix analysis suggested that the human raters’ scoring tendencies and GPT are largely aligned, indicating that both raters scored the essays similarly. Based on our findings, we suggest that the Large Language Model (LLM)-based AWE chatbot has great potential to assist teachers in EFL writing classrooms.
endingpage:
12
format.extent:
12
identifier.citation:
Kim, K., Lee, J. H., & Shin, D. (2025). The potential advantages of using an LLM-based chatbot for automated writing evaluation for English teaching practitioners. Language Learning & Technology, 29(1), 1–12. https://hdl.handle.net/10125/73628
identifier.issn:
1094-3501
identifier.uri:
https://hdl.handle.net/10125/73628
language:
eng
llt.topic:
Language Teacher Education and Technology Forum
number:
1
publicationname:
Language Learning & Technology
publisher:
University of Hawaii National Foreign Language Resource Center Center for Language & Technology
rights.license:
Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License
rights.uri:
https://creativecommons.org/licenses/by-nc-nd/4.0/
site_url:
/item/10125-73628/
startingpage:
1
subject:
Automated Writing Evaluation, ChatGPT, L2 Writing, Many-Facet Rasch Model
title:
The potential advantages of using an LLM-based chatbot for automated writing evaluation for English teaching practitioners
type:
Column Text
volume:
29