Designing a computer-based assessment system for evaluating writing proficiency in the Indonesian language

Journal of Language and Pragmatics Studies, Volume 4 Number 1 (Apr 2025), p. 35-42 e-issn 2984-6051 DOI: https://doi.org/10.58881/jlps.v3i2 https://jurnal.ympn2.or.id/index.php/JLPS Designing a computer-based assessment system for evaluating writing proficiency in the Indonesian language Sri Kusuma Winahyu1, Endry Boeriswati2, Fathiaty Murtadho3, Fairul Zabadi4 Universitas Negeri Jakarta, Indonesia1,2,3 Badan Riset dan Inovasi Nasional, Indonesia4 1Email: Abstract - Writing is a fundamental language skill that students must master in school-based learning. This study aims to develop a computer-based assessment system for evaluating Indonesian opinion articles, addressing the challenges teachers face in manual writing assessments. Utilizing the initial phases of research and development (R&D) with the ADDIE model, this study conducted a needs analysis through literature reviews, interviews, and questionnaires. Findings indicate the necessity of a computer-based writing assessment system incorporating both machine and human raters. The machine rater evaluates mechanical aspects and vocabulary (word count) using pre-processing techniques in Natural Language Processing (NLP), supported by an Indonesian vocabulary database and punctuation programming. Meanwhile, the human rater, an Indonesian language teacher, conducts assessments via an interactive interface. Testing by four teachers on 40 students revealed that 97.6% of teachers responded positively to the system’s assessment process. This research is particularly relevant in the postCOVID-19 era, highlighting the positive role of technology in advancing language education and information technology. The system can be adapted for assessing other language skills, such as reading, and for large-scale applications like the Indonesian language proficiency test. Keywords: computer-based test; human rater; machine rater; natural language processing; opinion article 1. Introduction Writing is an essential language skill that students must master in educational settings, alongside listening, reading, and speaking. Within the Indonesian education curriculum, writing is emphasized across all grade levels. For instance, students in grade X learn to produce various texts such as anecdotal, exposition, and complex procedural texts, while grade XI students focus on short stories, rhymes, and reviews of films or dramas. At the grade XII level, students are introduced to writing historical stories, news, advertisements, and opinion articles, which are considered particularly challenging due to their argumentative structure (Wiratno & Santosa, 2019). Opinion articles require students to construct well-structured arguments presenting This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/) 35 Journal of Language and Pragmatics Studies, Volume 4 Number 1 (Apr 2025), p. 35-42 e-issn 2984-6051 DOI: https://doi.org/10.58881/jlps.v3i2 https://jurnal.ympn2.or.id/index.php/JLPS multiple perspectives on controversial issues, posing a significant challenge in both writing and assessment. Assessment of opinion articles is particularly complex due to the need to evaluate multiple components, including text structure, argument quality, and linguistic accuracy. This complexity, combined with the time-consuming nature of manual scoring, highlights the need for a more efficient and objective system. The COVID-19 pandemic further accentuated this need, as educators and researchers were compelled to adopt technology-based learning and assessment tools to facilitate distance education. Thus, a computer-based writing assessment system tailored to the Indonesian context is not only timely but also critical for enhancing the quality and efficiency of language education. The challenges of assessing opinion articles manually underscore the importance of developing an automated system. Manual assessment often leads to inconsistencies and delays, particularly when teachers are tasked with evaluating large volumes of student work. Furthermore, traditional assessment methods lack the scalability and adaptability required to meet the demands of modern education, particularly during crises like the COVID-19 pandemic. Automated assessment systems can provide consistent, objective evaluations while significantly reducing the workload for educators. In addition to addressing practical challenges, a computer-based assessment system can support the broader goals of language education by promoting self-directed learning. With an automated system, students can receive immediate feedback, enabling them to identify and address their weaknesses independently. Such systems can also be adapted for assessing other language skills, including reading and listening, as well as for conducting large-scale standardized tests such as the Indonesian language proficiency test. Therefore, this study is crucial for advancing both the pedagogical and technological dimensions of language education. Automated writing assessment has a long history, beginning with the development of essay grading machines in the 1960s. Early systems focused primarily on mechanical aspects of writing, such as grammar and vocabulary, but lacked the capacity to evaluate more nuanced features like rhetorical structure and coherence (Shermis et al., 2010). Subsequent advancements in computational linguistics led to the development of systems capable of assessing text semantics and organization. For example, Latent Semantic Analysis (LSA) has been used to evaluate the semantic similarity between student essays and model answers, achieving accuracy rates exceeding 86% (Ratna et al., 2015). In the Indonesian context, two notable systems have been developed for automated writing assessment. The first system evaluates the mechanical and lexical aspects of sentences produced by Indonesian language learners, while the second, SIMPLE-O, uses LSA to assess student essays by comparing them to model answers provided by instructors (Ratna et al., 2015). However, both systems have significant limitations. They primarily focus on surface-level features and fail to account for rhetorical and linguistic nuances, making them unsuitable for assessing opinion articles in a high school setting. This study seeks to address these gaps by developing a comprehensive computer-based writing assessment system tailored to the unique requirements of Indonesian language education. What constraints are faced by Indonesian language teachers when manually assessing opinion articles written by grade XII students? How can a computer-based writing assessment system be designed to address these constraints and meet the needs of Indonesian language teachers? This study aims to explore the challenges faced by teachers in manually assessing opinion articles and to develop a computer-based assessment system tailored to the In (...truncated)