Students’ perceptions of using ChatGPT in a physics class as a virtual tutor

(2023) 20:63 Ding et al. Int J Educ Technol High Educ https://doi.org/10.1186/s41239-023-00434-1 International Journal of Educational Technology in Higher Education RESEARCH ARTICLE Open Access Students’ perceptions of using ChatGPT in a physics class as a virtual tutor Lu Ding1* , Tong Li2, Shiyan Jiang3 and Albert Gapud4 *Correspondence: 1 Department of Counselling and Instructional Sciences, UCOM 3858, University of South Alabama, Mobile, AL 36688, USA 2 Center for Emerging Media Design and Development, Ball State University, Muncie, IN, USA 3 Teacher Education and Learning Sciences, North Carolina State University, Raleigh, NC, USA 4 Department of Physics, University of South Alabama, Mobile, AL, USA Abstract The latest development of Generative Artificial Intelligence (GenAI), particularly ChatGPT, has drawn the attention of educational researchers and practitioners. We have witnessed many innovative uses of ChatGPT in STEM classrooms. However, studies regarding students’ perceptions of ChatGPT as a virtual tutoring tool in STEM education are rare. The current study investigated undergraduate students’ perceptions of using ChatGPT in a physics class as an assistant tool for addressing physics questions. Specifically, the study examined the accuracy of ChatGPT in answering physics questions, the relationship between students’ ChatGPT trust levels and answer accuracy, and the influence of trust on students’ perceptions of ChatGPT. Our finding indicates that despite the inaccuracy of GenAI in question answering, most students trust its ability to provide correct answers. Trust in GenAI is also associated with students’ perceptions of GenAI. In addition, this study sheds light on students’ misconceptions toward GenAI and provides suggestions for future considerations in AI literacy teaching and research. Keywords: GenAI, ChatGPT, Perception, Misconception, Physics problems Introduction Generative Artificial Intelligence (GenAI) has overhauled the landscape of educational practices. GenAI is a subclass of machine learning (ML) algorithms that can learn from text, images, audio, and video to produce new content based on trained data (Kasneci et al., 2023). Unlike other supervised algorithms, known as conditional models, GenAI produces artifacts with a wide variety and complexity. GenAI increased its prominence in the zeitgeist of the world in November of 2022 when OpenAI released the third major version of their chatbot, Chapt GPT (GTP-3). The release shocked the world with its capability to produce human-like text and conversations (Hu, 2023); in just two months the platform gained 100 million users and generated a plethora of headlines worldwide. GPT (Generative Pre-trained Transformer) models are trained using a large amount of publicly available online digital content. The data used to train the GPT-3 model came from various sources: the Common Crawl dataset, the expanded WebText dataset, two internet-based books corpora, and English Wikipedia (Brown et al., 2020). Since ChatGPT models were trained based on a large corpus of text data with a complicated language model (more than 175 billion parameters), ChatGPT can comprehend human © The Author(s) 2023. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creativecommons.org/licenses/by/4.0/. Ding et al. Int J Educ Technol High Educ (2023) 20:63 language and respond to complex and varied prompts meanwhile maintaining contextual coherence in conversations (OpenAI, 2022). Due to its ability to perform a wide range of tasks, educators have suggested that ChatGPT can be used as a tool to support teaching across a wide range of subjects, including programming (Sun et al., 2022), engineering (Qadir, 2023), journalism and media (Pavlik, 2023), nursing education (O’Connor & ChatGPT, 2023), and business (Alshater, 2022). Beyond subject-specific applications, ChatGPT has been proposed to be able to support teachers in creating syllabi and curricula, be used for flipped classrooms (Lo, 2023), and support adaptive learning, automated essay grading, and personalized tutoring (Baidoo-Anu & Ansah, 2023). Despite its competence in supporting teaching and learning, a literature review has revealed varying performance levels of ChatGPT across different subjects. Notably, it has demonstrated outstanding performance in economics, and satisfactory performance in programming, but falls short of expectations in mathematics (Lo, 2023). Current knowledge about how students perceive GenAI and how it can be used for teaching and learning remains limited. It is imperative to examine GenAI from the student’s perspective to understand how and what pedagogical solutions are needed to minimize the challenges that GenAI introduces while maximizing its potential for teaching and learning. In this study, we particularly tested one GenAI—ChatGPT—in an authentic physics class to understand student perceptions toward GenAI. We implemented ChatGPT in the classroom by utilizing its tutoring assistant potential as suggested by Baidoo-Anu and Ansah (2023). In STEM education, an instructor usually needs to teach a substantial number of students with various levels of proficiency and understanding (Karabenick, 2003). Consequently, students’ questions are more likely to be left unresolved leading to increased confusion. In this study, we are particularly interested in investigating how students perceive ChatGPT as a virtual tutor. To address this inquiry, we aim to answer the following research questions: 1. What is the accuracy of ChatGPT for addressing physics problems? 2. Do students’ trust in ChatGPT’s answers differ by the accuracy of the answers? 3. To what extent do students’ trust level influence their perception of ChatGPT? Background AI in education AI has been widely used in education for various purposes prior to the emergence of ChatGPT. Many intelligent tutor systems have been developed to monitor students’ learning processes, analyze their performance, and provide immediate personalized instructions and feedback. Some dialogue-based Intelligent tutor systems, such as AutoTutor, not only track students’ knowledge status and engage them in adaptive conversations but also detect and resp (...truncated)