Comparing Google and ChatGPT as Assistive Tools for Students in Solving Programming Exercises

Abstract: The study explores the field of technology-enhanced learning, providing insights into the evolving educational landscape and the impact of Large Language Models (LLMs), such as ChatGPT, in comparison to established educational tools like search engines. This paper specifically compares two different tools, ChatGPT and Google, in aiding students to solve programming exercises within their education and their impact on students’ learning. An experiment with first-year computer science students was conducted, dividing them into two groups, each using separate tools to solve a programming exercise and completing a related questionnaire. The results indicate significant differences in the tools usages, outcomes of the task completed and acquired knowledge. ChatGPT users had higher success rates for completing the task, but solely copied code from their tool. Students using Google had a lower success rate despite spending more time on the task. However the knowledge testing questionnaire showed that the group using Google demonstrated a better understanding of the topic at hand. In this study one of the main conclusions is that while ChatGPT offers efficiency in generating solutions, it tends to hinder students’ learning process, potentially due to their inexperience in utilizing LLMs for educational purposes. Students using a tool that lessens their learning and shows tendencies of copying existing solutions instead of solving problems by themselves might lead to universities having to adapt.