Exploring generative AI in higher education: a RAG system to enhance student engagement with scientific literature

Abstract

Introduction: This study explores the implementation and evaluation of OwlMentor, an AI-powered learning environment designed to assist university students in comprehending scientific texts. OwlMentor was developed participatorily and then integrated into a course, with development and evaluation taking place over two semesters. It offers features like documentbased chats, automatic question generation, and quiz creation. Methods: We used the Technology Acceptance Model to assess system acceptance, examined learning outcomes, and explored the influence of general self-efficacy on system acceptance and OwlMentor use. Results: The results indicated complex relationships between perceived ease of use, perceived usefulness, and actual use, suggesting the need for more dynamic models of system acceptance. Although no direct correlation between OwlMentor use and learning gains was found, descriptive results indicated higher gains among users compared to non-users. Additionally, general selfefficacy was strongly related to perceived usefulness, intention to use, and actual use of the system. Discussion: These findings highlight the importance of aligning AI tools with students’ needs and existing learning strategies to maximize their educational benefits.