Can you beat ChatGPT?

Introduction:

The integration of advanced natural language processing tools such as ChatGPT, Copilot, and Gemini into various domains, including education, has prompted a re-evaluation of traditional teaching methods. ChatGPT’s potential to enhance critical thinking and problem-solving skills while highlighting the importance of information verification aligns with the educational objective of preparing students for real-world challenges. This report examines the use of ChatGPT in an educational setting, specifically within the ITEC311 Network Security and Applications unit, through an assessment designed to evaluate students’ abilities to leverage AI tools for theoretical and technical problems.

Research Questions:

1. How effective is ChatGPT in assisting students with theoretical and technical problems?
2. Can the use of ChatGPT enhance critical thinking and problem-solving skills among students?
3. What are the limitations of ChatGPT in providing accurate and contextually relevant answers?
4. How do students’ abilities to verify and correct ChatGPT-generated responses contribute to their learning outcomes?

Research Methodology:

This project involved designing an assessment comprising two parts. In Part I, students were given theoretical and technical problems to solve using any tools, including ChatGPT. In Part II, students were provided with both theoretical and technical problems and ChatGPT-generated answers, which they had to evaluate for correctness and provide justifications. The students’ performance in both parts was analysed to understand the effectiveness of ChatGPT in enhancing learning outcomes and critical thinking skills.

Assessment Task Description:

The assessment included the following components:

• Part I: Theoretical Questions (A1, A2, A3, A4) and Technical Questions (A5, A6, A7)- The theoretical questions are worth a total of 12 marks, while the technical questions are worth 13 marks. Students used Artificial Intelligence (AI) tools and other resources to solve these problems.
• Part II: Theoretical Questions (B1, B2) and Technical Questions (B3, B4, B5)- This part also followed the same marking distribution for theoretical and technical questions. Students evaluated the correctness of ChatGPT-generated answers and provided their justifications.

Data from student responses were collected and analysed to assess their performance in both parts. Marks for each question were recorded, and the total scores for Part A and Part B were calculated. The analysis aimed to identify how effectively students solved the problems and evaluated ChatGPT’s answers.

Results and Analysis

The outcomes reveal varying levels of effectiveness in students’ use of ChatGPT for problem-solving and evaluation. The following demonstrates some key observations.

1. Performance in Part A:
   – Theoretical Questions: Students demonstrated a moderate level of proficiency in using ChatGPT to solve theoretical problems, with most scores ranging from 8 to 10 out of 12.
   – Technical Questions: Scores varied more widely, indicating differing levels of understanding and ability to leverage ChatGPT for technical problem-solving.
2. Performance in Part B:
   – Theoretical Questions: Students struggled in identifying correct and incorrect ChatGPT responses. 20 out of 32 students scored within the range of 0 to 5.
   – Technical Questions: The evaluation of technical answers showed a broader range of performance, suggesting that technical problem verification posed more challenges.
3. Critical Thinking and Problem-Solving: The assessment demonstrated that 56.25% students could effectively critique ChatGPT responses, contributing to enhanced critical thinking skills. However, the ability to provide accurate corrections varied, reflecting the complexity of the problems.
4. Limitations of ChatGPT: The quality of ChatGPT’s answers varied, and students noted instances of nonsensical or incorrect information. This highlights the importance of teaching students to critically evaluate AI-generated content.

In summary, students performed better in Part I compared to Part II. The average marks obtained in Part I were 17.48 out of 25, while the average for Part II was 13.19. One reason for this difference could be that in Part I, students could leverage different AI-tools to generate solutions, providing a more structured and guided approach to problem-solving. In Part II, however, students could not solely rely on ChatGPT for answers; they needed to independently assess and validate the AI’s responses, which is a more challenging and skill-intensive task.

kabiswas1