Gaze cluster analysis reveals heterogeneity in attention allocation and predicts learning outcomes – Nature

Report on Gaze Cluster Analysis for Attention Measurement in Instructional Videos

Introduction

Instructional videos are pivotal in modern education, especially in Massive Open Online Courses (MOOCs), providing accessible knowledge transfer. The Sustainable Development Goal (SDG) 4, Quality Education, emphasizes inclusive and equitable quality education and lifelong learning opportunities for all. Maintaining learner attention during video instruction is crucial for effective learning outcomes aligned with this goal.

Traditional gaze measures, such as Inter-Subject Correlation (ISC), assume a single focal point of attention, which may not capture the complexity of attention allocation in dynamic video content. This study introduces a novel gaze measure, Gaze Cluster Membership (GCM), leveraging unsupervised machine learning (DBSCAN algorithm) to identify multiple meaningful foci of attention, thereby enhancing the assessment of learner engagement.

Methodology

1. Sample: 121 adults (mean age 25.76 years, 67% female) participated, with ethical approval and informed consent obtained.
2. Video Stimuli:
   • “Bang! You’re Dead” (6 minutes) – a suspenseful short film used to study attention in narrative contexts.
   • “Programming in Minecraft” (15:46 minutes) – an instructional video teaching Python programming in a Minecraft Education environment.
3. Instruments:
   • Mental effort was self-reported post-viewing of the instructional video.
   • Knowledge tests assessed programming knowledge before and after viewing.
4. Data Acquisition: Eye tracking (500 Hz) and EEG data were collected in controlled laboratory settings.
5. Data Analysis: ISC and GCM were computed to measure attention. GCM was derived using DBSCAN clustering on gaze positions per video frame.

Results

1. Cluster Detection:
   • For “Bang! You’re Dead,” 13,189 gaze clusters were detected across 8,630 frames, with participants’ gaze within clusters 94% of the time.
   • For “Programming in Minecraft,” 31,366 clusters were detected across 23,664 frames, with 92% gaze cluster membership.
   • Less than 4% of clusters were rated as not meaningful, confirming high validity of cluster detection.
2. Comparison of GCM and ISC:
   • GCM effectively differentiated attention in multi-foci scenes where ISC values were low, suggesting ISC’s limitations in complex scenes.
   • GCM showed higher sensitivity to attention allocation, aligning with SDG 4 by improving educational assessment accuracy.
3. GCM and Mental Effort: GCM significantly predicted self-reported mental effort during the instructional video (p < 0.01), indicating its validity as an attention measure.
4. GCM and Knowledge Acquisition:
   • GCM predicted post-test knowledge scores, controlling for prior knowledge (p = 0.01).
   • Attention measured by GCM was especially predictive during programming-related content periods, highlighting the importance of focused attention for learning.
   • A decline in GCM over time was more pronounced in participants with lower post-test knowledge, emphasizing the need for engagement strategies.

Discussion

This study presents GCM as an innovative, high temporal resolution measure of attention that accommodates multiple meaningful focal points within video content. This approach addresses limitations of traditional ISC measures and enhances understanding of learner engagement, directly supporting SDG 4 by promoting effective and inclusive education.

The use of unsupervised machine learning for gaze clustering eliminates biases associated with predefined Areas of Interest (AOIs), enabling scalable and objective attention assessment across diverse educational materials.

Findings suggest that instructional video design should incorporate elements that sustain attention, such as interactive features, to mitigate attention decline over time and improve learning outcomes, contributing to lifelong learning opportunities under SDG 4.

Limitations include the controlled laboratory setting and the need for replication in real-world environments. Future research should explore webcam-based eye tracking for broader accessibility and investigate the differentiation of attention-related internal states such as motivation and fatigue.

Conclusion

The development of Gaze Cluster Membership (GCM) offers a robust tool for measuring attention in instructional videos, with demonstrated predictive validity for mental effort and knowledge acquisition. This advancement supports the Sustainable Development Goal 4 by enhancing the quality and effectiveness of educational content delivery. GCM’s application can inform instructional design, enabling educators to identify challenging video segments and optimize learner engagement, thereby fostering inclusive and equitable quality education.

Implications for Sustainable Development Goals (SDGs)

• SDG 4 (Quality Education): GCM contributes to improving educational quality by providing precise attention metrics that inform instructional video design and learner engagement strategies.
• SDG 9 (Industry, Innovation, and Infrastructure): The application of machine learning algorithms like DBSCAN in educational research exemplifies innovation in learning technologies.
• SDG 10 (Reduced Inequalities): By enabling scalable and unbiased attention measurement, GCM supports equitable access to quality education across diverse learner populations.

1. Sustainable Development Goals (SDGs) Addressed or Connected

1. SDG 4: Quality Education
   • The article focuses on improving attention measurement in instructional videos, which are widely used in online learning and MOOCs.
   • It addresses the enhancement of learning outcomes and educational quality through better understanding of learner attention.
2. SDG 9: Industry, Innovation and Infrastructure
   • The article presents an innovative machine learning approach (DBSCAN) for gaze cluster analysis to measure attention.
   • It contributes to technological advancements in educational tools and methodologies.
3. SDG 3: Good Health and Well-being
   • Although indirectly, the study involves neurophysiological and cognitive aspects such as mental effort and attention, which relate to cognitive health and well-being.

2. Specific Targets Under the Identified SDGs

1. SDG 4: Quality Education
   • Target 4.3: Ensure equal access for all women and men to affordable and quality technical, vocational and tertiary education, including university.
   • Target 4.4: Increase the number of youth and adults who have relevant skills, including technical and vocational skills, for employment, decent jobs and entrepreneurship.
   • Target 4.7: Ensure that all learners acquire knowledge and skills needed to promote sustainable development, including through education for sustainable development and sustainable lifestyles.
2. SDG 9: Industry, Innovation and Infrastructure
   • Target 9.5: Enhance scientific research, upgrade the technological capabilities of industrial sectors, including encouraging innovation and research development.
3. SDG 3: Good Health and Well-being
   • Target 3.4: Promote mental health and well-being.

3. Indicators Mentioned or Implied to Measure Progress Towards the Identified Targets

1. Indicators related to SDG 4 (Quality Education):
   • Measurement of learner attention through gaze cluster membership (GCM) as an indicator of engagement and cognitive effort.
   • Post-test knowledge scores assessing learning outcomes after instructional video viewing.
   • Self-reported mental effort ratings as a proxy for cognitive engagement.
   • Temporal analysis of attention during relevant content periods in instructional videos.
2. Indicators related to SDG 9 (Innovation):
   • Use of machine learning algorithms (DBSCAN) for gaze data clustering as a technological innovation indicator.
   • Application of neurophysiological measures (EEG) and eye tracking to assess attention.
3. Indicators related to SDG 3 (Mental Health):
   • Assessment of mental effort as an indicator of cognitive well-being during learning.

4. Table of SDGs, Targets, and Indicators

Source: nature.com