Video Engagement Analysis Using Multimodal Continuous Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing engagement detection systems in online classes lack accuracy due to poor label quality, data imbalance, and reliance on high-level features that neglect important movement and behavioral information, leading to generalized engagement levels and ineffective engagement prediction.
Innovation Solution
An AI-based system that analyzes multi-modal features, including high-level visual characteristics, spatial-temporal characteristics, and audio features, using machine learning algorithms like TCN and MLP, to generate a continuous scalar engagement score, with suggestions for improvement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If traditional systems use four engagement classes (HD, DE, EG, HE), then the system is simple to implement, but the measurement precision of engagement levels deteriorates due to intra-class variation
Solution Approach 1:
The patent segments the engagement assessment into two distinct levels: (1) four discrete engagement classes (HD, DE, EG, HE) for high-level categorization, and (2) continuous engagement scores (0-1 scale) for fine-grained measurement. This segmentation allows the system to maintain simple classification while achieving precise measurement through the continuous score dimension.
Solution Approach 2:
The patent introduces a new dimension by adding continuous engagement scores as a second measurement axis alongside the traditional four classes. This dimensional expansion transforms the system from a single-dimensional classification (4 categories) to a two-dimensional assessment (class + continuous score), enabling precise differentiation within each class while preserving the simplicity of the original classification framework.
2Object-affected harmful factors
If the system focuses only on high-level visual features, then background noises are neglected, but movement and behavioral information are lost
Solution Approach 1:
The patent merges multiple feature extraction streams into a unified engagement assessment system: (1) high-level visual features (head position, body posture) that filter background noise, (2) spatiotemporal features (movement patterns, behavioral sequences) that capture dynamic information, and (3) audio features that provide additional context. This combination allows the system to retain both noise rejection capabilities and rich information capture.
Solution Approach 2:
The system implements multi-functional feature extraction modules that serve multiple purposes: visual features simultaneously provide engagement classification and noise filtering, while spatiotemporal features add movement analysis capabilities. Each module is designed to be universally applicable across different engagement scenarios, enhancing overall system robustness without sacrificing specialized information capture.
3Power
If AI-based analysis uses only facial spatial features and temporal features, then the model is computationally efficient, but prediction accuracy deteriorates
Solution Approach 1:
The patent combines multiple AI analysis streams with different computational characteristics: (1) facial spatial feature extraction using efficient CNN architectures, (2) temporal feature analysis using lightweight RNN/LSTM models, (3) spatiotemporal joint modeling using optimized 3D CNNs, and (4) audio feature processing using efficient spectral analysis. This merging achieves superior accuracy while maintaining computational efficiency through architecture optimization and feature fusion.
Solution Approach 2:
The system dynamically adjusts the complexity of AI models based on the specific analysis context and available computational resources. For real-time applications with limited resources, the system prioritizes efficient facial and temporal feature extraction. For offline analysis with higher computational capacity, it activates more intensive spatiotemporal joint modeling to maximize accuracy. This dynamic adaptation balances power consumption and prediction accuracy.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The invention relates to a computer implemented method and system for analyzing a video. The method comprises the steps of receiving, via a receiving module, data of the video comprising a series of images showing the subject; extracting, via an extracting module, features related to the subject from the video, the extracting step comprising: extracting, via a first extracting module, a first feature comprises one or more visual characteristics from one or more images of the series of images comprising the video; and extracting, via a second extracting module, a second feature comprises one or more spatial and/or temporal characteristics from the video; and processing, via an analyzing module, the extracted features to determine a score as a scalar representative of an engagement level of the subject.