Multimodal Action Recognition via Temporal Fusion of Video and Sensor Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multimodal data fusion techniques for activity recognition, particularly in healthcare, are not accurate enough for reliable human activity classification and decision-making due to limitations in capturing temporal dependencies and correlations across different data modalities.
Innovation Solution
A multimodal sensing system that uses a combination of video cameras and wearable sensors to extract features from video and other data modalities, employing multi-layer long-short-term memory networks for temporal aggregation and fusion, enabling the creation of a fused feature representation that captures real-world actions across multiple data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing multimodal data fusion schemes are used, then multiple data modalities can be processed, but the accuracy of action classification and decision-making is insufficient
Solution Approach 1:
The patent segments the action recognition process into distinct temporal phases (pre-action, during-action, post-action) and processes each phase separately with phase-specific feature extraction and fusion strategies. This segmentation allows the system to capture temporal dependencies at different granularities, improving classification accuracy while maintaining reliability through phased decision-making
Solution Approach 2:
The patent introduces a temporal dimension by processing data in sequential phases rather than treating all modalities simultaneously. It adds temporal ordering to the fusion process, where early-phase features inform later-phase processing, creating a hierarchical temporal structure that enhances both accuracy and reliability of action classification
2Measurement precision
If temporal dependencies are explicitly modeled through multi-layer LSTM networks, then action classification accuracy improves, but system complexity increases
Solution Approach 1:
The patent divides the complex temporal processing into multiple sequential LSTM layers, each handling specific temporal aspects of different phases. The first LSTM layer processes early-phase temporal patterns, while subsequent layers refine these patterns with later-phase information, breaking down the overall complexity into manageable sequential stages
Solution Approach 2:
The system dynamically adjusts the processing depth and fusion strategy based on the detected action phase. During critical transition phases, the system engages deeper LSTM processing to capture complex temporal dependencies, while during stable phases, it uses lighter processing, thereby adapting computational complexity to actual temporal needs
3Loss of information
If multiple sensor modalities are integrated, then uncertainty in data decreases, but the difficulty of processing and fusing diverse data types increases
Solution Approach 1:
The patent applies modality-specific feature extraction and processing strategies tailored to each data type's characteristics. Video data receives spatial-temporal feature extraction, while sensor data undergoes temporal pattern recognition, with each modality processed using optimized techniques suited to its local properties before unified fusion
Solution Approach 2:
The patent introduces phase-based temporal features as intermediary representations that bridge different data modalities. These intermediate temporal embeddings serve as a common language between video and sensor data, facilitating their fusion while accounting for modality-specific characteristics through phase-aligned processing
Data Source
AI summary
A multimodal sensing system includes various devices that work together to automatically classify an action. A video camera captures a sequence of digital images. At least one other sensor device captures other sensed data (e.g., motion data). The system will extract video features from the digital images so that each extracted image feature is associated with a time period. It will extract other features from the other sensed data so that each extracted other feature is associated with a time period. The system will fuse a group of the extracted video features and a group of the extracted other features to create a fused feature representation for a time period. It will then analyze the fused feature representation to identify a class, access a data store of classes and actions to identify an action that is associated with the class, and save the identified action to a memory device.