Multimodal DNN Movement State Recognition via Sensor-Video Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing user movement state recognition technologies rely solely on sensor data, struggling to accurately identify detailed states like window shopping or pedestrian crossing, and face challenges with combining video and sensor data for precise recognition, especially when encountering unexpected data patterns.
Innovation Solution
A deep neural network (DNN) model is trained using both video and sensor data, with an annotation label rearranging section generating specific annotation data to enable the DNN to recognize movement states by distinguishing between predetermined classes and a miscellaneous-other class, allowing for high-precision recognition across various data sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a simple classification model (e.g., SVM) is used to combine video data and sensor data, then the device complexity is reduced, but the measurement precision of movement state recognition deteriorates due to differing levels of abstraction between video and sensor data
Solution Approach 1:
The patent segments the processing of video data and sensor data into separate feature extraction pathways. Video data undergoes frame extraction, object detection, and action recognition to generate video features. Sensor data undergoes separate feature extraction to generate sensor features. These segmented processing streams are then integrated at the feature level, allowing each data type to be processed optimally before combination, thereby improving recognition precision while managing complexity through modular architecture
Solution Approach 2:
The patent introduces an intermediary feature abstraction layer that mediates between raw video data and raw sensor data. Both data types are transformed into standardized feature representations (video features and sensor features) that operate at comparable levels of abstraction. This intermediary feature layer enables effective integration and comparison of heterogeneous data sources, resolving the precision problem caused by mismatched abstraction levels
2Adaptability or versatility
If a miscellaneous-other class is added to classify unexpected data patterns, then the adaptability of the classification system is improved, but the measurement precision deteriorates because this class contains a greater number of instances with wide-ranging data patterns that are difficult to distinguish
Solution Approach 1:
The patent implements a dynamic classification framework where the system can adaptively adjust its behavior based on input data characteristics. When unexpected patterns are detected, the system dynamically routes them through additional verification pathways rather than immediately classifying them as miscellaneous-other. This dynamic approach allows the system to maintain high precision for expected classes while still accommodating unexpected patterns through flexible, context-dependent classification logic
Solution Approach 2:
The patent applies preliminary analysis and verification steps before final classification. Unexpected data patterns undergo additional preprocessing, feature analysis, and contextual evaluation before being assigned to the miscellaneous-other class. This preliminary action filters out many patterns that could have been misclassified, reducing the burden on the miscellaneous-other class and improving overall precision by resolving ambiguities early in the processing pipeline
Data Source
AI summary
A movement state recognition multitask DNN model training section 46 trains a parameter of a DNN model based on an image data time series and a sensor data time series, and based on first annotation data, second annotation data, and third annotation data generated for the image data time series and the sensor data time series. Training is performed such that a movement state recognized by the DNN model in a case in which input with the image data time series and the sensor data time series matches movement states indicated by the first annotation data, the second annotation data, and the third annotation data. This thereby enables information to be efficiently extracted and combined from both video data and sensor data, and also enables movement state recognition to be implemented with high precision for a data set including data that does not fall in any movement state class.