Sensor-Aided Spatio-Temporal Feature Extraction for Video Stabilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for extracting spatio-temporal features from user-generated videos are ineffective due to camera motion, which corrupts the motion information of recorded objects or scenes, making it difficult to analyze videos captured by uncontrolled devices like smartphones.
Innovation Solution
A method that utilizes auxiliary sensor data from devices like accelerometers and gyroscopes to segment video frames based on camera view motion, isolating 'stable segments' with minimal camera motion and modifying feature descriptors to account for camera motion, ensuring that extracted features represent real object motion independently of camera movement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If temporal features are extracted from user-generated videos, then motion information of objects can be captured, but camera motion corrupts the motion information making extraction ineffective
Solution Approach 1:
The video is segmented into stable segments and unstable segments based on camera motion detection. Temporal features are extracted only from stable segments where camera motion is minimal, ensuring accurate motion information capture while avoiding corruption from camera movement.
Solution Approach 2:
Sensor data from accelerometers and gyroscopes are used as intermediary information to detect and characterize camera motion. This intermediary data allows the system to identify and compensate for camera movement effects, separating true object motion from camera-induced motion artifacts.
2Measurement precision
If video stabilization using gyroscopes is applied, then camera motion can be corrected, but device complexity increases
Solution Approach 1:
The system uses the device's own built-in sensors (accelerometer and gyroscope) that are already present in smartphones to detect and correct camera motion. This self-service approach avoids adding external stabilization equipment, maintaining device simplicity while achieving effective motion correction.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method, apparatus and computer program product are provided for extracting spatio-temporal features with the aid of sensor information. An exemplary method comprises receiving video data and auxiliary sensor data and associating the two with timestamp information. The method may also include segmenting an input data stream into stable segments and extracting temporal features from the associated video data. The method may further include extracting temporal features either form the whole video or only from the video data where little or no stable segments are detected and performing camera view motion compensation by using information provided by the auxiliary sensors for modifying the feature-descriptors.