Video Feature Maps Using STSS for Motion Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Spatio-temporal convolutional neural networks (CNNs) struggle to effectively learn motion dynamics in video due to their compute-intensiveness and the need for additional supervision.
Innovation Solution
Implement a spatio-temporal self-similarity (STSS) tensor to enhance CNNs by generating spatial and temporal similarity maps, using 3D convolution layers, multi-layer perceptrons (MLPs), and soft-argmax to extract motion features, which are integrated into a residual block for end-to-end learning without additional supervision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 3D convolutional neural networks or optical flows are used to learn motion information in video, then motion learning capability is improved, but computational complexity increases
Solution Approach 1:
The patent segments the video processing task into two parts: (1) generating spatio-temporal feature maps from input video frames using 3D convolutions, and (2) computing optical flow fields separately from these feature maps. This segmentation allows motion information to be extracted without directly applying computationally intensive optical flow algorithms to the original high-resolution video data, thereby reducing overall computational complexity while preserving motion learning capability.
Solution Approach 2:
The patent introduces spatio-temporal feature maps as an intermediary representation between the input video and the motion analysis process. These feature maps serve as a compressed, abstracted form of the video data that captures essential spatio-temporal patterns. By performing optical flow computation on these intermediate feature maps rather than on the original video data, the system achieves motion learning with reduced computational burden.
2Measurement precision
If spatio-temporal convolutional neural networks are used to learn motion dynamics, then motion recognition accuracy is improved, but training requirements become more stringent
Solution Approach 1:
The patent implements a self-supervised learning mechanism where the network automatically generates its own training signals through the optical flow computation process. The optical flow fields derived from spatio-temporal feature maps serve as self-generated supervision signals that guide the training of the 3D convolutional network. This self-service approach eliminates the need for external annotated motion data, making the system easier to train while maintaining high motion recognition accuracy.
3Measurement precision
If spatio-temporal feature maps are generated and integrated with video data, then motion feature extraction is improved, but processing time increases
Solution Approach 1:
The patent applies partial action by computing optical flow fields only for specific spatio-temporal regions or at selected time intervals rather than for the entire video sequence continuously. The system selectively processes frames or regions where motion changes are detected, performing optical flow computation only when necessary. This partial processing approach maintains motion feature extraction accuracy while significantly reducing overall processing time compared to continuous full-frame optical flow computation.
Data Source
AI summary
An electronic device using a spatio-temporal self-similarity (STSS) and a method of operating the electronic device are disclosed. The electronic device may generate an STSS tensor including a spatial self-similarity map and spatial cross-similarity maps for each position of a video feature map corresponding to an input video based on a temporal offset and a spatial offset. STSS feature vectors may be generated from the STSS tensor by decreasing a dimension of the spatial offset and maintaining a dimension of the temporal offset for each position of the STSS tensor. An STSS feature map may be generated by integrating the dimension of the temporal offset for each position of the STSS feature vectors. An inference on the input video may be based on a result of adding the STSS feature map to the video feature map.