Learning Network Spatial-Temporal Feature Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural networks designed for image processing are inefficient in capturing both spatial and temporal features of videos, requiring excessive resources and failing to fully characterize temporal changes due to their complex design and high computational demands.
Innovation Solution
A learning network with separate units for spatial and temporal feature extraction, using 2D and 1D convolutional kernels respectively, to generate spatial-temporal feature representations, reducing network size and improving training and usage efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a complicated neural network is designed to extract feature information from video and images in both spatial and temporal dimensions, then the accuracy of spatial-temporal feature representation is improved, but the processing resources and storage resources required for training, storage and use of the neural network increase significantly
Solution Approach 1:
The patent divides the neural network into separate spatial processing units and temporal processing units. The spatial units extract spatial features from individual frames, while temporal units process the temporal relationships between frames. This segmentation allows each unit to be optimized independently, reducing overall network complexity while maintaining accurate spatial-temporal feature representation.
2Measurement precision
If a neural network is designed to consider temporal dynamic changes in a group of consecutive images or video frames, then the feature representation accuracy is improved, but the processing resources and storage resources required increase
Solution Approach 1:
The patent separates spatial feature extraction and temporal feature extraction into distinct processing stages. Spatial convolutional neural networks process individual frames to extract spatial features, while temporal processing units subsequently analyze the temporal relationships between these features across frames. This segmentation reduces the computational burden compared to processing all spatial-temporal relationships simultaneously.
Solution Approach 2:
The patent processes spatial dimensions (width and height of image frames) separately from the temporal dimension (sequence of frames). By extracting spatial features first and then applying temporal processing to the spatial feature sequences, the network efficiently handles multi-dimensional data without requiring excessive computational resources for simultaneous processing of all dimensions.
3Productivity
If a static image neural network is directly used to extract feature representation of each image or frame of video, then the processing resources are reduced, but the temporal dynamic changes in the video are not considered
Solution Approach 1:
The patent applies a two-stage processing approach where spatial convolutional neural networks first extract spatial features from individual frames (maintaining processing efficiency), and then separate temporal processing units analyze the temporal relationships between these spatial features across the video sequence. This ensures temporal dynamic information is captured without sacrificing processing efficiency.
Data Source
AI summary
Implementations of the subject matter described herein provide a solution for extracting spatial-temporal feature representation. In this solution, an input comprising a plurality of images is received at a first layer of a learning network. First features that characterize spatial presentation of the images are extracted from the input in a spatial dimension using a first unit of the first layer. Based on a type of a connection between the first unit and a second unit of the first layer, second features at least characterizing temporal changes across the images are extracted from the first features and/or the input in a temporal dimension using the second unit. A spatial-temporal feature representation of the images is generated partially based on the second features. Through this solution, it is possible to reduce learning network sizes, improve training and use efficiency of learning networks, and obtain accurate spatial-temporal feature representations.