Relative Position Network for Video Action Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks for action classification in video data face challenges with high memory requirements and instability, particularly when processing videos with hundreds or thousands of frames, leading to inaccurate classification of actions with large variations over time.
Innovation Solution
The method involves determining a weighted combination of feature responses associated with a sequence of actions over time using a set of weights and processing these combinations along with temporal position values through a neural network, employing a relative-position network (RPN) layer to recognize actions by generating and attenuating feature responses based on temporal relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Convolutional Neural Networks are used for video analysis, then classification capability is provided, but memory requirements become excessively large (100s of GB)
Solution Approach 1:
The patent segments the video processing task by separating spatial feature extraction (using CNNs on individual frames) from temporal relationship processing (using RPN to process feature responses across time). This segmentation allows the system to avoid loading entire video sequences into memory simultaneously, reducing memory requirements from 100s of GB to manageable levels while maintaining classification capability.
Solution Approach 2:
The patent introduces a temporal dimension to the feature representation by generating feature responses that encode temporal relationships between frames. The RPN processes these multi-dimensional feature responses (combining spatial features with temporal positioning information) to recognize actions, enabling the system to handle temporal variations without increasing spatial memory requirements.
2Adaptability or versatility
If Recurrent Neural Networks are used to process video frames, then ability to classify events with large variations over time is improved, but instability increases and training effectiveness decreases
Solution Approach 1:
The patent introduces an intermediary mechanism (RPN layer) that bridges spatial feature extraction and temporal action recognition. Instead of using recurrent connections that cause instability, the RPN acts as a mediator that processes temporal relationships through a stable, non-recursive architecture, maintaining the ability to handle large temporal variations while ensuring training stability and effective generalization.
Solution Approach 2:
The patent replaces the mechanical recurrent feedback mechanism (which causes instability) with a different computational approach. The RPN uses a feedforward architecture with temporal position encoding and weighted feature combination to achieve temporal modeling without the instability inherent in recurrent connections, substituting one mechanism for another that achieves the same functional goal with improved reliability.
3Productivity
If neural networks process hundreds or thousands of video frames, then action recognition coverage is improved, but accuracy decreases due to instability
Solution Approach 1:
The patent extracts and emphasizes the most critical temporal relationships by generating feature responses that highlight key temporal patterns rather than processing all frame details equally. The RPN extracts salient temporal features and combines them with spatial features, allowing the system to maintain high accuracy while processing large numbers of frames by focusing computational resources on the most informative temporal relationships.
Data Source
AI summary
The present disclosure provides a method of recognising a first action. The method comprises determining a weighted combination of a first plurality of feature responses associated with a sequence of second actions over a period of time using a set of weights at a particular time instance in the period of time. The method then recognises the first action by processing, using a neural network, the weighted combination of the first plurality of feature responses and temporal position values of each of the first plurality of feature responses associated with the sequence of second actions.