Recursive Temporal Perception Models With Stateful Frame Integration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models for autonomous and semi-autonomous systems face challenges in leveraging temporal context due to computational constraints, leading to noisy predictions and increased complexity, which limits the number of historical frames that can be utilized for meaningful insights.
Innovation Solution
The development of recursive-temporal models that utilize intermediate features from backbone layers to generate temporal training data, incorporating recursive layers to update states continuously, allowing for efficient training and inference while maintaining temporal stability without significant computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple input frames are used to provide temporal context, then prediction accuracy is improved, but computational demand and processing complexity increase significantly
Solution Approach 1:
The model is segmented into distinct components: a backbone network for feature extraction from individual frames, and a temporal module for integrating temporal context. This segmentation allows the system to process multiple frames without overwhelming computational complexity by dividing the task into manageable stages.
Solution Approach 2:
The patent transforms the temporal dimension by using a temporal module that processes sequences of backbone features. Instead of directly processing multiple full-resolution input frames which would create exponential computational growth, the system extracts features first (reducing dimensionality) then applies temporal processing in an additional dimension.
2Loss of information
If multiple input frames are processed per prediction, then temporal context is improved, but latency and processing resources increase
Solution Approach 1:
The backbone network performs preliminary feature extraction from each input frame before temporal integration. By pre-processing frames to extract essential features and discarding redundant pixel-level data, the system reduces the computational burden of subsequent temporal processing, thereby reducing latency while preserving temporal context.
Solution Approach 2:
The patent extracts only the necessary feature representations from each frame through the backbone network, separating essential temporal information from redundant data. This extraction approach allows the temporal module to focus on integrating meaningful temporal patterns without being burdened by raw pixel data from multiple frames.
3Stability of the object's composition
If post-processing techniques are applied to compensate for noise, then prediction stability is improved, but the model cannot effectively utilize temporal context during prediction
Solution Approach 1:
The temporal module serves as an intermediary between the backbone network and the prediction head. It receives backbone features from multiple frames and integrates temporal context before passing refined features to the prediction head. This intermediary structure enables the model to effectively utilize temporal context during prediction while maintaining stability, rather than relying on post-processing techniques.
Data Source
AI summary
In various examples, machine learning models that benefit from temporal context while being computationally efficient to train and use are described herein. For instance, the disclosed systems and methods may apply a temporal series of images to a model and use intermediate features output from one or more backbone layers of the model as training data. In some examples, one or more recursive layers and/or one or more head layers of the model—or another model—may be trained using the training data by applying the intermediate features to the recursive layer(s). The recursive layer(s) may output a state representative of a temporal combination of the intermediate features, and the state may be applied to the head layer(s) to make one or more predictions. During inference, the recursive layer(s) may, in some examples, continuously update the state based on previous states of the recursive layer(s).