Gated Transformer Video Processing for Reusing Frame Features

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing neural networks for processing video content inefficiently utilize computing resources due to lack of consideration for spatial and temporal redundancies, leading to increased computational expense and delays in object detection tasks.

Innovation Solution

Implementing a gated transformer neural network that leverages both spatial and temporal redundancies by using gating mechanisms to identify unchanged and changed content, allowing features to be reused or recomputed only when necessary, thereby reducing the amount of data processed.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional neural networks process video content on a per-image basis, then object detection can be performed, but computational resources are inefficiently utilized due to not considering spatial and temporal redundancies

Engineering Contradiction:

Improveobject detection speedVSAvoidcomputational expense

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The video content is segmented into unchanged regions and changed regions based on temporal redundancy analysis. The processing system divides the video stream into segments that can be handled differently - unchanged segments reuse features from previous frames while changed segments undergo full processing, thereby reducing overall computational expense while maintaining detection speed.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary analysis to identify unchanged content in video frames before conducting object detection. By pre-identifying regions that remain static across frames, the system can skip redundant processing steps for these regions, reducing computational expense while preserving detection accuracy for dynamic regions.

Inventive Principle:

Principle #10Preliminary action

2Measurement precision

If all features are recomputed for each frame, then accurate object detection is achieved, but processing time increases due to redundant computations

Engineering Contradiction:

Improvedetection accuracyVSAvoidprocessing delay

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system applies different processing qualities to different regions of video frames based on their temporal characteristics. Unchanged regions receive minimal or no processing (reusing features from previous frames), while changed regions receive full processing attention. This local differentiation maintains detection accuracy for dynamic regions while reducing overall processing time.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system discards redundant feature computations for unchanged regions by reusing features from previous frames, and recovers accurate detection results by focusing computational resources on changed regions. This selective discarding and recovering approach reduces processing delay while maintaining detection accuracy where it matters most.

Inventive Principle:

Principle #34Discarding and recovering

3Reliability

If comprehensive feature processing is performed on all video content, then detection accuracy is maintained, but power utilization increases

Engineering Contradiction:

Improvedetection reliabilityVSAvoidpower utilization

Core Design Contradiction:

ReliabilityVSUse of energy by stationary object

Solution Approach 1:

The system dynamically adjusts processing intensity based on the temporal characteristics of different video regions. Processing power is dynamically allocated to changed regions while unchanged regions receive minimal processing. This dynamic adaptation maintains detection reliability for moving objects while significantly reducing overall power utilization compared to static full-frame processing.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS12469281B2Processing video content using gated transformer neural networks

Publication Date: 2025.11.11 QUALCOMM INC

AI summary

Certain aspects of the present disclosure provide techniques and apparatus for processing a video stream using a machine learning model. An example method generally includes generating a first group of tokens from a first frame of the video stream and a second group of tokens from a second frame of the video stream. A first set of tokens associated with features to be reused from the first frame and a second set of tokens associated with features to be computed from the second frame are identified based on a comparison of tokens from the first group of tokens to corresponding tokens in the second group of tokens. A feature output is generated for portions of the second frame corresponding to the second set of tokens. Features associated with the first set of tokens are combined with the generated feature output into a representation of the second frame.