CNN Video Processing via Temporal Static Region Skipping

Solution Overview

Problem

Existing machine learning-based video analysis for computer vision applications is inefficient due to direct per-frame processing of high frame rates, which ignores temporal correlations and results in increased complexity and reduced accuracy, while specialized video architectures do not guarantee the same accuracy as image recognition architectures.

Innovation Solution

The method involves retaining and reusing per-layer convolutional neural network intermediate data between frames to skip calculations for temporally static regions, optimizing processing by storing and retrieving intermediate results from a buffer to reduce convolution and activation calculations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If direct per-frame processing is used for high frame rate video sequences, then processing speed is improved, but computational complexity increases and accuracy is reduced

Engineering Contradiction:

Improveprocessing speedVSAvoidcomputational complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The patent performs preliminary identification of temporally static image areas between frames before executing the full convolutional neural network processing. By pre-processing to detect static regions, the system prepares information that enables subsequent skipping of redundant calculations, thus reducing computational complexity while maintaining processing speed for high frame rate video sequences

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent extracts and identifies temporally static image areas from video sequences by comparing consecutive frames. By separating static regions from dynamic regions, the system can apply different processing strategies - skipping CNN processing for static areas and full processing for dynamic areas, thereby reducing overall computational complexity while maintaining accuracy

Inventive Principle:

Principle #2Taking out (Extraction)

2Speed

If direct per-frame processing is used for high frame rate video sequences, then processing speed is improved, but accuracy is reduced

Engineering Contradiction:

Improveprocessing speedVSAvoidaccuracy

Core Design Contradiction:

SpeedVSMeasurement precision

Solution Approach 1:

The patent performs preliminary identification of temporally static image areas between frames before executing the full convolutional neural network processing. By pre-processing to detect static regions, the system prepares information that enables subsequent skipping of redundant calculations, thus reducing computational complexity while maintaining processing speed for high frame rate video sequences

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent extracts and identifies temporally static image areas from video sequences by comparing consecutive frames. By separating static regions from dynamic regions, the system can apply different processing strategies - skipping CNN processing for static areas and full processing for dynamic areas, thereby reducing overall computational complexity while maintaining accuracy

Inventive Principle:

Principle #2Taking out (Extraction)

3Productivity

If intermediate data is retained and reused between frames, then redundant calculations are eliminated, but memory usage increases

Engineering Contradiction:

Improveprocessing efficiencyVSAvoidmemory usage

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The patent applies local quality by retaining intermediate CNN data only for identified temporally static image areas rather than for entire frames. This selective retention strategy stores memory only where needed for skipped processing regions, optimizing the balance between processing efficiency gains from redundancy elimination and memory consumption for storing intermediate representations

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS20240406433A1Methods and apparatus for efficient execution of convolutional neural networks for compressed video sequences

Publication Date: 2024.12.05 INTEL CORP

AI summary

Example apparatus disclosed includes at least one memory, machine readable instructions, and programmable circuitry to at least one of instantiate or execute the machine readable instructions to process a first frame of a video sequence with a neural network, store intermediate outputs of at least one of a convolution layer or a pooling layer of the neural network, the intermediate outputs associated with the first frame, process a second frame of the video sequence based on the intermediate outputs associated with the first frame to skip processing of a temporally static area of the second frame by the at least one of the convolution layer or a pooling layer, the temporally static image area common to the first frame and the second frame.