Video Inference Quantization With Frame-Adaptive Precision Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional video classification methods are computationally intensive and memory-intensive, making them difficult to deploy on resource-limited devices like autonomous vehicles and mobile platforms, as they require high precision processing for all video frames, leading to inefficiency and high resource usage.
Innovation Solution
A dynamic network quantization technique that trains a lightweight policy network to determine optimal precision for each video frame using Gumbel SoftMax sampling, allowing flexible quantization of recognition networks to reduce computational and memory resources without sacrificing accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional classification methods use high precision processing for all video frames, then recognition accuracy is maintained, but computational cost and memory usage increase significantly
Solution Approach 1:
The patent implements dynamic precision adjustment by training a policy network that selects different precision levels for different video frames based on their importance. The recognition network switches between high precision (e.g., 32-bit) and low precision (e.g., 4-bit) processing dynamically, rather than using a fixed precision level for all frames. This resolves the contradiction by making precision adaptive to actual needs.
Solution Approach 2:
The patent applies different precision levels to different parts of the video data - specifically, different precision levels are applied to different video frames based on their classification importance. Important frames undergo high-precision processing while less important frames use low-precision processing. This local differentiation resolves the contradiction between maintaining overall accuracy and reducing total computational cost.
2Measurement precision
If conventional classification methods process all video frames with high precision, then classification accuracy is improved, but memory requirements increase
Solution Approach 1:
The policy network dynamically determines the precision level for each video frame, allowing the system to use low precision (reducing memory usage) for unimportant frames while reserving high precision for important frames. This dynamic allocation resolves the contradiction between maintaining classification accuracy and reducing overall memory requirements.
Solution Approach 2:
Different precision levels are applied locally to different video frames based on their importance. This selective approach ensures that memory-intensive high-precision processing is applied only where necessary, while low-precision processing handles the majority of frames, thus resolving the contradiction between accuracy and memory usage.
3Measurement precision
If high precision processing is applied to all video frames, then recognition performance is maintained, but processing speed decreases
Solution Approach 1:
The system dynamically adjusts precision levels based on frame importance determined by the policy network. Low-precision processing is applied to unimportant frames to maintain high processing speed, while high-precision processing is applied to important frames to maintain recognition performance. This resolves the contradiction between performance and speed.
Solution Approach 2:
Different precision levels are applied to different video frames based on their local importance characteristics. This selective precision application ensures that processing speed is maximized for unimportant frames while recognition performance is maintained for important frames, resolving the contradiction between overall performance and processing speed.
4Measurement precision
If data量和model深度 are increased to improve accuracy, then classification performance is improved, but FLOPs and computational complexity increase
Solution Approach 1:
The patent changes the precision parameter (bit depth) dynamically based on frame importance. Instead of using a fixed high precision for all frames, the system adjusts the precision parameter down for unimportant frames. This parameter change reduces computational complexity while maintaining classification performance for important frames.
Solution Approach 2:
The precision level is made dynamic rather than static, allowing the system to adapt computational complexity to actual needs. The policy network determines the appropriate precision level for each frame, enabling the system to reduce overall computational complexity while maintaining performance where necessary.
Data Source
AI summary
A recognition network is trained for a selected video frame at a desired highest precision using back-propagation and a policy network is trained using back-propagation from the trained recognition network. The recognition network is trained at a lower precision specified by a policy recommended for the selected video frame by the trained policy network. A frame of a given video is inputted to the trained policy network for determination of a precision policy for processing the frame. Video inferencing is performed utilizing the trained policy network and the trained recognition network based on the precision policy.