Motion Vector Unit for Selective ML Video Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional machine learning models for video content categorization are computationally expensive and lack real-time execution support, especially in applications without inference acceleration, making them inefficient for real-time processing in video encoding and decoding systems.
Innovation Solution
A system that includes a motion vector unit and a machine learning engine, where the motion vector unit pre-processes frames to determine temporal locality, activating the ML engine only when significant changes occur, and generating a quantization parameter map to optimize video encoding, thereby reducing computational resources and improving processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional machine learning models are used for video content categorization, then content categorization accuracy is improved, but computational complexity and processing time increase significantly
Solution Approach 1:
The patent segments the video processing task by applying the ML engine selectively only to frames containing scene changes rather than all frames. The motion vector unit identifies scene changes by comparing motion vectors between consecutive frames, and only triggers ML engine processing when significant changes are detected. This segmentation approach maintains categorization accuracy for relevant frames while dramatically reducing overall computational complexity.
Solution Approach 2:
The patent implements partial action by processing only a subset of frames (those with scene changes) through the computationally intensive ML engine, rather than applying it to every frame. The motion vector unit acts as a filter to identify which frames require ML processing, performing partial processing that is sufficient for real-time categorization while avoiding unnecessary computational overhead on static frames.
2Measurement precision
If machine learning engine is activated for every frame, then content categorization is performed continuously, but processing speed and real-time performance deteriorate
Solution Approach 1:
The patent implements periodic action by activating the ML engine intermittently based on scene change detection rather than continuously for every frame. The motion vector unit periodically assesses frame differences using motion vector comparison, and the ML engine is triggered only when scene changes are detected. This periodic activation pattern maintains real-time processing speed while ensuring categorization is performed at all critical moments.
3Reliability
If full frame processing is performed, then no scene changes are missed, but energy consumption and computational resources increase
Solution Approach 1:
The patent applies preliminary action by using the motion vector unit to pre-screen frames and identify scene changes before activating the ML engine. This preliminary detection step uses low-computation motion vector comparison to filter frames, ensuring that only frames with actual scene changes proceed to ML processing. This approach maintains reliable scene change detection while minimizing energy consumption by avoiding unnecessary ML engine activation.
Data Source
AI summary
Systems, apparatuses, and methods for performing machine learning content categorization leveraging video encoding pre-processing are disclosed. A system includes at least a motion vector unit and a machine learning (ML) engine. The motion vector unit pre-processes a frame to determine if there is temporal locality with previous frames. If the objects of the scene have not changed by a threshold amount, then the ML engine does not process the frame, saving computational resources that would typically be used. Otherwise, if there is a change of scene or other significant changes, then the ML engine is activated to process the frame. The ML engine can then generate a QP map and/or perform content categorization analysis on this frame and a subset of the other frames of the video sequence.