Video Throughput via Neural Processing Unit Batching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multimedia systems face inefficiencies in utilizing device hardware for video processing, leading to high latency and increased overhead during high-resolution and high-frames-per-second video streaming, particularly in 4G and 5G networks.
Innovation Solution
The system configures devices to utilize neural processing units in conjunction with video hardware by batching video frames based on scene changes, allowing for parallel processing and load balancing between video hardware and neural processing units, with the neural processing unit operating in training or generation modes depending on motion vector information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If video frames are processed sequentially by video hardware, then processing accuracy is maintained, but processing latency increases and throughput decreases
Solution Approach 1:
The patent segments video frame processing into two distinct paths: traditional video hardware processing for I-frames and neural processing unit processing for P-frames and B-frames. This segmentation allows parallel processing of different frame types, increasing throughput while maintaining accuracy for reference frames through hardware processing.
Solution Approach 2:
The patent introduces a new processing dimension by adding neural processing unit capability alongside traditional video hardware. This creates a dual-processing architecture where different frame types can be handled by different processing units simultaneously, breaking the sequential processing bottleneck.
2Productivity
If neural processing unit is used for all video frames, then processing speed increases, but processing overhead and computational load increase
Solution Approach 1:
The patent applies different processing qualities to different frame types: full neural processing for non-reference frames (P-frames, B-frames) where speed is critical, and traditional hardware processing for reference frames (I-frames) where accuracy is paramount. This local differentiation optimizes the balance between speed and overhead.
Solution Approach 2:
The patent changes the processing parameter (processing unit selection) based on frame type. By identifying frame types through picture type indicators in the bitstream, the system dynamically selects between hardware and neural processing, reducing overall computational overhead while maintaining high throughput.
3Reliability
If video hardware processes all frames, then processing accuracy is maintained, but hardware utilization efficiency decreases
Solution Approach 1:
The patent creates a universal processing system where both video hardware and neural processing unit can handle video frame processing. The video hardware maintains accuracy for reference frames while the neural processing unit handles predictive frames, achieving multi-functionality that improves both accuracy and utilization efficiency.
Solution Approach 2:
The patent introduces a frame type identification mechanism that acts as an intermediary, directing appropriate frames to appropriate processing units. This mediator ensures that reference frames go to hardware for accuracy while non-reference frames go to neural processing for efficiency, optimizing hardware utilization.
Data Source
AI summary
Methods, systems, and devices for improved video throughput using deep learning video coding are described. A device may receive a bitstream including a set of video frames. The device may batch the set of video frames into a first subset of video frames and a second subset of video frames based on a change in a reference scene associated with the set of video frames. The device may select a mode of operation for a neural processing unit of the device based on the batching. The device may generate a set of video packets including the first subset of video frames, the second subset of video frames, or both, based on the neural processing unit and the selected mode of operation.