Neural Network Video Encoding Tile Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video encoding and decoding technologies face challenges in efficiently processing video data, particularly in terms of memory footprint and processing speed, especially when using neural networks for encoding and decoding.
Innovation Solution
The proposed solution involves processing an input tensor representing picture data by dividing it into multiple tiles in spatial dimensions, allowing for independent or parallel processing of these tiles within multiple pipelines. This approach reduces memory requirements and improves processing performance without increasing computational complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If video data is compressed using neural networks, then compression ratio is improved, but memory footprint increases
Solution Approach 1:
The patent divides the video picture into multiple tiles in spatial dimensions, allowing each tile to be processed independently by separate neural network pipelines. This segmentation reduces the memory footprint by enabling parallel processing of smaller regions rather than requiring all memory resources to be available simultaneously for the entire picture, while still achieving improved compression ratio through neural network-based processing.
2Manufacturing precision
If video data is processed with high computational complexity, then processing quality is improved, but processing speed decreases
Solution Approach 1:
By dividing the picture into multiple tiles that can be processed in parallel, the patent enables higher processing quality through neural networks while maintaining processing speed. The parallel processing of segmented tiles reduces the time penalty associated with complex neural network operations, as multiple tiles are processed simultaneously rather than sequentially.
Solution Approach 2:
The patent introduces dynamic tile sizing where different tiles can have different sizes based on content characteristics and processing requirements. This allows the system to adaptively allocate computational resources, processing complex regions with higher detail and simpler regions with lower detail, thereby optimizing the balance between processing quality and speed.
3Ease of operation
If uniform tile sizes are used, then processing simplicity is maintained, but processing efficiency decreases
Solution Approach 1:
The patent implements different tile sizes for different regions of the picture based on local content characteristics. Important regions with high visual information are allocated larger tiles for more detailed processing, while less important regions use smaller tiles. This local quality approach improves processing efficiency by matching computational resources to actual content needs, while maintaining relative simplicity through rule-based tile size determination.
Data Source
AI summary
The present disclosure relates to picture encoding and decoding of image regions on tile-basis. In particular, multiple components of an input tensor including a first and second component in spatial dimensions is processed within multiple pipelines. The processing of the first component includes dividing the first component in the spatial dimensions into a first plurality of tiles. Likewise, the processing of the second component includes dividing the second component in the spatial dimensions into a second plurality of tiles. The respective first and second plurality of tiles are then processed each separately. Among the first and second plurality of tiles there are at least two respective collocated tiles differing in size. In case of compression, the processing of the first and/or second component includes picture encoding, rate distortion optimization quantization, and picture filtering. In case of decompression, the processing includes picture decoding and picture filtering.