Sub-Tile Grid Sampling for Low-Latency Neural Video Codecs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The real-time deployment of neural video codecs (NVCs) on hardware is hindered by computational and memory inefficiencies due to per-pixel grid sampling, leading to high latency and energy consumption, especially on GPU and NPU architectures, and block-based interpolation results in lower compression efficiency and blocking artifacts.
Innovation Solution
A sub-tile-based grid sampling architecture aggregates per-pixel operations into rectangular sub-tiles with homogeneous motion vectors, reducing latency and memory access inefficiencies by optimizing data transfers and reusing shared reference pixels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If per-pixel grid sampling is used in neural video codecs, then compression efficiency is improved, but latency and energy consumption increase significantly
Solution Approach 1:
The patent divides the video frame into multiple tiles, and each tile is further divided into sub-tiles. This segmentation allows the system to process smaller regions independently, reducing the overall latency while maintaining compression efficiency. The sub-tile structure enables parallel processing and optimizes memory access patterns.
Solution Approach 2:
The patent performs preliminary classification of pixels within sub-tiles to identify homogeneous regions before performing grid sampling. This preliminary action allows the system to optimize the sampling process by treating similar regions uniformly, reducing computational overhead and latency.
2Measurement precision
If per-pixel grid sampling is used in neural video codecs, then compression efficiency is improved, but energy consumption increases significantly
Solution Approach 1:
By segmenting the frame into tiles and sub-tiles, the patent reduces the total number of memory accesses required for grid sampling. This segmentation enables more efficient memory utilization and reduces energy consumption associated with data transfers.
Solution Approach 2:
The patent merges adjacent pixels with similar characteristics into the same sub-tile, allowing shared reference pixel usage. This merging reduces redundant computations and memory accesses, thereby lowering energy consumption while maintaining compression efficiency.
3Loss of time
If block-based interpolation is used instead of per-pixel grid sampling, then latency and energy consumption are reduced, but compression efficiency and video quality deteriorate
Solution Approach 1:
The patent introduces a hierarchical segmentation structure with tiles divided into sub-tiles, enabling a middle ground between block-based and per-pixel approaches. This segmentation allows for more precise processing than block-based methods while avoiding the full computational cost of pure per-pixel approaches.
Solution Approach 2:
The patent applies different processing strategies to different sub-tiles based on local characteristics. Homogeneous regions are processed more coarsely to reduce latency, while heterogeneous regions receive finer processing to maintain compression efficiency, achieving local optimization throughout the frame.
4Adaptability or versatility
If per-pixel grid sampling is implemented on GPU/NPU architectures, then neural video codec functionality is achieved, but computational inefficiency and high energy consumption occur
Solution Approach 1:
The patent segments the computation into tile-level and sub-tile-level operations that are better suited for GPU/NPU architectures. This segmentation enables more efficient utilization of parallel processing resources and reduces memory bandwidth requirements, improving computational efficiency.
Solution Approach 2:
The patent changes the organizational parameters of the data structure from a flat per-pixel approach to a hierarchical tile-sub-tile structure. This parameter change optimizes memory access patterns and computational workflows for hardware architectures, improving productivity while maintaining neural video codec functionality.
Data Source
AI summary
Real-time neural video codecs face significant latency and energy bottlenecks due to pixel-level grid sampling, which requires irregular, fine-grained memory accesses and limits efficient hardware acceleration. To address this, a sub-tile-based grid sampling technique is disclosed herein. The technique determines super tile sizes using motion vector gradients, neural network parameters, and available on-chip memory. A super tile is split into sub-tiles by detecting motion boundaries through motion vector analysis, where a sub-tile has homogeneous motion vectors. For each sub-tile, a reference bounding box is computed to enable efficient block transfers of reference data, and per-pixel metadata is generated for feature interpolation. The pipelined, parallelizable solution reduces number of memory accesses and computational overhead, compared to existing pixel-based techniques.