GPU Point Sprites for Video Decoding Inverse Transform
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Lower performance CPUs in devices like smartphones and tablets struggle to decode media efficiently, especially when handling other processing tasks, and existing GPU integration methods fail to fully leverage high-performance graphics technologies without requiring additional plug-ins or applications.
Innovation Solution
The use of point sprites on a GPU for dequantizing and applying inverse transforms to transform coefficients, offloading these processes from the CPU and utilizing browser-native program code to perform media decoding without the need for additional plugins, thereby enhancing decoding performance and reducing bandwidth requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If CPU is used for media decoding, then decoding can be performed, but decoding performance is insufficient on low-performance devices and CPU load increases
Solution Approach 1:
The patent introduces GPU as an intermediary component between the encoded media data and the final decoded output. The CPU prepares transform coefficients and transfers them to the GPU, which then performs the computationally intensive inverse transform and dequantization operations, effectively mediating the decoding process to distribute workload according to each processor's strengths
Solution Approach 2:
The patent replaces the CPU's mechanical processing of inverse transform and dequantization with the GPU's parallel processing architecture. By substituting the CPU's sequential processing mechanism with the GPU's massively parallel processing capability, the system achieves significantly higher decoding performance while reducing CPU load
2Productivity
If GPU is used for decoding with traditional approaches, then some decoding load is offloaded, but high-performance graphics technologies are not fully utilized and additional plug-ins are required
Solution Approach 1:
The patent makes the GPU serve multiple functions: it performs both traditional graphics rendering operations and media decoding operations (inverse transform and dequantization). By programming the GPU to handle both graphics and decoding tasks using standard GPU instruction sets, the system eliminates the need for separate plug-ins or applications, as the GPU's universal processing capability covers both functionality
Solution Approach 2:
The patent changes the operational parameters of the GPU by utilizing its native parallel processing architecture and standard graphics instruction sets for decoding operations, rather than forcing CPU-style sequential processing. This parameter change allows the GPU to operate at peak efficiency using its designed strengths, achieving full utilization of high-performance graphics technologies
3Productivity
If all transform coefficients are transferred to GPU, then complete decoding can be performed, but bandwidth requirements increase
Solution Approach 1:
The patent extracts and removes zero-valued transform coefficients from the data transfer stream before sending data to the GPU. By identifying and eliminating these redundant zero values, the system reduces the quantity of data that needs to be transferred over the bus between CPU and GPU memory, thereby reducing bandwidth requirements without affecting decoding quality
Solution Approach 2:
The patent performs preliminary processing on the transform coefficients before transfer to the GPU, specifically identifying and marking zero-valued coefficients for omission. This preliminary action prepares the data in an optimized format that reduces transfer bandwidth requirements while ensuring the GPU receives only the necessary non-zero coefficients for accurate reconstruction
Data Source
AI summary
A GPU loads point sprites that represent coded blocks of transform coefficients of one or more frames encoded in a bitstream and loads a transform kernel as a transform kernel texture. The GPU constructs an output frame using an inverse transform on the coded blocks of transform coefficients by transforming the point sprites with the transform kernel texture and by optionally dequantizing the point sprites. A single render pass may be used in which the rasterization formula performs the inverse transform and optionally dequantization. To preserve bandwidth, a CPU may refrain from sending the GPU at least some zero valued transform coefficients for the point sprites. Also, to reduce processing, the transform coefficients can remain in a zig-zag arrangement. The transform kernel texture used in the decoding can correspond to a modified version of the basis matrices used to encode the frame, which compensates for the zig-zag arrangement.