Game-Generated Motion Vectors for Low-Latency Remote Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding methods in real-time video streaming, particularly in remote gaming, are inefficient due to high latency caused by computationally intensive block-based motion estimation, which can be improved by integrating the encoding process with the video rendering process to leverage additional data sources like motion vectors generated by 3D graphics engines.
Innovation Solution
Integrating a 3D graphics engine with a video codec engine to provide final rendered frames and properly formatted motion vector data, allowing the codec engine to skip motion estimation, converting per-pixel motion vectors to per-block motion vectors, and injecting these into the encoding process concurrently with chroma subsampled video frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If block-based motion estimation is used in existing video coding methods, then video encoding quality is improved, but encoding time and latency increase significantly
Solution Approach 1:
The graphics engine pre-calculates per-pixel motion vectors during the rendering process before encoding begins. This preliminary computation of motion data eliminates the need for time-consuming block-based motion estimation during encoding, thereby reducing encoding time and latency while maintaining accurate motion representation for high-quality encoding
Solution Approach 2:
The patent extracts motion vector information directly from the graphics engine's rendering output, separating the motion estimation function from the encoding process. By taking out the motion vector generation from the traditional encoding pipeline and obtaining it from the graphics engine instead, the system avoids computationally intensive block-based motion estimation while preserving encoding quality
2Measurement precision
If block-based motion estimation is used, then motion accuracy is improved, but computational power requirements increase
Solution Approach 1:
The graphics engine serves its own motion estimation needs by generating per-pixel motion vectors as a byproduct of its normal rendering operations. This self-service approach allows the system to obtain accurate motion data without requiring separate computational resources for motion estimation, thereby reducing overall computational power requirements while maintaining high motion accuracy
Solution Approach 2:
The patent merges the motion vector generation process with the graphics rendering process. By combining these two functions, the system obtains accurate per-pixel motion vectors from the graphics engine without requiring separate block-based motion estimation computations, thereby reducing computational power requirements while maintaining motion accuracy
3Adaptability or versatility
If video encoding and graphics rendering operate separately, then system modularity is improved, but encoding efficiency decreases
Solution Approach 1:
The patent introduces motion vector data as an intermediary between the graphics engine and video encoder. This intermediary data format allows the two separate systems to communicate efficiently, enabling the encoder to utilize pre-computed motion information from the graphics engine without requiring tight integration, thus maintaining modularity while dramatically improving encoding efficiency
Data Source
AI summary
Systems and methods for integrated graphics rendering are disclosed. In certain embodiments, the systems and methods utilize a graphics engine, a video encoding engine, and remote client coding engine to render graphics over a network. The systems and methods involve the generation of per-pixel motion vectors, which are converted to per-block motion vectors at the graphics engine. The graphics engine injects these per-block motion vectors into a video encoding engine, such that the video encoding engine may convert those vectors into encoded video data for transmission to the remote client coding engine.