Video Streaming with Early Motion Vectors for Lower Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video streaming systems suffer from high latency, particularly in cloud computing arrangements, leading to input errors and performance issues due to delays in processing and transmission.
Innovation Solution
The system transmits motion vectors generated during a geometry pass before the completion of the rendering process, allowing earlier decoding at the client device, and complements this with residual data to reduce latency. This approach uses motion vectors from the geometry pass instead of codec motion estimation, enabling earlier motion compensation and compatibility with existing codecs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If cloud computing arrangements are used to stream video content, then processing power and hardware capabilities are enhanced, but latency increases causing input errors and performance issues
Solution Approach 1:
The patent segments the video encoding process into two independent parts: motion vectors (transmitted early from geometry pass) and residual data (transmitted later). This segmentation allows the client to perform motion compensation using early-transmitted motion vectors, significantly reducing latency while maintaining the benefits of cloud processing.
Solution Approach 2:
The patent applies preliminary action by transmitting motion vectors before the rendering process is complete. The geometry pass generates motion vectors that are transmitted to the client device ahead of time, allowing the client to prepare for decoding and display without waiting for the full rendering pipeline to finish on the server.
2Measurement precision
If motion vectors are transmitted after rendering completion, then data accuracy is maintained, but decoding time is delayed increasing latency
Solution Approach 1:
Motion vectors are generated during the geometry pass (preliminary rendering stage) and transmitted to the client before the lighting pass completes. This preliminary transmission of motion data enables the client to start decoding and preparing frames earlier, reducing overall latency while maintaining data integrity through subsequent residual data transmission.
Solution Approach 2:
By separating motion vector transmission from residual data transmission, the system can transmit motion information earlier without compromising final image quality. The motion vectors provide sufficient accuracy for motion compensation on their own, with residual data completing the picture later, thus decoupling decoding time from rendering completion.
3Adaptability or versatility
If codec motion estimation is used, then compatibility with existing codecs is maintained, but latency cannot be reduced further
Solution Approach 1:
The patent introduces motion vectors from the geometry pass as an intermediary that bridges the gap between server rendering and client decoding. These geometry-pass motion vectors serve as a mediator that enables early motion compensation without requiring changes to existing video codecs, maintaining compatibility while reducing latency.
Solution Approach 2:
The system changes the parameter source from codec-based motion estimation to geometry-pass motion vectors. This parameter change allows motion information to be extracted earlier in the rendering pipeline, enabling latency reduction while the residual data transmission completes the picture, maintaining compatibility with existing codec frameworks.
Data Source
AI summary
A system comprising a server configured to stream video content comprising a plurality of rendered image frames to a client device and a geometry processing unit that performs a geometry pass on a rendering scene to generate geometric information. The geometry processing unit outputs one or more motion vectors. A transmitting unit transmits the one or more motion vectors to the client device. A lighting processing unit performs a lighting pass on the scene being rendered in dependence upon the generated geometric information. A residual calculation unit generates residual information of a difference between an image frame rendered on the basis of the geometry and the lighting passes and a preceding rendered image frame in the video content after being motion compensated by applying the one or more motion vectors to the preceding rendered image frame. The transmitting unit is configured to transmit the residual information to the client device.