Cloud Video Streaming Using Early Motion Vectors and Residual Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video streaming systems suffer from high latency, particularly in cloud computing arrangements, leading to input errors and performance issues due to delays in processing and transmission.
Innovation Solution
The system transmits motion vectors generated during a geometry pass before the completion of the rendering process, allowing early motion compensation at the client device, and sends residual data to update the motion-compensated image, reducing latency by utilizing existing codecs and potentially employing machine learning for infilling under poor network conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If cloud computing arrangements are used to stream video content, then processing power and hardware capabilities are enhanced, but latency increases causing input errors and performance issues
Solution Approach 1:
The video stream is divided into two separate data types: motion vectors and residual data. Motion vectors are transmitted first and decoded independently, while residual data is transmitted subsequently. This segmentation allows the client device to perform motion compensation using only the motion vectors received so far, reducing latency without sacrificing processing power.
Solution Approach 2:
Motion vectors are extracted and transmitted before the complete video frame is rendered and transmitted. This preliminary extraction and transmission of motion information allows the client device to start decoding and displaying video content earlier, reducing latency while still benefiting from cloud-based processing power when residual data arrives.
2Loss of time
If motion vectors are transmitted before rendering completion, then latency is reduced, but data transmission complexity increases
Solution Approach 1:
The patent introduces an intermediary data structure that separates motion vector information from residual pixel data. This intermediary approach allows independent transmission and decoding of motion vectors, simplifying the client device's processing by enabling motion compensation before complete frame reception, thus reducing latency without excessive complexity.
3Manufacturing precision
If residual data is transmitted to update motion-compensated image, then video quality is maintained, but transmission time increases
Solution Approach 1:
The patent extracts motion vector information from the complete video frame data and transmits it separately before the residual data. This extraction allows the client device to decode and display motion-compensated video content sooner, reducing transmission time while maintaining video quality through subsequent residual data updates.
Data Source
Figure 1~2
Figure 3
Figure 4~5
AI summary
A system comprising a server configured to stream video content comprising a plurality of rendered image frames to a client device, the server comprising a geometry processing unit configured to perform a geometry pass on a scene being rendered to generate geometric information, wherein the geometry processing unit is configured to output one or more motion vectors, a transmitting unit configured to transmit the one or more motion vectors to the client device, a lighting processing unit configured to perform a lighting pass on the scene being rendered in dependence upon the generated geometric information, and a residual calculation unit configured to generate residual information indicative of a difference between an image frame rendered on the basis of the geometry pass and the lighting pass, and a preceding rendered image frame in the video content which has been motion compensated by applying the one or more motion vectors to the preceding rendered image frame, wherein the transmitting unit is configured to transmit the residual information to the client device.