Predicted Video Frame Playback for Low-Latency Stutter Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Variable network transmission times in video and audio streaming systems cause stutter due to irregular frame arrivals, which is particularly problematic in low-latency scenarios like cloud game streaming, where buffering is not practical.
Innovation Solution
Client devices decode received frames and extrapolate predicted frames using motion vectors, applying post-processing to either decoded or predicted frames based on arrival delays, and discarding unnecessary frames to maintain smooth motion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If buffering is used to accumulate frames before presentation, then video stutter is eliminated, but latency increases
Solution Approach 1:
The system performs preliminary actions by pre-decoding frames ahead of their scheduled presentation time and storing them in a frame buffer. This allows frames to be ready for immediate presentation without buffering delays during playback, as the decoding work is completed in advance during periods when network conditions permit
Solution Approach 2:
The video stream is segmented into individual frames that are processed independently. Each frame is decoded, timestamped, and buffered separately, allowing the system to manage frame presentation individually without requiring buffering of entire video sequences, thus reducing overall latency while preventing stutter
2Loss of time
If frames are presented at regular intervals without buffering, then latency is reduced, but video stutter occurs due to irregular frame arrivals
Solution Approach 1:
A frame buffer acts as an intermediary between the network receiver and the video presenter. It absorbs the irregularities in frame arrival times by storing frames temporarily and releasing them at regular presentation intervals, preventing stutter while maintaining low latency through selective buffering based on timestamps
Solution Approach 2:
The system dynamically adjusts frame presentation timing based on actual frame arrival patterns and network conditions. Frames are presented at regular intervals when possible, but the system adapts by using extrapolated frames or adjusting timing when network variability causes irregular arrivals, maintaining smooth playback without excessive buffering
3Stability of the object's composition
If frame repeats and skips are used to handle late frames, then frame presentation continuity is maintained, but video quality degrades due to visible freeze
Solution Approach 1:
Instead of repeating late frames or skipping frames, the system creates copies of previously decoded frames and stores them in the frame buffer. When frames arrive late, these pre-existing frame copies can be used for presentation, avoiding the visible freeze effects of frame repetition while maintaining continuity
Solution Approach 2:
The system performs preliminary decoding and buffering of frames in advance. When a frame arrives late, previously decoded frame copies are already available in the buffer for immediate presentation, eliminating the need for frame repeats or skips and maintaining both continuity and quality
Data Source
AI summary
A client device receives a stream of encoded frames and decodes the frames to generate decoded frames. The frame decoding is performed in parallel with extrapolating predicted frames from the decoded frames. The client device uses the decoded frames to generate output frames and discards the predicted frames. In response to detecting an arrival delay with respect to an encoded frame, the client device uses the predicted frames to generate output frames.