Video Streaming Virtual Resolution Scaling Without Intra-Frame Stutter
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional video streaming systems experience latency and visual stutter when changing the resolution of a streamed game due to the insertion of larger intra-coded frames, which disrupt the decoding process and impact user experience.
Innovation Solution
Implement virtual resolution adjustment by scaling the effective resolution of video data within a larger frame while padding with a padding region, allowing predictive encoding without inserting intra-coded frames, and adjusting resolution based on network conditions and content complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If the resolution of a streamed video is reduced to control bitrate and network resources, then network resource usage is reduced, but latency and visual stutter increase due to insertion of intra-coded frames
Solution Approach 1:
The video frame is segmented into a valid picture region and a padding region. The scaling operation is applied only to the valid picture region, while the padding region maintains the original resolution dimensions. This segmentation allows the system to reduce the effective resolution for encoding purposes while preserving the original frame size structure, avoiding the need for intra-coded frames and maintaining smooth playback without latency spikes
Solution Approach 2:
A padding region is introduced as an intermediary element between the scaled valid picture region and the original frame structure. This padding region acts as a buffer that maintains the original resolution dimensions while allowing the valid picture region to be encoded at a lower resolution. The intermediary padding region enables resolution scaling without disrupting the video decoding process or requiring intra-coded frames
2Adaptability or versatility
If intra-coded frames are inserted to change resolution, then resolution adaptation is achieved, but visual quality deteriorates due to quantization losses and compression artifacts
Solution Approach 1:
The system dynamically adjusts the scaling resolution of the valid picture region based on network conditions while maintaining the original frame dimensions through padding. This dynamic approach allows continuous resolution adaptation without discrete intra-coded frames, as the scaling can be applied progressively to inter-coded frames. The dynamic scaling maintains visual quality by avoiding the quantization losses associated with intra-coded frames while still adapting to changing network conditions
3Loss of energy
If the resolution of streamed video is changed, then network bandwidth requirements are reduced, but decoding process is disrupted causing visual stutter
Solution Approach 1:
The padding region is preliminarily established in the frame structure before scaling operations are applied. By pre-defining the padding region boundaries and characteristics, the system prepares the frame structure to accommodate resolution scaling without disrupting the decoding process. This preliminary action ensures that subsequent scaling operations can be performed on the valid picture region while maintaining decoding stability and avoiding visual stutter
Data Source
AI summary
In various examples, network conditions associated with a video stream are observed over time to determine a virtual scaling factor that may be applied to the images of one or more frames of the video stream to generate scaled images that may be appended with a padding region that maintains the original resolution of the video and/or video stream such that a receiving device may crop the padding region from the received video stream during the decoding process without restarting or including additional intra-coded frames.