Adaptive Frame Rate Control for Low Latency Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current game and application streaming technologies suffer from high latency due to limitations in the software stack, which often require lowering resolutions and frame rates to mitigate pipeline delays, resulting in a less responsive user experience.
Innovation Solution
A system and method that dynamically adjust frame rates and resolutions to achieve high frame rates (up to 240 FPS) while maintaining an acceptable balance with streaming bandwidth, using a server to execute graphics applications and transmit frames to a client device, which actively monitors and controls the video stream to reduce latency without increasing bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If the frame rate is increased to reduce latency, then the responsiveness to user input is improved, but the streaming bandwidth requirement increases
Solution Approach 1:
The system dynamically adjusts the frame rate based on real-time network conditions and user input frequency. The frame rate is not fixed but varies adaptively, increasing when latency reduction is needed and decreasing when bandwidth is constrained, thereby resolving the contradiction between responsiveness and bandwidth consumption
Solution Approach 2:
The patent changes the temporal parameter (frame rate) of the video stream to optimize responsiveness. By adjusting this parameter dynamically rather than maintaining a constant high frame rate, the system achieves low latency when needed while conserving bandwidth during periods of lower interaction, thus resolving the technical contradiction
2Speed
If the resolution is decreased to reduce processing time, then the hardware processing speed is improved, but the video quality deteriorates
Solution Approach 1:
The system dynamically adjusts resolution based on the actual need for responsiveness versus quality. When latency reduction is the priority, resolution may be temporarily decreased to enable faster processing. When quality is more important or network conditions allow, resolution is increased, creating a dynamic balance between processing speed and video quality
Solution Approach 2:
The patent applies different quality levels to different regions or time periods of the video stream. Rather than maintaining uniform high resolution throughout, the system selectively adjusts quality in response to specific conditions such as user input events or network bandwidth availability, thereby achieving fast processing when needed while preserving quality where and when possible
3Stability of the object's composition
If v-sync is enabled to synchronize with display refresh rate, then the display stability is improved, but the frame rate is capped and in-game latency increases
Solution Approach 1:
The system dynamically switches between v-sync enabled and disabled states based on current performance requirements. When display stability is sufficient and responsiveness is prioritized, v-sync is disabled to allow higher frame rates. When stability becomes an issue, v-sync is re-enabled, creating a dynamic adaptation that resolves the contradiction between stability and latency
Solution Approach 2:
The patent anticipates the latency penalty of v-sync by proactively adjusting other parameters or pre-processing frames to compensate. Rather than simply accepting the latency when v-sync is enabled, the system takes preliminary actions to offset the delay, such as optimizing frame generation timing or using predictive techniques, thereby mitigating the harmful effect while maintaining stability
Data Source
AI summary
Embodiments of the present invention provide a low-latency approach for local or remote application streaming that reaches high FPS targets without overloading the available streaming bandwidth, for example, by limiting the bit rate to the same value that is used by traditional 60 FPS streaming solutions. A client device and server device cooperate to actively monitor and control a video stream to maintain an acceptable balance between latency and video quality by adjusting the frequency or resolution when necessary to improve the streaming experience. When the server device captures and transmits frames at a higher rate, the software stack executing on the client device is able to display frames with less delay, even on a display device limited to 60 Hz, thereby achieving additional latency reduction.