Latency Tracking Engine for Video Rendering Pipeline Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern gaming and streaming applications face significant latency issues due to varying frame rendering times, leading to visual artifacts like frame tears and stutters, which degrade the user experience, and are often misattributed to network delays when in fact they are dominated by processing and rendering delays on gaming servers and local computers.
Innovation Solution
The implementation of a latency tracking engine (LTE) that synchronizes the frame processing pipeline stages, eliminates the CPU instruction queue, and uses a variable refresh rate display to ensure frames are rendered and transmitted at a rate matching the display refresh rate, allowing immediate presentation of frames upon receipt, thereby reducing latency and eliminating the need for a de-jitter buffer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If frames are rendered at high rates to match display refresh rate, then user experience and immersion are improved, but latency increases due to processing and rendering delays
Solution Approach 1:
The system performs preliminary actions by pre-rendering frames at high rates and maintaining a buffer of ready-to-display frames. This allows the display to immediately show pre-rendered frames without waiting for current frame completion, thereby reducing perceived latency while maintaining high frame rates.
Solution Approach 2:
The patent segments the frame processing pipeline into distinct stages (CPU preparation, GPU rendering, encoding, transmission, decoding, display) and optimizes each independently. This segmentation allows parallel processing of multiple frames at different pipeline stages, reducing overall latency while maintaining high frame rates.
2Reliability
If frame rendering rate is increased to eliminate visual artifacts, then display quality is improved, but processing complexity and resource requirements increase
Solution Approach 1:
The system dynamically adjusts the frame rendering rate based on actual display refresh rate measurements and pipeline performance. Rather than fixed high-rate rendering, the system optimizes the frame rate in real-time to match display capabilities and processing capacity, reducing unnecessary processing complexity while maintaining display quality.
Solution Approach 2:
The patent changes key parameters including frame timing offsets, refresh rate synchronization points, and pipeline buffer sizes to optimize performance. By dynamically adjusting these parameters based on measured performance metrics, the system achieves high display quality without excessive processing complexity.
3Loss of time
If frames are transmitted immediately upon rendering to reduce latency, then response time is improved, but network synchronization issues and frame tears increase
Solution Approach 1:
The system implements feedback mechanisms that monitor display refresh timing, network transmission status, and pipeline performance. This feedback is used to dynamically adjust frame transmission timing, synchronization offsets, and buffer management to prevent frame tears and synchronization issues while maintaining low latency.
Solution Approach 2:
The patent employs periodic actions by synchronizing frame transmission with display refresh cycles and using periodic timing offsets to coordinate pipeline stages. This periodic synchronization ensures frames are transmitted at optimal intervals that match display capabilities, reducing tears while maintaining timing efficiency.
Data Source
AI summary
Disclosed are apparatuses, systems, and techniques that reduce latency of frame processing pipelines. The techniques include but are not limited to causing a display to set a refresh rate that matches a frame rendering rate of an application and rendering, with the frame rendering rate, a plurality of frames. Rendering frames includes generating, using a first processing unit, sets of instructions associated with respective frames and generated starting at times spaced with the refresh rate, processing, using a second processing unit, the sets of instructions to render the frames, and causing the display to display the rendered frames.