On-the-Fly Video Encoding Pipeline for Ultra-Low Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional video processing systems suffer from high latency due to the frame-based pipeline structure, which delays encoding and decoding processes, making them unsuitable for ultra-low latency applications.
Innovation Solution
Implementing a video transmitting and receiving system with a frame segment-based pipeline structure that allows on-the-fly encoding and decoding, where pixel data is processed in portions, enabling early output of encoded bitstreams and display of decoded frames before the entire frame is fully received or generated.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If a frame-based pipeline structure is used, then the video processing is simple and stable, but the latency is long and cannot meet ultra-low latency requirements
Solution Approach 1:
The patent divides a complete video frame into multiple frame segments (e.g., first frame segment, second frame segment, etc.). Each frame segment can be independently encoded, transmitted, decoded, and displayed. This segmentation allows the system to process and display partial frame data before the entire frame is received, significantly reducing latency while maintaining manageable processing complexity through modular operations.
2Loss of time
If the video encoder waits for all pixel data to be received before encoding, then the encoding quality is ensured, but the encoding latency increases
Solution Approach 1:
The video encoder performs preliminary encoding actions on received frame segments before all pixel data for the complete frame is received. The encoder can start encoding the first frame segment as soon as its data is available, rather than waiting for the entire frame. This preliminary action reduces encoding latency while maintaining encoding quality through progressive refinement as more segments are received and encoded.
3Loss of time
If the transmitting circuit waits for complete encoded data before output, then the data integrity is ensured, but the transmission latency increases
Solution Approach 1:
The encoded video data is segmented into multiple NAL (Network Abstraction Layer) streams corresponding to different frame segments. The transmitting circuit can output these NAL streams progressively as they are encoded, rather than waiting for complete frame encoding. Each NAL stream carries valid video data that can be independently decoded and displayed, ensuring data integrity while reducing transmission latency through progressive transmission.
4Loss of time
If the video decoder waits for complete bitstream before decoding, then the decoding accuracy is ensured, but the decoding latency increases
Solution Approach 1:
The video bitstream is segmented into multiple NAL streams, each corresponding to a frame segment. The video decoder can decode these NAL streams progressively as they are received from the transmitting circuit, rather than waiting for the complete bitstream. Each decoded frame segment provides accurate video data that can be displayed immediately, reducing decoding latency while maintaining decoding accuracy through independent segment decoding.
5Loss of time
If the display circuit waits for complete decoded data before display, then the display quality is ensured, but the display latency increases
Solution Approach 1:
The decoded video data is organized into frame segments that can be independently displayed. The display circuit can render and display decoded frame segments as they become available, rather than waiting for complete frame decoding. This segmentation allows progressive display of video content with reduced latency while maintaining display quality through proper handling of partial frame data and synchronization.
Data Source
AI summary
A video transmitting system includes a source buffer, a video encoder, a bitstream buffer, and a transmitting circuit. The source buffer receives pixel data of pixels of a video frame. The video encoder retrieve pixel data of a portion of the pixels of the video frame from the source buffer, and starts encoding the pixel data of the portion of the pixels before pixel data of a last pixel of the video frame is received by the source buffer. The bitstream buffer receives a network abstraction layer (NAL) stream from the video encoder, wherein the NAL stream is generated by encoding the pixel data of the portion of the pixels. The transmitting circuit retrieves the NAL stream from the bitstream buffer, and starts outputting the NAL stream before the pixel data of the last pixel of the video frame is encoded by the video encoder.