Real-Time Synthetic Video Streaming With GPU-Memory Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing real-time synthetic media generation systems face challenges in achieving low latency and high quality video transmission without significant resource usage, particularly in two-way interactions, due to limitations in encoding techniques and hardware encoders like NVenc API.
Innovation Solution
A system architecture that utilizes GPU memory for direct output to encoding, avoids hard-disk storage, employs P frames for low motion video, and sends IDR frames only under specific conditions, along with client-side post-processing, to optimize video quality and reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If hardware encoders like NVenc API are used for real-time encoding, then encoding speed is improved, but video quality deteriorates
Solution Approach 1:
The patent segments the video encoding process into two distinct types of frames: IDR frames (independent refresh frames) and P frames (predictive frames). This segmentation allows different encoding strategies to be applied to different frame types, optimizing both quality and speed. IDR frames use higher quality encoding when needed, while P frames use faster encoding for low-motion content, resolving the contradiction between speed and quality.
Solution Approach 2:
The system dynamically switches between IDR and P frame encoding modes based on motion detection and content analysis. The encoder adapts its behavior in real-time, using hardware encoders for P frames (speed-optimized) and software encoders for IDR frames (quality-optimized), allowing the system to achieve both high speed and high quality depending on the specific content requirements.
2Manufacturing precision
If software encoders are used for high quality encoding, then video quality is improved, but encoding speed deteriorates
Solution Approach 1:
By segmenting frames into IDR and P types, the system can selectively apply software encoding only when necessary (for IDR frames at key moments) while using hardware encoding for the majority of P frames, thus maintaining high quality where needed while achieving high speed for most content.
Solution Approach 2:
Instead of always using high-quality software encoding, the system applies it partially only to IDR frames and only when motion thresholds are exceeded. This partial application of excessive encoding resources achieves the necessary quality without the performance penalty of continuously using software encoders.
3Manufacturing precision
If frequent IDR frames are sent to maintain quality, then video quality is improved, but bandwidth consumption and latency increase
Solution Approach 1:
The system uses periodic action by sending IDR frames only at specific intervals (when motion thresholds are exceeded or at key content transitions) rather than continuously. This periodic transmission of quality-optimized frames maintains video quality at critical moments while reducing overall bandwidth consumption and latency during low-motion periods.
Solution Approach 2:
The system creates a copy of the video stream with different encoding characteristics (P frames) that can be transmitted more efficiently. By having both IDR and P frame copies available, the system can select the appropriate copy based on current conditions, reducing bandwidth usage while maintaining quality where needed.
4Quantity of substance
If hard-disk storage is used for video buffering, then storage capacity is improved, but latency and resource usage increase
Solution Approach 1:
The patent replaces the mechanical hard-disk storage system with a memory-based buffering approach using GPU and CPU memory. This substitution eliminates the mechanical read/write operations and associated latency, allowing for faster data access and processing while reducing overall system resource usage and improving real-time performance.
Data Source
AI summary
In one embodiment, a method includes generating, by a memory of a GPU of a server computing device, synthetic video content for a real-time video stream and encoding, by a memory of a GPU of the server computing device, the synthetic video content. The method further includes multiplexing, by a memory of a CPU of the server computing device, the encoded synthetic video content; and transmitting the multiplexed encoded synthetic video content from the server computing device to a client computing device, without storing (1) the generated synthetic video content (2) the encoded synthetic video content and (3) the multiplexed encoded synthetic video content in a hard-disk memory of the server computing device during the method.