Personalized Perceptual Video Encoding with Neural QP Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video transmission technologies for mission-critical applications like tele-operated driving struggle to efficiently reduce bitrate without compromising visual quality, as they often rely on generic QoS policies and mean opinion scores, which may not cater to the diverse needs of professional users.
Innovation Solution
Implementing a perceptual-quality-optimized rate control system that uses a convolutional neural network to determine a quantization parameter (QP) for each frame, leveraging per-user opinion models and supervised training to maintain consistent perceptual quality, thereby optimizing encoding for individual user preferences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a fixed high bitrate is used to guarantee sufficient visual quality for all operators in all conditions, then visual quality is maintained, but bandwidth is wasted when the codec is not stressed
Solution Approach 1:
The patent implements dynamic bitrate adaptation by replacing fixed bitrate transmission with variable bitrate control. The system continuously adjusts the quantization parameter (QP) based on real-time scene complexity, motion activity, and user preferences, allowing the bitrate to dynamically match actual transmission needs rather than maintaining a constant high bitrate regardless of conditions.
Solution Approach 2:
The patent changes the key parameter of quantization parameter (QP) to control bitrate. By adjusting QP values frame-by-frame or sequence-by-sequence based on scene characteristics and user models, the system achieves both quality guarantee and bandwidth efficiency. The QP is not fixed but adapts to changing transmission conditions and user-specific quality requirements.
2Productivity
If per-user opinion models are used to optimize encoding for individual preferences, then encoding efficiency is improved, but system complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-training user opinion models and storing user preference profiles before actual encoding occurs. The system includes a user database that stores pre-computed QP recommendations and preference parameters. During real-time encoding, the system simply retrieves the appropriate user model and applies pre-computed guidance, avoiding the need for complex real-time user preference analysis.
Solution Approach 2:
The patent uses copying by creating a database of pre-trained user models that can be retrieved and applied to different users. Instead of building a completely new opinion model from scratch for each user, the system copies and adapts existing user profiles and preference patterns, significantly reducing computational complexity while maintaining personalization benefits.
3Loss of time
If frame-by-frame encoding with neural networks is used to achieve lowest latency, then latency is reduced, but computational requirements increase
Solution Approach 1:
The patent applies partial action by performing neural network-based quality estimation only for key frames or selected frames rather than all frames. The system uses QP cascading and predictive models to estimate quality for non-key frames based on neighboring frames, reducing the number of expensive neural network computations while maintaining overall quality assessment accuracy and enabling real-time processing.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
There are provided apparatuses, methods and computer program products. In accordance with an embodiment, there is disclosed a method comprising obtaining, in an encoder, frames of video data for transmission; providing the frames of the video data for rate control. The rate control comprises obtaining a target value for at least one of an opinion score and a personalized opinion score; obtaining a normalization quantification parameter; using the normalization quantification parameter to encode or decode a frame of the video data, and calculate a distortion metric; executing a convolutional neural network to extract at least one feature regarding the frame of the video data, and using a dense neural network and a regressor to determine a value for a quantification parameter to achieve the target value when compressing the frame of video data. The dense neural network is trained using the extracted features, the different quantification parameters with their associated processed distortion metrics, and the target quantification parameter that relates to a target opinion score as inputs; and uses the extracted features, the quantification parameter and the processed distortion metric as inputs to determine a value for a quantification parameter to achieve the target opinion score value when compressing the video data The frame is encoded with the quantification parameter to generate an encoded frame.