Video Encoding Pipeline with Machine Learning Block

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing video processing technologies face challenges in optimizing encoding efficiency, decoding efficiency, and video quality due to variations in prediction samples caused by encoding parameters, leading to suboptimal resource usage and perceivable visual artifacts.

Innovation Solution

Incorporating a machine learning block, such as a convolutional neural network, into the video encoding and decoding pipelines to analyze image data and adaptively adjust encoding and decoding parameters based on content-dependent characteristics, improving the matching between prediction samples and source image data, and reducing visual artifacts through filter parameter adjustment.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If traditional prediction techniques are used to encode image data, then encoding process is simple, but encoding efficiency and video quality are suboptimal due to poor matching between prediction samples and source image data

Engineering Contradiction:

Improvematching accuracy between prediction samples and source image dataVSAvoidcomplexity of encoding pipeline

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

A machine learning block is introduced as an intermediary component between the traditional encoding pipeline and the prediction sample generation. This block analyzes image data characteristics and adaptively adjusts encoding parameters to improve prediction accuracy without completely redesigning the encoding system.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system dynamically changes encoding parameters based on content-dependent characteristics detected by the machine learning block. By adapting parameters such as prediction mode selection and transform settings to the specific image content, the system achieves better matching accuracy without fixed complexity.

Inventive Principle:

Principle #35Parameter changes

2Manufacturing precision

If encoding parameters are varied to improve video quality, then prediction sample matching improves, but resource usage increases

Engineering Contradiction:

Improvevideo qualityVSAvoidresource usage

Core Design Contradiction:

Manufacturing precisionVSUse of energy by moving object

Solution Approach 1:

The machine learning block selectively applies complex processing only to regions or frames where it will provide significant quality improvement. By identifying content-specific opportunities for enhancement, the system avoids unnecessary resource expenditure on already-optimized segments.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The system uses feedback from the machine learning block's analysis of image data characteristics to dynamically adjust encoding parameters. This closed-loop approach ensures that resource-intensive operations are only applied when they will actually improve video quality, optimizing the trade-off between quality and resource usage.

Inventive Principle:

Principle #23Feedback

3Productivity

If machine learning techniques are applied to improve encoding efficiency and video quality, then prediction sample matching improves, but visual artifacts increase

Engineering Contradiction:

Improveencoding efficiencyVSAvoidvisual artifacts

Core Design Contradiction:

ProductivityVSObject-generated harmful factors

Solution Approach 1:

The system dynamically adjusts encoding parameters based on content characteristics to optimize the balance between compression efficiency and artifact reduction. By adapting parameters such as quantization strength and prediction mode to the specific image content, the system maintains high encoding efficiency while minimizing visible artifacts.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11616960B2Machine learning video processing systems and methods

Publication Date: 2023.03.28 APPLE INC

AI summary

System and method for improving video encoding and/or video decoding. In embodiments, a video encoding pipeline includes a main encoding pipeline that compresses source image data corresponding with an image frame by processing the source image data based at least in part on encoding parameters to generate encoded image data. Additionally the video encoding pipeline includes a machine learning block communicatively coupled to the main encoding pipeline, in which the machine learning block analyzes content of the image frame by processing the source image data based at least in part on machine learning parameters implemented in the machine learning block when the machine learning block is enabled by the encoding parameters; and the video encoding pipeline adaptively adjusts the encoding parameters based at least in part on the content expected to be present in the image frame to facilitate improving encoding efficiency.