Hybrid Video Encoding via Frame Type Segmentation

Solution Overview

Problem

Conventional platforms face inefficiencies in encoding media items due to the lack of consideration for frame types, leading to increased computing resource consumption and latency, as they either use hardware encoders for quick processing but low quality or software encoders for high quality but increased latency, without accurately predicting user access patterns.

Innovation Solution

A system that identifies and analyzes frame types of media items using metrics such as GOP structure and temporal dependency models to selectively choose between hardware, software, or hybrid encoders for each frame, optimizing encoding based on frame impact and resource constraints.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If hardware encoder is used for all frames, then encoding speed is improved, but encoding quality deteriorates

Engineering Contradiction:

Improveencoding speedVSAvoidencoding quality

Core Design Contradiction:

SpeedVSManufacturing precision

Solution Approach 1:

The patent applies local quality by differentiating encoding approaches based on frame types. Impactful frames (e.g., I-frames, keyframes) are encoded with software encoders to ensure high quality, while non-impactful frames (e.g., P-frames, B-frames) are encoded with hardware encoders for speed. This selective approach optimizes overall encoding quality without sacrificing unnecessary processing resources on frames that have minimal impact on perceived quality.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent segments the video stream into different frame categories based on their impact level. By identifying and classifying frames as either impactful or non-impactful, the system can apply different encoding strategies to each segment. This segmentation enables the platform to process frames differently according to their specific requirements, resolving the contradiction between speed and quality.

Inventive Principle:

Principle #1Segmentation

2Manufacturing precision

If software encoder is used for all frames, then encoding quality is improved, but latency increases

Engineering Contradiction:

Improveencoding qualityVSAvoidlatency

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent applies local quality by differentiating encoding approaches based on frame types. Impactful frames (e.g., I-frames, keyframes) are encoded with software encoders to ensure high quality, while non-impactful frames (e.g., P-frames, B-frames) are encoded with hardware encoders for speed. This selective approach optimizes overall encoding quality without sacrificing unnecessary processing resources on frames that have minimal impact on perceived quality.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent applies partial action by using software encoding only for impactful frames rather than all frames. This partial application of the more resource-intensive software encoding approach achieves the necessary quality improvement where it matters most, while avoiding the excessive latency that would result from applying software encoding to every frame in the video stream.

Inventive Principle:

Principle #16Partial or excessive action

3Device complexity

If uniform encoding approach is used for all frames, then device complexity is reduced, but resource consumption increases

Engineering Contradiction:

Improveencoding process complexityVSAvoidcomputing resource consumption

Core Design Contradiction:

Device complexityVSUse of energy by moving object

Solution Approach 1:

The patent segments the video stream into different frame categories based on their impact level. By identifying and classifying frames as either impactful or non-impactful, the system can apply different encoding strategies to each segment. This segmentation enables the platform to process frames differently according to their specific requirements, resolving the contradiction between speed and quality.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes the encoding parameters dynamically based on frame characteristics. By adjusting which encoder type is applied to each frame based on its impact level, the system optimizes resource consumption without requiring complex manual configuration. The encoding parameters (hardware vs. software) are automatically adapted to match the content requirements of each frame.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS20240214582A1Co-optimization of hardware-based encoding and software-based encoding

Publication Date: 2024.06.27 GOOGLE LLC

AI summary

A media item to be provided to users of a platform is identified. The media item includes multiple frames. Metrics for the frames of the media item are obtained. A frame type associated with each of the frames is determined based on the metrics. The media item is determined, based on the frame type associated with each frames, to be encoded using two or more of a hardware encoder, a software encoder, or a hybrid encoder. The frames of the media item are encoded using the two or more of the hardware encoder, the software encoder, or the hybrid encoder in accordance with the determination.