Hybrid Video Compression With NN Base Layer for Machine Vision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video compression standards struggle to optimize for both human and machine consumption, as neural-network-based computer vision algorithms are impacted by artifacts from classical codecs, and existing methods for machine tasks are computationally intensive and inflexible.
Innovation Solution
A hybrid framework using a neural-network-based base layer for machine tasks and a traditional enhancement layer for human viewing, where the base layer's reconstructed images serve as predictors for the enhancement layer, allowing for flexible computation and energy-efficient video compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If classical video compression standards are used, then bitrate efficiency is improved, but machine task performance deteriorates due to artifacts like ringing and blocking
Solution Approach 1:
The patent divides the video compression system into two distinct segments: a base layer optimized for machine tasks using neural networks, and an enhancement layer optimized for human viewing using traditional codecs. This segmentation allows each layer to be optimized for its specific purpose without compromise
Solution Approach 2:
The patent changes the optimization parameters from traditional signal fidelity metrics to machine task performance metrics. The base layer uses neural networks trained to preserve features important for machine tasks while allowing artifacts that are imperceptible to machines, fundamentally changing what constitutes 'quality' in the compression process
2Reliability
If neural-network-based methods are used for machine tasks, then machine task performance is improved, but computational complexity increases
Solution Approach 1:
The computational load is segmented between two layers: the base layer uses neural networks for machine tasks, while the enhancement layer uses traditional, computationally lighter codecs for human viewing. This allows the system to use heavy computational resources only where necessary for machine tasks
Solution Approach 2:
The patent applies neural networks partially - only in the base layer for machine task optimization - rather than throughout the entire compression pipeline. This partial application achieves the necessary machine task performance while avoiding excessive computational complexity in portions of the system where it provides no benefit
3Manufacturing precision
If traditional codecs are used, then human viewing quality is maintained, but machine task algorithms are impacted by artifacts
Solution Approach 1:
The patent creates separate optimization paths: the base layer prioritizes machine task performance with neural network-based compression that preserves task-relevant features, while the enhancement layer prioritizes human visual quality with traditional coding tools. Each layer serves its specific purpose without compromising the other
Solution Approach 2:
The patent applies different quality standards to different parts of the compressed video stream. The base layer maintains high quality for features important to machine tasks (local to machine relevance), while allowing lower quality in aspects important to human perception but less critical for machine tasks
Data Source
AI summary
In one implementation, we propose a scalable framework where a base layer uses NN-based methods to compress the content for computer vision machine tasks and enhancement layer(s) use traditional predictive coding for human viewing. Typically, the based layer performs NN-based analysis to generate a latent tensor, which is entropy coded to produce the base layer bitstream. By performing synthesis on the latent tensor, an inter-layer predictor can be obtained for the enhancement layer(s). Since many machine tasks are not required to be performed for each frame, the base layer may skip analysis for some frames. The synthesis may be performed at the base layer or the enhancement layer(s). In one example, the base layer compresses features optimized for a machine task and the enhancement layer(s) rely on predictive coding. In another example, the enhancement layer(s) can use traditional scalable video compression methods.