Video Codec Connectors for Human and Machine Vision Outputs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video compression methods optimized for human viewing fail to meet the accuracy requirements of machine tasks, leading to poor performance when used for tasks such as object detection and classification due to differing quality metrics between human and machine vision.
Innovation Solution
Introduce a connector at the decoder output to adapt decoded video content for secondary tasks by using filtering functions like average pooling and convolutional layers, optimizing the codec parameters for specific machine tasks while maintaining compatibility with human viewing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Illumination intensity
If video compression is optimized for human viewing quality, then visual quality is improved, but accuracy for machine tasks deteriorates
Solution Approach 1:
The patent segments the decoding process into two distinct paths: a first decoding path optimized for human viewing quality and a second decoding path optimized for machine task accuracy. This segmentation allows each path to use different processing parameters and algorithms tailored to its specific purpose, resolving the contradiction between visual quality and machine task performance
Solution Approach 2:
The patent creates a universal decoding system that can serve multiple functions: it can output decoded video optimized for human viewing, optimized for machine tasks, or both simultaneously. The multi-functional decoder includes multiple output interfaces that can be selectively activated based on the intended use case, allowing the same compression system to satisfy both human visual quality requirements and machine task accuracy requirements
2Productivity
If compression bitrate is reduced, then transmission efficiency is improved, but task accuracy deteriorates
Solution Approach 1:
The patent implements dynamic parameter adjustment in the second decoding path, where processing parameters such as filtering strength, block sizes, and algorithm selection are dynamically adapted based on the specific machine task requirements and available bitrate. This dynamic adjustment allows the system to maintain task accuracy across varying compression levels by optimizing the decoding process for each specific scenario
Solution Approach 2:
The patent changes key decoding parameters in the second path compared to the first path, including using different filter types, block partitioning schemes, and reconstruction algorithms. These parameter changes are specifically designed to preserve features important for machine tasks even at low bitrates, while the first path uses parameters optimized for human visual perception
3Productivity
If template matching prediction is used, then compression efficiency is improved, but compatibility with existing standards deteriorates
Solution Approach 1:
The patent introduces an intermediary processing layer between the standard compliant decoder and the template matching prediction algorithm. This intermediary layer performs format conversion and feature extraction that bridges the gap between standard video formats and the requirements of template matching, allowing the algorithm to be applied to standard-compliant decoded video without requiring changes to the underlying compression standard
Data Source
AI summary
A processing module, or connector, adapts an output of a codec, or a decoded output, to a form suitable for an alternate task. In one embodiment, the output of a codec is used for a machine task and the connector adapts this output to a form suitable for a video display. In another embodiment, metadata accompanies the codec output, which can instruct the connector how to adapt the codec output for an alternate task. In other embodiments, the processing module performs averaging over a N×M window, or convolution.


