Patch-Based Video Coding for Machine-Learning Regions of Interest
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video compression/decompression systems for machine learning applications are inefficient in coding only regions of interest, leading to high bitrate and pixel rate requirements, and lack flexibility to adapt to evolving machine learning tasks.
Innovation Solution
Detect and encode rectangular or suitable-shaped regions of interest into video atlases, using metadata to map their positions and sizes, and encode these atlases with video coders like AVC, HEVC, or VVC, without encoding the full frame at the same resolution, allowing for flexible and efficient coding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional video codecs encode the full frame at high resolution, then video quality is maintained, but bitrate and pixel rate requirements become excessively high
Solution Approach 1:
The video frame is segmented into multiple patches or regions of interest. Only these specific patches are encoded and transmitted at high resolution, while the rest of the frame is either not encoded or encoded at lower resolution. This segmentation allows the system to maintain video quality in important regions while significantly reducing the overall bitrate requirement.
Solution Approach 2:
Different regions of the video frame are assigned different quality levels based on their importance. Regions containing objects of interest are encoded at high resolution to maintain video quality, while less important regions are encoded at lower resolution or omitted entirely. This local quality approach optimizes the balance between video quality and bitrate efficiency.
2Measurement precision
If traditional video codecs encode the full frame at high resolution, then video quality is maintained, but processing complexity and computational resources increase
Solution Approach 1:
The video processing pipeline is segmented to identify and process only the important patches or regions of interest. By dividing the frame into manageable patches and selectively encoding only those that contain significant content, the processing complexity is reduced while maintaining video quality in the important regions.
Solution Approach 2:
Instead of processing the entire frame at high resolution, the system performs partial action by encoding only the necessary patches or regions of interest at high resolution. This partial processing approach significantly reduces computational resources and processing complexity while still maintaining the required video quality for machine learning tasks.
3Ease of manufacture
If traditional video codecs encode all regions uniformly, then encoding is simple, but flexibility to adapt to evolving machine learning tasks is reduced
Solution Approach 1:
The encoding approach is made dynamic by allowing the selection and configuration of patches or regions of interest based on the specific machine learning task requirements. The system can adaptively adjust which regions are encoded, their resolution, and their importance weights according to evolving task needs, providing flexibility while maintaining a relatively simple encoding framework.
Solution Approach 2:
The patch-based encoding system is designed to be universal and adaptable to multiple machine learning tasks. By encoding regions of interest in a standardized format with metadata indicating their importance and location, the system can serve various machine learning applications (object detection, tracking, recognition, etc.) without requiring task-specific encoding modifications, thus achieving both simplicity and adaptability.
Data Source
AI summary
Devices and techniques related to implementing patch based video coding for machines are discussed. Such patch based video coding includes detecting regions of interest in a frame of video, extracting the detected regions of interest to one or more atlases absent the frame at a resolution not less than the resolution of the regions of interest, and encoding the one or more atlases to a bitstream.