Multi-Task Video Feature Bitstreams for Real-Time Machine Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding technologies struggle to efficiently encode and decode video for both human and machine consumption, particularly in scenarios where real-time analysis and decision-making are required, such as in surveillance and IoT applications, due to the complexity of video quality, data usage, and computational demands of traditional neural networks.
Innovation Solution
A task-specific decoder architecture that includes a neural network backbone and head, capable of decoding video and extracting features for machine consumption, utilizing a bitstream architecture with separate substreams for video, features, and neural network descriptions, allowing for efficient encoding and decoding of video for both human and machine use.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If video is encoded using conventional video coding technologies, then video can be transmitted and decoded for human consumption, but processing speed and real-time analysis capability deteriorate due to the need to process large video data from multiple cameras
Solution Approach 1:
The patent segments the video processing task into two independent streams: a video bitstream for human consumption and a feature bitstream for machine analysis. The encoder separately extracts features using a neural network backbone and encodes them independently, allowing parallel processing and reducing the computational burden on real-time analysis systems.
Solution Approach 2:
The patent extracts relevant features from the video data using a neural network backbone before encoding. By taking out only the essential feature information rather than transmitting the complete video data, the system reduces the quantity of data that needs to be processed for machine analysis while maintaining processing speed.
2Productivity
If a single bitstream is used for both human and machine consumption, then transmission is simplified, but processing efficiency deteriorates because the same data must be processed for both purposes
Solution Approach 1:
The patent divides the bitstream into two separate streams: a video bitstream containing compressed video data for human consumption and a feature bitstream containing extracted feature representations for machine analysis. This segmentation allows each stream to be optimized for its specific purpose, improving processing efficiency for machine tasks while maintaining video quality for human viewing.
Solution Approach 2:
The patent creates a multi-functional system where the feature extraction neural network backbone serves multiple purposes: it extracts features for machine analysis, provides representations for compressed feature encoding, and enables efficient processing for both human and machine consumption through the dual bitstream architecture.
3Measurement precision
If feature extraction is performed using deep learning techniques, then machine analysis accuracy is improved, but processing time increases due to computational complexity
Solution Approach 1:
The patent performs feature extraction using a pre-trained neural network backbone in advance, before the actual analysis task is completed. By pre-extracting features and encoding them in the feature bitstream, the system reduces the computational time required during real-time analysis while maintaining high accuracy through the use of pre-trained deep learning models.
Data Source
AI summary
A system and method for video and feature coding of neural-network structures used for multi-task machine learning includes an encoder, decoder and a decoder-compliant bitstream. A task-specific video decoder includes a first decoder receiving a bitstream having at least one feature and a description of a neural network backbone used to generate the bitstream, and a task-specific neural network head. The neural network head recreates the neural network backbone from the description, receives a feature from the bitstream a generates a task-specific output.