Auxiliary Frame Encoding for Multi-Sensor Video Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-resolution image data from multiple image sensors poses a challenge for encoders to maintain an acceptable frame rate, as it requires copying and synchronization of data between encoders, which is cumbersome and inefficient, especially when determining motion vectors.
Innovation Solution
The use of auxiliary frames, which are not intended for display but assist in encoding by providing complementary image data, allows encoders to operate independently and reduce or eliminate the need for data copying by using virtual pixel regions and spatially rearranging image data to minimize the need for inter-encoder data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data copying is performed between encoders for reference encoding, then motion vector search can be performed, but encoding complexity and synchronization requirements increase
Solution Approach 1:
The image frame is divided into first and second portions that are encoded by separate encoders. Each encoder operates independently on its portion without requiring data copying from the other encoder, eliminating synchronization complexity while still enabling motion vector search within each portion's reference frame
Solution Approach 2:
An auxiliary frame is introduced as an intermediary structure that stores reference data for both encoders. The auxiliary frame contains the first reference frame portion for the first encoder and the second reference frame portion for the second encoder, eliminating the need for direct data copying between encoders while providing necessary reference data for motion vector search
2Reliability
If the entire reference frame is copied between encoders, then complete motion vector search is enabled, but data transfer requirements and processing time increase
Solution Approach 1:
Each encoder only needs access to the relevant portion of the reference frame for motion vector search. The auxiliary frame provides localized reference data (first reference frame portion for first encoder, second reference frame portion for second encoder) rather than requiring the entire reference frame to be copied, reducing data transfer time while maintaining search completeness within each portion
3Productivity
If encoders operate independently without data copying, then encoding speed increases, but reference data availability for motion vector search is reduced
Solution Approach 1:
The auxiliary frame serves as an intermediary that provides reference data to both encoders without requiring them to copy data from each other. Each encoder can independently access the relevant portion of the auxiliary frame for motion vector search, maintaining reference data availability while enabling independent operation and higher encoding throughput
Data Source
Figure 1
Figure 2
Figure 3~5
AI summary
The present invention relates to the field of image encoding. In particular, it relates to methods and devices where the concept of auxiliary frames may be employed to reduce or remove the need of copying data, for reference encoding purposes, between encoders which encode different parts of an image frame. This purpose is achieved by spatially modifying (S104) original image data before encoding (S106, S108) it using the encoders, and using (S110) the encoded image data as image data of an auxiliary frame. The auxiliary frame is referenced by an inter frame comprising motion vectors corresponding to a restoration of the auxiliary frame image data back to a spatial arrangement of the original image data.