Intermediate View Synthesis Using Disparity-Based Hybrid Video Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D video coding techniques for stereo and multi-view displays require a high amount of data due to the inclusion of depth or disparity values, leading to reduced compression ratios and increased computational load, while maintaining high-quality view synthesis.
Innovation Solution
A hybrid video decoder and encoder that spatially subdivides frames into sub-regions and uses inter- and intra-view prediction modes, extracting disparity vectors and residuals to reconstruct intermediate views, allowing for efficient synthesis without additional depth data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If depth or disparity values are added to each sample of camera views, then intermediate view synthesis capability is improved, but the amount of data to be conveyed per camera view increases
Solution Approach 1:
The patent extracts disparity information from the video content itself by identifying corresponding points between different camera views and calculating disparity vectors, rather than adding separate depth maps. This extracts the necessary geometric information directly from the multi-view data, avoiding the need for additional depth data while enabling intermediate view synthesis.
Solution Approach 2:
The patent uses disparity vectors as an intermediary representation to bridge between different camera views. These vectors serve as a compact mediator that encodes the geometric relationship between views, allowing the synthesis of intermediate views without transmitting full depth maps or additional color data.
2Adaptability or versatility
If disparity data is added to color data, then intermediate view synthesis is enabled, but compression efficiency deteriorates
Solution Approach 1:
The patent merges the disparity information with the existing video coding framework by integrating disparity vector coding into the motion compensation process. Instead of treating disparity data as a separate component that requires independent compression, it combines disparity estimation with the existing inter-view prediction mechanisms, allowing joint optimization of compression and synthesis capability.
Solution Approach 2:
The patent changes the representation parameters by using compact disparity vectors (with limited precision and scope) instead of full-precision depth maps. By quantizing disparity values and limiting the search space for disparity estimation, the patent transforms the data representation to achieve both synthesis capability and compression efficiency.
3Loss of energy
If proprietary compression techniques are used for color plus depth/disparity data, then compression efficiency is improved, but device complexity increases
Solution Approach 1:
The patent makes the video coding system universal by designing a disparity coding mechanism that works within the existing H.264/MPEG-4 AVC framework without requiring proprietary extensions. The inter-view prediction and disparity vector coding can be applied to any multi-view video sequence, making the solution broadly applicable rather than requiring view-specific or content-specific proprietary techniques.
4Adaptability or versatility
If the number of camera views is increased to cover all multi-view display requirements, then view coverage is improved, but compression efficiency deteriorates
Solution Approach 1:
The patent performs preliminary action by pre-calculating and encoding disparity vectors between adjacent camera views during the encoding process. These pre-computed geometric relationships are stored in the bitstream and can be reused during decoding to synthesize any intermediate view, eliminating the need to pre-render or transmit all possible intermediate views.
Data Source
AI summary
Hybrid video decoder supporting intermediate view synthesis of an intermediate view video from a first- and a second-view video which are predictively coded into a multi-view data signal with frames of the second-view video being spatially subdivided into sub-regions and the multi-view data signal having a prediction mode is provided, having: an extractor configured to respectively extract, from the multi-view data signal, for sub-regions of the frames of the second-view video, a disparity vector and a prediction residual; a predictive reconstructor configured to reconstruct the sub-regions of the frames of the second-view video, by generating a prediction from a reconstructed version of a portion of frames of the first-view video using the disparity vectors and a prediction residual for the respective sub-regions; and an intermediate view synthesizer configured to reconstruct first portions of the intermediate view video.