Multiview Video Encoding Using Joint Temporal Inter-View Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current multiview video encoding and decoding methods are inefficient due to independent encoding of videos from different viewpoints, which fails to leverage temporal and spatial correlations between views, leading to increased network bandwidth and storage requirements.
Innovation Solution
A joint temporal/inter-view processing method that uses motion compensated temporal filtering (MCTF) and disparity compensated inter-view filtering (DCVF) to decompose multiview videos into low and high band frames, with a reference picture list managing temporal, spatial, and synthesized reference pictures for adaptive prediction modes, including view synthesis, to generate a synthetic view of the scene.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If independent encoding of videos from different viewpoints is used, then encoding simplicity is maintained, but compression efficiency deteriorates
Solution Approach 1:
The patent merges the encoding processes of multiple viewpoint videos by introducing inter-view prediction mechanisms. The encoder at the first viewpoint uses motion compensation and disparity compensation to predict frames from the second viewpoint, combining temporal and spatial correlations into a unified encoding framework that improves compression efficiency while maintaining manageable complexity
Solution Approach 2:
The patent creates a universal prediction framework that handles both temporal prediction (within a single viewpoint) and spatial prediction (across different viewpoints) using similar motion compensation techniques. This multi-functional approach allows the same encoding infrastructure to serve multiple prediction purposes, improving efficiency without proportionally increasing complexity
2Productivity
If inter-view correlations are used for prediction, then compression efficiency is improved, but system complexity increases
Solution Approach 1:
The patent segments the prediction process into distinct components: temporal prediction using motion compensation within each viewpoint, and spatial prediction using disparity compensation between viewpoints. This segmentation allows each component to be optimized independently and reused across different video streams, managing system complexity while achieving improved compression through inter-view correlations
3Productivity
If motion compensated temporal filtering and disparity compensated inter-view filtering are applied, then compression efficiency increases, but computational requirements increase
Solution Approach 1:
The patent performs preliminary motion estimation and disparity estimation on reference frames before actual prediction. By pre-computing motion vectors and disparity vectors from previously decoded frames, the system reduces the computational burden during real-time prediction operations, allowing MCTF and DCVF to be applied efficiently with reduced real-time computational requirements
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Multiview videos are acquired by overlapping cameras. Side information is used to synthesize multiview videos. A reference picture list is maintained for current frames of the multiview videos, the reference picture indexes temporal reference pictures and spatial reference pictures of the acquired multiview videos and the synthesized reference pictures of the synthesized multiview video. Each current frame of the multiview videos is predicted according to reference pictures indexed by the associated reference picture list with a skip mode and a direct mode, whereby the side information is inferred from the synthesized reference picture. In addition, the skip and merge modes for single view video coding are modified to support multiview video coding by generating a motion vector prediction list by also considering neighboring blocks that are associated with synthesized reference pictures.