Immersive Video Atlas Encoding for Decoder-Side Depth Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing immersive video technologies struggle to support 6 degrees of freedom (6DoF) by providing depth estimation for rotational and translational movements, particularly in omnidirectional images, limiting the realism and immersion of virtual reality experiences.
Innovation Solution
A method for encoding and decoding immersive video that allows depth estimation on the decoder side, using a syntax structure to classify and encode view images into basic and additional images, generate atlases, and include metadata for depth estimation, enabling dynamic viewport image synthesis based on user motion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth information is encoded for all view images, then rendering quality is improved, but data transmission bandwidth and storage requirements increase
Solution Approach 1:
The patent segments the encoding approach by applying different depth information strategies to different view images. Basic view images receive full depth information encoding, while additional view images use depth estimation or pruning, dividing the problem into manageable parts with different resource requirements.
Solution Approach 2:
The patent changes the parameter of depth information representation by switching between encoding actual depth maps and estimating depth maps through reprojection and refinement processes. This parameter change allows dynamic adjustment between quality and bandwidth based on the specific view image type.
2Device complexity
If depth estimation is performed on decoder side, then device complexity is reduced, but processing time increases
Solution Approach 1:
The patent applies preliminary action by pre-calculating and encoding depth information for basic view images during the encoding phase. This allows the decoder to reuse this pre-processed depth information for additional views through reprojection, reducing real-time processing requirements.
Solution Approach 2:
The patent uses copying by reprojecting depth maps from basic view images to generate depth information for additional view images. Instead of encoding or estimating depth independently for each additional view, the system copies and transforms existing depth information, significantly reducing processing time.
3Adaptability or versatility
If 6DoF movement is supported, then immersion and realism are improved, but system complexity increases
Solution Approach 1:
The patent adds another dimension to traditional omnidirectional video by incorporating depth information (transforming 2D images into 3D representations). This enables 6DoF movement support while managing complexity through selective depth encoding and efficient reprojection techniques for generating views in different spatial dimensions.
Data Source
AI summary
A video encoding method includes classifying a plurality of view images into a basic image and additional images, performing pruning on at least one of the plurality of view images on the basis of the classification result, generating an atlas on the basis of the pruning result, and encoding the atlas and metadata for the atlas. Here, the metadata includes a first flag indicating whether depth estimation needs to be performed on a decoder side.