Multiplane Image Tile Blending for Volumetric Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for encoding and decoding multiplane images (MPI) for volumetric scenes are inefficient in terms of memory and processing resources, particularly when rendering on end-user devices like mobile devices or Head-Mounted Displays (HMD).
Innovation Solution
The proposed method involves obtaining metadata and tiles, where each tile is associated with a depth and has a tile number determined as a monotonic function of the corresponding depth. This allows for the generation of a viewport image by decoding and blending tiles in monotonic order, optimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiplane images are encoded with full-resolution layers for each depth slice, then view synthesis quality is improved, but memory requirements and processing load increase significantly
Solution Approach 1:
The patent divides the multiplane image into multiple depth slices or layers, where each layer represents a specific depth range. Instead of processing the entire high-resolution scene at once, the system segments it into manageable layers that can be processed independently and blended together, reducing peak memory requirements while maintaining view synthesis quality.
Solution Approach 2:
The patent introduces a depth dimension organization to the traditional 2D image processing pipeline. By arranging image layers along the depth axis and using monotonic depth ordering, the system efficiently manages memory usage through depth-based layer composition, allowing progressive rendering from near to far layers without loading all layers simultaneously into memory.
2Reliability
If all tiles are decoded and rendered simultaneously, then rendering completeness is improved, but processing load and memory usage increase
Solution Approach 1:
The patent performs preliminary sorting of tiles based on their depth values before the rendering process. By pre-organizing tiles in monotonic depth order, the system enables progressive rendering where tiles can be processed and displayed in sequence from nearest to farthest, allowing early tiles to be rendered and displayed before all tiles are fully processed, thus reducing overall processing load while maintaining completeness.
3Quantity of substance
If depth-based monotonic ordering is applied to tile rendering, then memory efficiency is improved, but implementation complexity increases
Solution Approach 1:
The patent changes the rendering parameter from arbitrary or raster-order processing to depth-based monotonic ordering. By associating each tile with its depth value and processing tiles in ascending depth order, the system achieves efficient memory management where only a subset of layers needs to be in memory at any given time, as closer layers are rendered first and can be discarded before distant layers are processed.
Data Source
AI summary
Methods and devices for encoding, decoding and transmitting a three-dimensional scene initially represented as a multiplane image (MPI) are provided. Each layer of the MPI is split into patches based on the transparency component. Patches of a layer are grouped in a tile. The greater the depth of the layer, the greater the identifying number of the tile. When several tiles are packed in an atlas image, the same monotonic (i.e. ascending or descending) function according to depth applies to atlas numbers. At the decoding side, the current viewport to render is initially cleared and each decoded tile is sequentially blended over from the nearest one to the furthest due to the numbering of the set of atlases and tiles. Pixels of a patch under rendering are projected onto pixels of the viewport image according to the depth of the tile comprising the patch and metadata indicating the position of the patch in the layer of the MPI the patch has been clustered from.