Multi-Plane Image Generation via Deep Neural Network Depth Probability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image generating devices face challenges in accurately calculating depth values for reconstructing three-dimensional images, leading to potential information loss and difficulties in expressing lighting or reflection effects, especially when simplifying three-dimensional meshes.
Innovation Solution
A deep neural network model is designed to generate multi-plane images using actual photographed data, employing a softmax function and geometric calibration to calculate probability values for depth layers, allowing for the composition of images from multiple viewpoints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If depth values are calculated using stereo camera or multi-camera images to generate 3D mesh, then three-dimensional image reconstruction is achieved, but measurement precision of depth values deteriorates leading to quality degradation
Solution Approach 1:
The patent replaces the traditional mechanical/optical depth calculation method (stereo camera or multi-camera based 3D mesh reconstruction) with a computational approach using deep neural networks. The system uses a trained neural network to directly predict depth maps from input images, substituting the complex geometric calculation and mesh processing with a learned model that achieves both high speed and high precision depth estimation without requiring 3D mesh construction
Solution Approach 2:
The patent changes the fundamental parameter representation from discrete depth values obtained through geometric calibration to continuous probability distributions over depth layers. The neural network outputs soft attention maps representing probability distributions across multiple depth layers, allowing for more nuanced and accurate depth representation. This parameter transformation enables the system to handle depth uncertainty and improve overall measurement precision
2Loss of information
If all points of the image are used to generate three-dimensional mesh, then complete information is retained, but device complexity increases due to huge mesh capacity
Solution Approach 1:
The patent extracts only the essential depth information needed for view synthesis from the input images, rather than constructing a complete and detailed 3D mesh of all image points. The neural network extracts depth maps and soft attention maps that capture the critical structural information, discarding redundant details that would increase mesh complexity without contributing to view synthesis quality. This extraction approach maintains information fidelity while dramatically reducing computational complexity
Solution Approach 2:
The patent segments the depth space into multiple discrete depth layers, with each layer represented by a simplified 2D map rather than a full 3D mesh. Instead of creating a single complex 3D mesh containing all points, the system divides the scene into L depth layers, where each layer is represented by a manageable 2D depth map. This segmentation reduces the overall complexity from a huge 3D mesh to L separate 2D maps, making the system more tractable while preserving essential spatial information
3Device complexity
If three-dimensional mesh is simplified to reduce complexity, then device complexity decreases, but information loss occurs reducing image quality
Solution Approach 1:
The patent creates simplified 2D copies (depth maps) of the 3D scene at multiple depth layers, rather than working with a single complex 3D mesh. Each depth layer is represented by a 2D map that captures the essential geometric information at that depth plane. These 2D copies serve as sufficient representations for view synthesis tasks, eliminating the need to maintain and process a complete high-resolution 3D mesh while preserving the necessary spatial and geometric information
4Reliability
If texture is applied to three-dimensional mesh to improve image quality, then visual fidelity improves, but ability to express lighting and reflection effects according to viewpoints deteriorates
Solution Approach 1:
The patent adds the depth dimension explicitly by representing the scene as L separate depth layers with associated 2D maps, rather than relying solely on texture mapping on a 3D mesh. By organizing information across multiple depth planes, the system can naturally handle viewpoint changes and compute appropriate lighting and reflection effects for each depth layer, achieving both visual fidelity and viewpoint adaptability through this dimensional organization
Data Source
AI summary
Disclosed are an electronic device generating a multi-plane image of an arbitrary viewpoint and an operating method thereof. The electronic device includes a memory that stores a plurality of images, and at least one processor that executes a machine learning-based multi-plane image generating module generating an image of an arbitrary viewpoint from the plurality of images stored in the memory. When the multi-plane image generating module is executed, the processor selects at least two images from the plurality of images, by performing geometric calibration on the plurality of images, calculates a plane sweep volume including at least two layer images, performs an operation of a deep neural network to generate a plurality of multi-plane images having probability values, and generates the image of the arbitrary viewpoint by composing the plurality of multi-plane images and stores the image of the arbitrary viewpoint in the memory.