Mixed-Reality System Using Multi-Camera 3D Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for recognizing a real-world environment in real-time with high precision are inadequate for mixed-reality applications, failing to simultaneously achieve real-time properties and high-precision measurement, which is essential for rendering high-quality global illumination and accurate position tracking in MR environments.
Innovation Solution
A system that includes a server and a portable display device with image acquisition devices, a three-dimensional-space-data storage part, a table storage part, a color-information determining part, a user-environment determining part, a virtual-illumination-information generating part, and a rendering part, which associates pixel position information with three-dimensional shape elements and updates color information to reflect real-time changes, enabling accurate rendering of virtual objects in a mixed-reality environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a 3D laser scanner is used to photograph three-dimensional information with high precision, then measurement precision is improved, but the time required for measurement increases significantly (10 minutes minimum, 30 minutes for standard quality), making it unsuitable for real-time applications
Solution Approach 1:
The patent divides the 360-degree measurement task into multiple overlapping views captured by multiple cameras positioned around the observation space. Each camera captures a portion of the scene, and these partial views are later synthesized to reconstruct the complete 3D environment, thereby reducing the time required compared to a single comprehensive scan.
Solution Approach 2:
The system uses standard cameras that can capture both color and depth information through multi-view geometry, replacing specialized expensive equipment like 3D laser scanners. This multi-functional approach allows ordinary cameras to perform 3D reconstruction tasks that previously required dedicated high-precision scanning devices.
2Manufacturing precision
If a 3D laser scanner is used to achieve high measurement precision, then manufacturing precision is improved, but the cost increases to a few million to a few tens of million yen each, making mass deployment unsuitable
Solution Approach 1:
The patent replaces expensive, durable 3D laser scanners with inexpensive standard cameras that can be mass-deployed. While individual cameras are cheaper and less robust, their low cost enables widespread deployment across multiple locations, and the system achieves comparable 3D reconstruction quality through the collective power of multiple camera views.
Solution Approach 2:
Instead of using a single expensive 3D scanner, the system creates multiple 2D image copies from different angles using standard cameras. These 2D copies are then processed through Structure-from-Motion algorithms to reconstruct the 3D environment, effectively copying the scanning function across multiple inexpensive devices.
3Device complexity
If Structure-from-Motion is used to construct 3D models inexpensively, then device complexity is reduced, but real-time properties are lost and the method cannot be directly applied for MR environment implementation
Solution Approach 1:
The patent performs 3D reconstruction using Structure-from-Motion in advance to create a static geometric framework of the environment. This preliminary 3D model serves as a foundation that can then be updated in real-time with minimal processing, as only changes need to be detected and integrated into the pre-established structure rather than reconstructing everything from scratch.
Data Source
AI summary
A system that enables real-time recognition of a real-space environment in a mixed-reality environment includes a server, a portable display device including a display unit for displaying a virtual object to a user and a photographing unit that photographs a real space, and image acquisition devices that acquire images individually from a plurality of fixed points where it is possible to photograph a region in a predetermined real space, the system making it possible to render the virtual object in a superimposed fashion on a real space or a photographed image of the real space, viewed by the user via the display unit, the server or the display device including a three-dimensional-space-data storage part, a table storage part, a color-information determining part, a color-information updating part, a user-environment determining part, a virtual-illumination-information generating part, and a rendering part.