Semantic 3D Reconstruction Guidance for Complete Surface Capture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image-based 3D reconstruction methods struggle to guide consumers in capturing the complete surface of a target object using a hand-held camera, leading to incomplete 3D models due to missing parts, particularly when using single photo reconstruction, spatial guidance, movement guidance, or semantic reconstruction methods that lack clear guidance for capturing multiple images.
Innovation Solution
A device and method that utilizes semantic information to estimate incomplete parts of a 3D model by generating a semantic 3D model, deforming a template model to identify errors, and providing guidance through sound, text, or figure cues to complete the scanning process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If spatial guidance with arrows is used to indicate scanned and unscanned parts, then users can understand which parts are taken or not taken, but the arrows become invisible when the target object is large and the camera preview does not show the arrow positions
Solution Approach 1:
The patent introduces a virtual hemispherical surface as an intermediary carrier for guidance indicators. This hemisphere is overlaid on the camera preview and rotates to align with the camera's field of view, providing a stable reference frame for displaying scanned and unscanned region indicators without being affected by the size of the target object or camera movement.
Solution Approach 2:
The patent transitions from displaying indicators directly in the 2D camera preview plane to displaying them on a 3D rotating hemisphere that represents the spatial arrangement of scanning directions. This dimensional change allows indicators to remain visible and meaningful regardless of camera angle or object size.
2Ease of operation
If movement guidance arrows are overlaid on camera preview to specify preferred camera movement, then users can be guided to scan complete surface, but users do not know exactly which parts are not taken and scanning remains incomplete or time-consuming
Solution Approach 1:
The patent implements a feedback mechanism that continuously monitors the scanning process and updates the hemisphere display to show which regions have been scanned and which remain unscanned. This real-time feedback allows users to understand the current state of reconstruction and adjust their scanning movements accordingly, ensuring complete coverage of all surface regions.
Solution Approach 2:
The patent segments the surface of the target object into multiple scanning regions represented on the hemisphere. Each region can be independently tracked to determine whether it has been scanned, allowing precise identification of incomplete areas and guiding users to scan specific regions that need completion.
3Ease of operation
If single photo reconstruction is used to reconstruct complete 3D model from single frontal image, then no guidance for capturing multiple images is required, but the back of the 3D model has artifacts with inaccurate shapes and colors
Solution Approach 1:
The patent performs preliminary action by pre-computing and displaying the complete hemisphere showing all possible scanning directions before the user begins capturing images. This allows users to understand the full scope of required scanning and plan their capture sequence accordingly, ensuring that images from multiple angles including the back are captured to avoid artifacts.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed is a device for reconstructing a 3D model. The device includes: a camera; and a processor configured to: determine a class of a subject; capture one or more images of the subject by using the camera; extract semantic information from the images based on the determined class; update the 3D model by using the images; estimate incomplete semantic parts of the 3D model; and present semantic guidance for the user to capture the incomplete semantic parts. The device achieves clearly understandable guidance for reconstructing a complete 3D model.