Single RGBD Camera Volumetric Capture for AR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for capturing high-quality volumetric reconstructions in augmented and virtual reality applications require expensive multi-view capture systems with complex setups, which are costly and unsuitable for real-time applications due to distorted geometry, poor texturing, and inaccurate lighting.
Innovation Solution
A single RGBD camera captures and stores calibration images, allowing for the generation of high-quality volumetric reconstructions by selecting and warping calibration images based on the object's pose and viewpoint, using convolutional neural networks to blend and refine the images for free viewpoint rendering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If complex multi-view capture rigs are used, then volumetric reconstruction quality is improved, but system cost and complexity increase significantly
Solution Approach 1:
The patent uses a single RGBD camera to capture calibration images that serve as templates for generating volumetric reconstructions from multiple viewpoints. Instead of using multiple physical cameras simultaneously, the system captures a set of images from different angles using one camera and then synthesizes multi-view reconstructions by warping and blending these calibration images, effectively creating virtual copies of the object from different perspectives
Solution Approach 2:
The system performs preliminary capture of calibration images showing the object from multiple angles before actual volumetric reconstruction is needed. These pre-captured images are stored and then reused to generate reconstructions from arbitrary viewpoints, eliminating the need for complex real-time multi-camera systems during actual operation
2Manufacturing precision
If complex multi-view capture rigs are used, then volumetric reconstruction quality is improved, but real-time processing capability deteriorates
Solution Approach 1:
The calibration images are captured and processed in advance, creating a library of templates that can be quickly retrieved and warped during actual volumetric reconstruction. This preliminary capture phase separates the computationally intensive image acquisition from the real-time reconstruction phase, enabling fast processing when needed
Solution Approach 2:
The system creates warped copies of calibration images to simulate views from different viewpoints. Instead of capturing multiple views simultaneously with multiple cameras, the system generates synthetic views by warping pre-captured images, significantly reducing the computational burden during real-time operation
3Device complexity
If a single RGBD camera is used, then system cost and complexity are reduced, but volumetric reconstruction quality may deteriorate
Solution Approach 1:
The system dynamically selects and weights different calibration images based on the desired viewpoint. When generating a volumetric reconstruction from a specific angle, the system identifies which pre-captured calibration images are most relevant and applies appropriate warping and blending operations, adapting the reconstruction process to the specific viewing requirements
Solution Approach 2:
The patent introduces convolutional neural networks as an intermediary to learn the mapping between calibration images and target viewpoint images. The neural network acts as a mediator that automatically learns how to warp and blend calibration images to produce high-quality reconstructions, eliminating the need for complex manual processing pipelines
Data Source
AI summary
A method includes receiving a first image including color data and depth data, determining a viewpoint associated with an augmented reality (AR) and/or virtual reality (VR) display displaying a second image, receiving at least one calibration image including an object in the first image, the object being in a different pose as compared to a pose of the object in the first image, and generating the second image based on the first image, the viewpoint and the at least one calibration image.