Augmented Reality Object Control With 3D Scene Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality devices face challenges in efficiently recognizing and manipulating real-world objects due to computationally intensive operations, leading to increased processing time, heat generation, and power consumption, especially when dealing with complex environments like indoor spaces with multiple furniture pieces.
Innovation Solution
The augmented reality device employs a camera to obtain spatial images, identifies planes such as walls and floors, generates a 3D model by extending these planes, performs 3D inpainting, and segments objects using 2D and 3D segmentation techniques to control virtual and real-world objects, reducing computational load and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If continuous deep neural network operations are performed for segmentation and inpainting, then augmented reality service quality is improved, but processing time increases and power consumption increases
Solution Approach 1:
The system performs preliminary actions by pre-processing images and preparing segmentation models before the actual augmented reality operation. The deep neural network models are pre-trained and stored, allowing the device to load and apply pre-computed segmentation results rather than performing real-time computational intensive operations, thus reducing power consumption while maintaining service quality
Solution Approach 2:
The patent uses pre-computed segmentation masks and 3D models as copies of the real-world scene. Instead of continuously processing live camera feeds through deep neural networks, the system uses these pre-generated segmentation data and 3D reconstructions to render augmented reality objects, significantly reducing real-time computational load and power consumption
2Reliability
If continuous deep neural network operations are performed for segmentation and inpainting, then augmented reality service quality is improved, but processing time increases
Solution Approach 1:
The system performs preliminary actions by pre-processing images and preparing segmentation models before the actual augmented reality operation. The deep neural network models are pre-trained and stored, allowing the device to load and apply pre-computed segmentation results rather than performing real-time computational intensive operations, thus reducing power consumption while maintaining service quality
Solution Approach 2:
The patent uses pre-computed segmentation masks and 3D models as copies of the real-world scene. Instead of continuously processing live camera feeds through deep neural networks, the system uses these pre-generated segmentation data and 3D reconstructions to render augmented reality objects, significantly reducing real-time computational load and processing time
3Reliability
If continuous deep neural network operations are performed for segmentation and inpainting, then augmented reality service quality is improved, but heat generation increases
Solution Approach 1:
The system performs preliminary actions by pre-processing images and preparing segmentation models before the actual augmented reality operation. The deep neural network models are pre-trained and stored, allowing the device to load and apply pre-computed segmentation results rather than performing real-time computational intensive operations, thus reducing power consumption while maintaining service quality
Solution Approach 2:
The patent uses pre-computed segmentation masks and 3D models as copies of the real-world scene. Instead of continuously processing live camera feeds through deep neural networks, the system uses these pre-generated segmentation data and 3D reconstructions to render augmented reality objects, significantly reducing real-time computational load and heat generation
4Productivity
If simple interpolation method is used for inpainting, then processing speed is improved, but virtual object generation accuracy deteriorates when background is bent
Solution Approach 1:
The patent uses pre-computed segmentation masks and 3D models as copies of the real-world scene. Instead of continuously processing live camera feeds through deep neural networks, the system uses these pre-generated segmentation data and 3D reconstructions to render augmented reality objects, significantly reducing real-time computational load and processing time
Solution Approach 2:
The system changes the parameter of inpainting methodology from simple interpolation to 3D-aware inpainting that uses depth information and 3D model constraints. This allows the system to maintain high processing speeds while achieving accurate virtual object generation even in complex bent background scenarios by leveraging three-dimensional spatial understanding
5Adaptability or versatility
If multiple furniture pieces are present in the scene, then scene complexity is improved, but selective deletion capability deteriorates
Solution Approach 1:
The patent applies segmentation by dividing the complex scene into distinct 3D objects and planes using deep neural network-based object detection and segmentation. Each furniture piece is segmented into separate 3D models with associated depth information and spatial coordinates, allowing the system to selectively manipulate individual objects without affecting others in the scene
Solution Approach 2:
The system transitions from 2D image processing to 3D spatial processing by generating three-dimensional models of objects and surfaces. This dimensional transformation enables selective deletion and manipulation of specific objects in space based on their 3D positions and orientations, providing precise control even in complex scenes with multiple overlapping furniture pieces
Data Source
AI summary
Provided are an augmented reality device for providing an augmented reality service for controlling an object in a real world space, and an operating method for the same. The method may include recognizing a first plane comprising a wall and a second plane comprising a floor from a spatial image based on a photograph of the real world space; generating a three-dimensional (3D) model of the real world space by extending the wall and extending the floor along the respective planes and performing 3D in-painting on an area of the extended wall and the extended floor that is hidden by an obstructive object; segmenting an object selected by a user input from the spatial image based on two-dimensional (2D) segmentation; and segmenting the object on the spatial image from the real world space based on a 3D model or 3D position information of the object using 3D segmentation.