RGB Camera 3D User Model for VR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current virtual reality systems require expensive and cumbersome motion sensing hardware to provide immersive experiences, limiting user interaction and visibility of their body within the virtual environment.
Innovation Solution
A method and system using a computing device with an RGB or RGB-D camera to construct a three-dimensional model of a user by segmenting their body from images, forming a 2D contour, and projecting it into a virtual environment through geometric transformation, eliminating the need for dedicated motion sensing hardware.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dedicated motion sensing hardware is used to provide immersive virtual reality experience, then user interaction and body visibility in virtual environment is improved, but system cost and device complexity increase
Solution Approach 1:
The patent creates a virtual copy of the user's body by capturing images with a standard camera, processing them through neural networks to generate 3D models, and rendering these models in the virtual environment. This copying approach replaces the need for expensive motion sensing hardware while maintaining the immersive experience of seeing one's body in the virtual world.
Solution Approach 2:
The patent substitutes mechanical motion sensing hardware with a software-based computer vision system. Instead of using physical sensors to detect body movements, the system uses image processing, neural networks, and geometric transformations to track and represent the user's body in the virtual environment, thereby eliminating complex hardware requirements.
2Ease of operation
If dedicated motion sensing hardware is used for virtual reality, then user interaction capability is improved, but system cost increases
Solution Approach 1:
The patent makes the standard camera multi-functional by using it not only for capturing images but also for tracking body movements and constructing 3D models. This universal approach allows a common, inexpensive device to perform the functions previously requiring specialized motion sensing hardware, thereby improving ease of operation without increasing hardware cost.
Solution Approach 2:
The system enables the camera itself to serve the dual purpose of both capturing visual content and tracking user movements for interaction. By processing the camera's own output through computer vision algorithms, the system eliminates the need for separate motion sensing hardware, reducing cost while maintaining interaction capability.
3Loss of information
If traditional VR hardware is used, then body visibility in virtual environment is improved, but ease of manufacture and accessibility worsen
Solution Approach 1:
The patent generates a visual copy of the user's body by processing camera images through neural networks to create 3D models that are then rendered in the virtual environment. This copying mechanism ensures complete body visibility without requiring specialized hardware, making the system easier to manufacture and more accessible to users with standard devices.
Data Source
AI summary
Methods and systems for constructing a three-dimensional (3D) model of a user in a virtual environment for full body virtual reality (VR) applications are described. The method includes receiving an image of the user captured using an RGB camera; detecting a body bounding box associated with the user using a first trained neural network; determining a segmentation map of the user, based on the body bounding box; determining a two-dimensional (2D) contour of the user from the segmentation map; forming a 3D extrusion model by extruding the 2D contour; and constructing the 3D model of the user in the virtual environment by applying a geometric transformation to the 3D extrusion model. Applications of full body VR include physical training and fitness sessions, games, control of computing devices, manipulation and display of data, interactive social media with VR, and the like.