Virtual Object Control via 2D and 3D Feature Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional motion sensors for controlling virtual objects require external connections and have low precision, as they can only detect hand motions, limiting user experience in applications like video games and instant messaging.
Innovation Solution
A method using a combination of two-dimensional and three-dimensional feature information from RGB and depth images, respectively, to generate human body model information for precise control of virtual objects without the need for additional motion sensors, utilizing image sensors and neural network models like Mobilenet-Openpose for feature identification and segmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a conventional motion sensor is used to control virtual objects, then hand motions can be detected, but the device requires external connections and has low precision
Solution Approach 1:
The patent replaces the mechanical motion sensor system with an optical imaging system. Instead of using physical sensors that require connections, the system uses cameras to capture images and processes them through algorithms to detect human pose and motion, thereby substituting a mechanical detection system with an optical one.
Solution Approach 2:
The patent creates a virtual copy of the human body through pose estimation algorithms. By capturing images and generating skeletal models that replicate human pose information, the system obtains motion data without physical sensors, achieving precise motion detection through digital reproduction rather than physical measurement.
2Measurement precision
If a conventional motion sensor is used, then hand motions can be detected, but only hand motions are detected limiting overall precision
Solution Approach 1:
The patent implements a universal pose estimation system that can detect not only hand motions but also body, head, and other肢体 movements. The skeletal model encompasses multiple body parts, allowing the same system to perform multiple detection functions simultaneously, thereby achieving both high precision and broad adaptability.
Solution Approach 2:
The patent transitions from one-dimensional hand motion detection to three-dimensional full-body pose estimation. By capturing spatial coordinates of multiple body joints in 3D space and tracking their movements, the system expands the detection dimension from limited hand gestures to comprehensive body motion, significantly improving both precision and versatility.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
Provided are a method for controlling virtual objects, a computer readable storage medium and an electronic device. The method includes: acquiring (101) a two-dimensional image and a depth image of a human body; acquiring (102) two-dimensional feature information of the human body according to the two-dimensional image; acquiring (103) three-dimensional feature information of the human body according to the depth image; combining (104) the two-dimensional feature information of the human body with the three-dimensional feature information of the human body to generate human body model information; and controlling (105) a virtual object according to the human body model information.