Virtual Character Image Augmentation Using 2D Pose and 3D Fitting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The digital augmentation of the human body in augmented reality faces significant technical challenges due to ambiguities in depth projection and variations in human body shapes, making 3D human pose estimation from RGB images under-constrained and ambiguous, and existing solutions require large data sets of 3D skeleton poses that may not cover all relevant poses.
Innovation Solution
An image processing system that includes a 2D pose estimation module and a 3D poses library, using an optimization algorithm to identify a 3D pose and shape from 2D data, allowing for automated image augmentation with virtual characters that can overlap or be integrated with human images, and is capable of generating augmented images in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If large data sets of 3D skeleton poses are used for training, then the coverage of relevant poses improves, but the data collection complexity and preprocessing burden increase
Solution Approach 1:
The system uses the subject's own 2D image to automatically generate the corresponding 3D pose estimation without requiring pre-collected training data of that specific subject. The optimization algorithm self-adapts to estimate poses from the input image directly, eliminating the need for extensive subject-specific data collection.
Solution Approach 2:
The system creates a virtual 3D skeleton model that copies and reproduces the pose from the 2D input image. Instead of relying on large training datasets to learn pose patterns, the system directly copies the pose information from the image into a 3D skeletal representation through optimization.
2Adaptability or versatility
If 3D human pose estimation is performed from RGB images, then the immersion and enjoyment of augmented reality experiences improve, but the estimation accuracy deteriorates due to depth projection ambiguities and body shape variations
Solution Approach 1:
The system transitions from 2D image space to 3D pose space by introducing a virtual skeleton model with depth information. The optimization algorithm solves the under-constrained problem by adding the dimensional constraint that the 3D skeleton projection must match the 2D pose landmarks, effectively using the third dimension to resolve ambiguities.
Solution Approach 2:
The system changes the parameter representation from direct 3D coordinate estimation to optimization-based pose fitting. By parameterizing the pose as a set of joint positions in a virtual skeleton and optimizing these parameters to match 2D landmarks, the system improves estimation accuracy despite depth ambiguities.
3Productivity
If automated image augmentation is implemented, then the productivity of generating augmented images improves, but the complexity of the processing system increases
Solution Approach 1:
The system performs preliminary pose estimation and 3D skeleton fitting before the actual image augmentation. By pre-computing the 3D pose from the 2D image using optimization, the system prepares the virtual character configuration in advance, enabling efficient real-time augmentation without complex runtime processing.
Solution Approach 2:
The virtual skeleton model serves as an intermediary between the 2D input image and the 3D augmented output. This intermediate representation simplifies the processing pipeline by providing a structured bridge that connects 2D pose detection with 3D character rendering, making the overall system more manageable despite the complexity of the transformation.
Data Source
AI summary
An image processing system includes a computing platform having a hardware processor and a system memory storing an image augmentation software code, a three-dimensional (3D) shapes library, and/or a 3D poses library. The image processing system also includes a two-dimensional (2D) pose estimation module communicatively coupled to the image augmentation software code. The hardware processor executes the image augmentation software code to provide an image to the 2D pose estimation module and to receive a 2D pose data generated by the 2D pose estimation module based on the image. The image augmentation software code identifies a 3D shape and/or a 3D pose corresponding to the image using an optimization algorithm applied to the 2D pose data and one or both of the 3D poses library and the 3D shapes library, and may output the 3D shape and/or 3D pose to render an augmented image on a display.