Virtual Accessory Rendering With 6DoF Pose Tracking Under Occlusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AR technology struggles to provide realistic virtual try-on experiences for accessories due to high degrees of freedom and self-occlusion issues, particularly with body parts like hands and fingers, leading to poor fit and failure to follow motion.
Innovation Solution
A method involving key point prediction and six degrees of freedom posture adjustment is used to simulate the posture of virtual accessory carriers, accounting for occlusion and motion, enhancing the rendering of accessories on body parts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional AR technology is used for virtual accessory try-on, then the implementation is simple, but the realism and accuracy of the virtual try-on effect is poor due to high degrees of freedom and self-occlusion issues
Solution Approach 1:
The patent segments the accessory carrier model into multiple body parts (hands, fingers, arms, etc.) and establishes specific posture parameters for each segment. This allows independent control and adjustment of each body part's posture, enabling accurate tracking of complex motions while managing the high degrees of freedom through modular parameter organization.
Solution Approach 2:
The patent introduces a six-degree-of-freedom (6DoF) posture parameter system that adds spatial dimensionality (position and orientation in 3D space) to the traditional 2D image processing approach. This dimensional expansion enables realistic virtual accessory placement by accounting for depth, rotation, and translation across multiple axes, resolving the self-occlusion and motion tracking problems.
2Measurement precision
If key point prediction and six degrees of freedom posture adjustment are implemented, then the accuracy of accessory fitting is improved, but the computational complexity and processing time increase
Solution Approach 1:
The patent performs key point prediction on the accessory carrier image in advance to identify critical anatomical landmarks (fingertips, joint positions, etc.) before the actual accessory rendering process. This preliminary action establishes the posture parameter baseline, enabling accurate virtual accessory fitting while reducing real-time computational complexity by pre-calculating the posture reference points.
3Reliability
If the system accounts for occlusion and rapid motion in posture adjustment, then the realism of virtual try-on is enhanced, but the processing requirements and computational load increase
Solution Approach 1:
The patent implements a dynamic posture parameter adjustment mechanism that continuously updates the six-degree-of-freedom parameters based on real-time key point prediction results. The system adapts to rapid motion and occlusion conditions by dynamically recalculating posture parameters while maintaining the virtual accessory's correct spatial relationship with the moving body parts, enabling realistic try-on under varying conditions.
Data Source
Figure 1~2
Figure 3
AI summary
The application provides a method, apparatus, electronic device, and storage medium for image rendering including: obtaining an accessory carrier image, and performing key point prediction on the accessory carrier image to generate a key point result; determining a six degrees of freedom posture value based on the key point result, a predetermined stereoscopic point data, and a device parameter; then adjusting a posture of a virtual accessory carrier model, where the virtual accessory carrier model is mounted with a virtual accessory model, and obtaining an accessory image of the virtual accessory model after posture adjustment based on the device parameter; and rendering the accessory image onto the accessory carrier image. In this application, the key point and the six degrees of freedom posture value of the accessory carrier are determined, the posture of the virtual accessory carrier model is adjusted by the six degrees of freedom posture value, so as to simulate the real posture of the accessory carrier. And then the image of the virtual accessory model is obtained based on the image acquisition device set by the device parameter, simulating the real wearing effect of the accessory at a specific angle, finally enhancing the rendering effect of the accessory worn by the carrier during occlusion and motion.