Avatar Rendering Using Uncorrelated Codec Portions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current avatar rendering systems face challenges in rendering avatars with accurate poses due to limited training data sets, leading to overfitting and correlation of uncorrelated body parts, resulting in incorrect pose rendering.
Innovation Solution
A method is developed to train a machine-learning model by uncorrelating codec portions representing different body parts, using a training process where each body part is randomly selected and trained in iterations, allowing the model to recognize and render avatars with any pose using uncorrelated codec portions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a decoder is trained on a limited training data set, then training time and data requirements are reduced, but the decoder overfits to the training data and correlates body parts that should not be correlated, resulting in incorrect pose rendering
Solution Approach 1:
The codec is divided into multiple codec portions, each corresponding to a specific body part. During training, only a subset of codec portions is activated in each training iteration, allowing the decoder to learn independent representations of different body parts without overfitting to correlated poses in the limited training data.
Solution Approach 2:
The training process preliminarily separates the learning of different body parts by randomly selecting subsets of codec portions for each training iteration. This preliminary action of uncorrelating body part representations during training enables the decoder to later combine them correctly for accurate pose rendering.
2Reliability
If excessive training data is used to prevent overfitting, then pose rendering accuracy improves, but training time and computational resources increase significantly
Solution Approach 1:
By segmenting the codec into body part-specific portions and training them independently in random subsets, the method achieves reliable pose rendering without requiring excessive training data. Each body part learns its representation independently, reducing the need for large datasets to capture all possible pose combinations.
Solution Approach 2:
Instead of training all body parts simultaneously with extensive data, the method uses partial action by activating only a subset of codec portions in each training iteration. This reduces the immediate training burden while still achieving comprehensive learning through multiple iterations with different random subsets.
3Reliability
If the neural network is enlarged to handle all pose combinations, then pose rendering accuracy improves, but device complexity and computational cost increase
Solution Approach 1:
The neural network processes segmented codec portions independently rather than handling all body parts simultaneously. This segmentation allows a smaller, more efficient network architecture to achieve accurate pose rendering by processing body part representations separately and combining them, avoiding the need for a large complex network.
Solution Approach 2:
The decoder is designed with multi-functionality to handle different random subsets of codec portions during training and inference. This universal design allows the same network structure to adapt to various body part combinations without requiring separate specialized networks for each pose scenario, reducing overall complexity.
Data Source
AI summary
In one embodiment, a system may access a codec that encodes an appearance associated with a subject and comprise codec portions that respectively correspond to body parts of the subject. The system may generate a training codec that comprises a first subset of the codec portions (a first set of body parts) and a modified second subset of the codec portions (muted body parts). The system may decode the training codec using a machine-learning model to generate a mesh of the subject. The system may transform the mesh of the subject based on a predetermined pose. The system may update the machine-learning model based on a comparison between the transformed mesh and a target mesh of the subject having the predetermined pose. The system in the present application can train a machine-learning model to render an avatar with a pose using uncorrelated codec portions corresponding to different body parts.