HMD Facial Expression Blendshape Prediction with Facial-Type Cohort Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models for predicting blendshape weights in HMDs are less accurate for HMD wearers with facial types not included in the training dataset, leading to inconsistent and unrealistic avatar representations.
Innovation Solution
Employ multiple machine learning models trained on diverse cohorts corresponding to different facial types, selecting the appropriate model based on the wearer's facial type for accurate blendshape weight prediction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single machine learning model is used for all wearers, then device complexity is reduced, but prediction accuracy deteriorates for wearers with facial types not included in training data
Solution Approach 1:
The patent segments the user population into distinct cohorts based on facial type characteristics. Instead of using a single universal model, multiple specialized machine learning models are created, each trained on data from a specific facial type cohort. This segmentation allows each model to optimize for its target cohort's specific facial geometry and expression patterns, thereby improving prediction accuracy for each group without requiring an overly complex universal model.
Solution Approach 2:
The patent applies local quality by tailoring the machine learning model's characteristics to match the specific cohort it serves. Each cohort receives a model trained on its specific facial type data, making the model locally optimized for that group's unique facial features. This approach ensures that each user group gets the appropriate level of specialization rather than a one-size-fits-all solution.
2Measurement precision
If multiple machine learning models are used for different facial types, then prediction accuracy is improved, but device complexity increases
Solution Approach 1:
The patent implements a dynamic model selection mechanism that automatically chooses the appropriate pre-trained model based on the detected facial type of the current user. Rather than maintaining all models active simultaneously, the system dynamically selects and switches between models based on real-time facial type classification. This dynamic approach provides the accuracy benefits of multiple specialized models while reducing the operational complexity of managing them all at once.
Solution Approach 2:
The patent applies preliminary action by pre-training multiple machine learning models on different facial type cohorts before deployment. These models are prepared in advance and stored for quick retrieval. When a user arrives, the system first classifies their facial type and then retrieves the corresponding pre-trained model, avoiding the need to train or initialize models on-the-fly. This preliminary preparation reduces runtime complexity while maintaining the benefits of specialized models.
3Ease of manufacture
If training data is limited to specific facial types, then model training is simpler, but adaptability to other facial types deteriorates
Solution Approach 1:
The patent segments the overall task of covering all facial types into multiple smaller, manageable training tasks. Instead of attempting to train one model on all possible facial types (which would be complex and data-intensive), the system divides the work into separate training campaigns for different facial type cohorts. Each training effort is simpler and more focused, yet the collection of segmented models collectively achieves comprehensive adaptability across all facial types.
Solution Approach 2:
The patent creates a universal system that handles multiple facial types through a collection of specialized models. The overall architecture is designed to be multi-functional, capable of serving any facial type by selecting from its library of pre-trained models. This universal approach allows the system to maintain simplicity in individual model training while achieving versatility across the entire user population.
Data Source
AI summary
A cohort corresponding to a wearer of a head-mountable display (HMD) is selected from a number of candidate cohorts that each correspond to a different facial type. A set of facial images of the wearer is captured using one or multiple cameras of the HMD. A machine learning model for the selected cohort is applied to the captured set of facial images to predict blendshape weights for the facial expression of the wearer exhibited within the captured set of images. Each candidate cohort has a differently trained machine learning model. The predicted blendshape weights for the facial expression of the wearer are retargeted onto an avatar corresponding to the wearer to render the avatar with the facial expression, and the rendered avatar is displayed.