Audio-Driven 3D Avatar Animation With Disentangled Content and Style
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current deep learning models for audio-driven 3D avatar animation on mobile devices require separate training of independent models for content and style extraction, leading to large model sizes, suboptimal performance, and resource constraints, necessitating extensive training data and time.
Innovation Solution
A shared feature extractor is used to extract audio features, with a machine learning model trained to disentangle content and style embeddings, allowing for efficient generation of parameter predictions, reducing model size and improving accuracy across various contents and styles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If separate independent models are trained for content and style extraction, then model performance for content and style can be optimized independently, but device complexity and model size increase significantly
Solution Approach 1:
The patent merges separate content and style extraction models into a single unified model that processes audio inputs to generate both content and style embeddings simultaneously. This integration reduces the overall model size and computational complexity while maintaining the ability to extract both types of information through shared feature extractors and disentangled embedding layers.
Solution Approach 2:
The unified model architecture is designed to perform multiple functions - extracting both content and style information from audio inputs using a single model structure. The model uses shared feature extractors that can handle both content and style processing, making the system more efficient and reducing the need for separate specialized models.
2Measurement precision
If separate independent models are trained for content and style extraction, then specialized processing can be applied to each, but training time and data requirements increase
Solution Approach 1:
By training a single unified model instead of separate models, the patent reduces the total training time required. The model learns to disentangle content and style embeddings simultaneously from audio inputs, eliminating the need for sequential or parallel training of multiple independent models and significantly reducing the overall training duration.
Solution Approach 2:
The universal model architecture handles both content and style extraction tasks within a single training process, reducing the cumulative training time that would be required if separate models were trained. The shared feature extractors and disentangled embedding layers are trained together to perform all extraction functions simultaneously.
3Measurement precision
If separate independent models are used, then resource consumption increases, but model performance can be optimized for each function
Solution Approach 1:
The unified model consolidates computational resources by using a single model structure for both content and style extraction, reducing the total computational load required compared to running separate models. The shared feature extractors and processing layers eliminate redundant computations while maintaining extraction performance through disentangled embedding representations.
Solution Approach 2:
The multi-functional model performs both content and style extraction using a single computational architecture, reducing resource consumption by eliminating the need for separate model instances. The efficient shared feature extractors and disentangled embedding layers process both types of information through unified computational paths, lowering overall energy and computational resource requirements.
Data Source
AI summary
A method includes obtaining, using at least one processing device of an electronic device, an audio input associated with a speaker. The method also includes extracting, using a feature extractor of a trained machine learning model, audio features from the audio input. The method further includes generating (i) one or more content parameter predictions using content embeddings extracted by a content encoder and decoded by a content decoder of the trained machine learning model and (ii) one or more style parameter predictions using style embeddings extracted by a style encoder and decoded by a style decoder of the trained machine learning model. The content embeddings and the style embeddings are based on the audio features of the audio input. The trained machine learning model is trained to generate the one or more content parameter predictions and the one or more style parameter predictions using disentangled content and style embeddings.