Neural Network Gesture Generation for Virtual Agents
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current game engines and animation engines face challenges in aligning human-like movements of virtual agents with their associated speech or text transcripts, making it difficult to create realistic virtual agents with social and emotional intelligence.
Innovation Solution
A neural network-based method for interactively generating emotive gestures for virtual agents, using a transformer network to align natural language inputs with biomechanical features, considering intended tasks, emotions, gender, and handedness, and training the model with multi-head self-attention and positional encoding to produce realistic and emotionally expressive gestures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If current game engines and animation engines are used to generate human-like movements, then movements can be produced, but alignment with speech or text transcripts is difficult
Solution Approach 1:
The patent replaces traditional mechanical animation engines with a neural network-based system that uses word embeddings, attention mechanisms, and sequence-to-sequence modeling to generate gestures. This substitution enables direct alignment between linguistic content and gestural output, achieving 97% reduction in pose error while maintaining system manageability through software-based processing.
Solution Approach 2:
The system transforms discrete speech/text parameters into continuous gestural parameters through neural network processing. By converting word embeddings into sequence representations and mapping them to joint rotation sequences, the system achieves precise alignment between linguistic intent and physical gesture, resolving the alignment precision problem.
2Reliability
If neural network models are used to generate emotive gestures, then gesture alignment and emotional expressiveness improve, but computational complexity increases
Solution Approach 1:
The patent segments the gesture generation task into distinct processing stages: word embedding generation, attention-based feature extraction, sequence-to-sequence transformation, and final gesture output. This segmentation allows each component to be optimized independently, achieving 91% gesture plausibility while managing computational complexity through modular architecture.
Solution Approach 2:
The patent introduces intermediate representations including word embeddings, attention weights, and latent sequence representations that mediate between input text and output gestures. These intermediaries enable the system to capture emotional and contextual nuances, improving gesture realism without requiring direct complex mappings.
3Loss of information
If traditional animation methods are used, then computational resources are conserved, but emotional expressiveness and gesture alignment are poor
Solution Approach 1:
The patent performs preliminary processing of input text into word embeddings and contextual representations before gesture generation. This preliminary action captures emotional and semantic information early in the pipeline, ensuring that emotional content is preserved and efficiently utilized during subsequent gesture synthesis, reducing information loss.
Solution Approach 2:
The system employs attention mechanisms that provide feedback loops between input text representations and generated gesture sequences. This feedback enables the model to refine emotional expressiveness by continuously aligning gesture output with the emotional content of the input, maximizing emotional information retention.
Data Source
AI summary
Systems and methods of the present invention for gesture generation include: receiving a sequence of one or more word embeddings, one or more attributes, a gesture generation machine learning model; providing the sequence of one or more word embeddings and the one or more attributes to the gesture generation machine learning model; and providing the second emotive gesture of the virtual agent from the gesture generation machine learning model. The gesture generation machine learning model is configured to: produce, via an encoder, an output based on the one or more word embeddings; generate one or more encoded features based on the output and the one or more attributes; and produce, via a decoder, a emotive gesture based on the one or more encoded features and the preceding emotive gesture. Other aspects, embodiments, and features are also claimed and described.