Avatar Lip-Sync Reconstruction From Imagined Speech Biosignals

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing lip-sync animation technologies require recorded voice data for generating speaking faces, limiting their use for patients who cannot speak or in quiet situations, and struggle with expressing detailed emotions and nuances, while brain-computer interface systems face challenges like low real-time decoding performance and high costs for invasive measurements.

Innovation Solution

A multimodal biosignal-based system that collects brain waves and electromyography during speaking imagination to generate avatar lip-sync animation, using a lip-sync reconstruction model to predict mouth and facial movements, enabling non-invasive communication of intentions and emotions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If voice-based lip-sync animation is used, then mouth shape reconstruction is achieved, but it cannot be used for patients who cannot speak or in quiet situations

Engineering Contradiction:

Improveapplicability for patients who cannot speakVSAvoiddependency on recorded voice data

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent introduces brain wave signals as an intermediary medium to bridge the gap between user intention and mouth shape reconstruction. Instead of directly requiring voice data, the system uses brain wave signals collected during speaking imagination as a mediator to infer and reconstruct the corresponding mouth movements, enabling the system to function for patients who cannot speak or in quiet environments

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces the mechanical/physical voice recording system with a neurological signal-based system. Instead of capturing actual voice waves and mouth movements through microphones and cameras, the system uses brain wave signals (electrical signals from the brain) to infer and generate the corresponding lip-sync animation, substituting the traditional mechanical voice-based approach with a neurological alternative

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Ease of operation

If brain-computer interface systems are used to read user intentions, then communication without direct speaking is enabled, but real-time decoding performance is low and recognition rate is low

Engineering Contradiction:

Improvecommunication without direct speakingVSAvoidreal-time decoding performance

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

The patent applies preliminary action by pre-training a deep learning model using paired data of brain wave signals and corresponding mouth shape animations during an offline phase. This pre-training creates a comprehensive mapping between brain wave patterns and facial movements, enabling the system to achieve high real-time decoding performance during actual use without requiring complex real-time learning computations

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent uses copying by creating a digital replica or model of the relationship between brain wave signals and mouth movements through deep learning. The trained model serves as a copy of the complex neural mapping, allowing the system to quickly infer mouth shapes from new brain wave inputs in real-time without reprocessing the entire complex relationship from scratch

Inventive Principle:

Principle #26Copying

3Measurement precision

If invasive brain wave measurement is used to improve communication accuracy, then recognition rate improves, but cost increases and usability decreases

Engineering Contradiction:

Improvebrain wave measurement accuracyVSAvoidusability for general public

Core Design Contradiction:

Measurement precisionVSEase of operation

Solution Approach 1:

The patent employs inexpensive, non-invasive EEG headsets that can be easily deployed and discarded or replaced if needed, rather than requiring expensive, complex, invasive measurement systems. These affordable, non-invasive devices make the technology accessible to the general public while still providing sufficient measurement precision for effective communication

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS20250391079A1Device and method for generating avatar lip-sync animation based on multimodal biosignals

Publication Date: 2025.12.25 KOREA UNIV RES & BUSINESS FOUND

AI summary

The present disclosure relates to a device and method for generating avatar lip-sync animation based on multimodal biosignals, The device comprises a multimodal data collection unit configured to collect data including biosignal data including brain waves when a user imagines speaking and image data; a preprocessing unit configured to preprocess the multimodal data; a feature extraction unit configured to extract feature vectors including the user's biosignal feature and facial feature from the preprocessed multimodal data; an avatar generation unit configured to generate an avatar; a lip-sync reconstruction unit configured to predict the mouth shape and facial movement when the user imagines speaking by inputting the extracted feature vectors to a pre-prepared lip-sync reconstruction model; and a lip-sync animation implementation unit for implementing an avatar lip-sync animation by applying the mouth shape and facial movement predicted by the lip-sync reconstruction unit to the avatar generated by the avatar generation unit.