Viseme Animation Curve Modulation for Realistic Singing Styles
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for transforming vocal performances into visual performances, such as lip-sync timing and mouth shapes for singing, are ill-suited and fail to accurately represent the additional challenges of melody and rhythm in singing, particularly in diverse singing styles.
Innovation Solution
A system and method that modulates animation curves using Melodic-accent (Ma) and Pitch-sensitivity (Ps) parameters to dynamically represent singing styles, considering the distinct roles of vowels and consonants in singing, with a four-dimensional (4D) space encompassing Jaw (Ja) and Lip (Li) parameterization, and utilizes a neural network for vowel modification and larynx movements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If speech-based viseme animation is used for singing, then lip-sync timing can be achieved, but the animation fails to accurately represent melody and rhythm characteristics
Solution Approach 1:
The patent introduces Ma (melodic accent) and Ps (pitch sensitivity) parameters to modulate viseme animation curves, transforming the speech-based animation system into one that can represent singing characteristics. These parameters change the temporal and amplitude properties of animation curves to reflect melodic and rhythmic features, resolving the contradiction between accurate singing representation and adaptability to diverse singing styles.
Solution Approach 2:
The system dynamically adjusts viseme animation curves based on extracted acoustic features from the singing audio. By making the animation parameters dynamic rather than static, the system can adapt to different singing styles, melodies, and rhythms, thereby improving both accuracy and versatility simultaneously.
2Measurement precision
If traditional lip-sync methods are used, then basic mouth shape synchronization is achieved, but the system cannot capture the distinct roles of vowels and consonants in singing
Solution Approach 1:
The patent segments the viseme animation into vowel and consonant components, allowing independent control and modulation of each. This segmentation enables accurate representation of their distinct roles in singing while providing a structured framework that manages complexity through modular organization of the animation control system.
Solution Approach 2:
The Ma and Ps parameters serve multiple functions: they modulate both vowel and consonant visemes, control temporal and amplitude characteristics, and adapt to diverse singing styles. This multi-functionality reduces the need for separate control mechanisms, managing system complexity while achieving precise vowel-consonant representation.
3Reliability
If speech animation curves are used directly, then processing is simple, but the animation lacks physiological plausibility for singing performance
Solution Approach 1:
The system transforms static speech animation curves into dynamic singing-specific curves by applying Ma and Ps parameter modulation. This dynamic adjustment ensures physiological plausibility by reflecting the actual temporal and amplitude characteristics of singing articulation, while the parameter-based approach manages complexity through a systematic modulation framework.
Solution Approach 2:
By changing the parameters that control animation curves (introducing Ma and Ps), the system achieves physiological plausibility for singing without requiring a completely new animation system. The parameter changes allow speech curves to be adapted into singing curves, balancing reliability improvement with controlled complexity increase.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
A system and method of modulating animation curves based on audio input. The method including: identifying phonetic features for a plurality of visemes in the audio input; determining viseme animation curves based on parameters representing a spatial appearance of the plurality of visemes; modulating the viseme animation curves based on melodic accent, pitch sensitivity, or both, based on the phonetic features; and outputting the modulated animation curves.