Lyrics Alignment Using Shared Audio-Text Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for lyrics alignment are inefficient for large vocabularies and different languages, particularly due to the complexity of treating each lyrical unit as a separate entity and the lack of flexibility in handling alternative pronunciations and language-specific alphabets, leading to computational inefficiencies and inaccurate alignment.
Innovation Solution
A model that uses contrastive learning to train audio and text encoders to generate embeddings in a shared space, aligning lyrics text with audio by determining the path of highest overall similarities between these embeddings, thus simplifying the training process and improving alignment accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If each lyrical unit is treated as a separate entity with direct probability estimation, then alignment can be performed, but computational complexity increases significantly with larger vocabulary sizes
Solution Approach 1:
The patent segments the lyrical units into character-level tokens rather than treating each complete lyrical unit as a separate entity. This segmentation reduces the vocabulary size from potentially thousands of unique lyrical units to a manageable set of character tokens, thereby reducing computational complexity while maintaining alignment accuracy through the sequential processing of these smaller units.
Solution Approach 2:
The patent changes the parameter of vocabulary representation from complete lyrical units to character-level tokens. This parameter change transforms the problem from estimating probabilities of many rare complete units to estimating probabilities of frequent character sequences, reducing computational complexity while preserving alignment information through the sequential character structure.
2Measurement precision
If phoneme representation is used to represent lyrical units, then alignment accuracy improves, but system complexity increases due to separate conversion tools and multiple pronunciation variants
Solution Approach 1:
The patent merges the text encoding and phoneme conversion functions into a single unified model. Instead of using separate conversion tools that require manual construction and cannot be trained jointly, the model directly processes character-level tokens and learns the mapping to audio representations end-to-end, eliminating system complexity while maintaining alignment accuracy.
Solution Approach 2:
The model performs its own text-to-phoneme conversion internally without relying on external conversion tools. By processing character-level tokens directly and learning the acoustic representations end-to-end, the system serves its own conversion needs, eliminating the complexity of separate phoneme conversion systems and enabling joint training.
3Measurement precision
If lyric transcription approaches are used with decoding schemes, then alignment can be achieved, but system complexity increases and training efficiency decreases
Solution Approach 1:
The patent extracts the alignment function from the complex lyric transcription pipeline and decoding schemes. By directly processing character-level tokens and computing alignment probabilities through the unified model without intermediate transcription steps, the system achieves alignment more efficiently while reducing system complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method includes obtaining lyrics text and audio for a media item and generating, using a first encoder, a first plurality of embeddings representing symbols that appear in the lyrics text for the media item. The method includes generating, using a second encoder, a second plurality of embeddings representing an acoustic representation of the audio for the media item. The method includes determining respective similarities between embeddings of the first plurality of embeddings and embeddings of the second plurality of embeddings and aligning the lyrics text and the audio for the media item based on the respective similarities. The method includes, while streaming the audio for the media item, providing, for display, the aligned lyrics text with the streamed audio.