Spoken Language Model With Quantized Prosody Feature Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative spoken language models, such as w2v-BERT and HuBert, are optimized for discriminative tasks and lack comprehensive representation of non-semantic speech features like prosody, loudness, and timbre, while multimodal models like SpeechGPT are limited by the scarcity of textual data.
Innovation Solution
A method that enhances generative spoken language models by computing the difference between encoded unit sequences of original and normalized speech data to extract non-semantic features, encoding these features quantized representations, and integrating them with discrete phoneme-related units in a deep learning model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If generative spoken language models are trained solely on speech data to avoid textual data scarcity, then model applicability to low-resource languages is improved, but the ability to represent non-semantic speech features comprehensively deteriorates
Solution Approach 1:
The speech data is segmented into discrete phoneme-related units and non-semantic features are extracted as separate components. This segmentation allows the model to process semantic and non-semantic information independently while maintaining both capabilities in the generative framework.
Solution Approach 2:
The patent adds a new dimension to the speech representation by introducing non-semantic feature sequences alongside phoneme sequences. This multi-dimensional approach enriches the speech data representation, enabling comprehensive capture of both semantic and non-semantic aspects without requiring textual data.
2Measurement precision
If prosodic information is incorporated through unit duration and fundamental frequency in pGSLM, then rhythmic representation is improved, but comprehensive capture of non-semantic aspects such as loudness and timbre deteriorates
Solution Approach 1:
Non-semantic features are extracted from the speech data as distinct components separate from the phoneme sequences. This extraction process isolates prosodic and other non-semantic information, allowing comprehensive capture of features like loudness, timbre, and rhythm without interfering with the phoneme-based semantic representation.
3Productivity
If discrete units are used for speech transformation in GSLM and STELA models, then processing efficiency is improved, but comprehensive representation of speech features deteriorates
Solution Approach 1:
The patent creates a composite representation by combining discrete phoneme-related units with non-semantic feature sequences. This composite structure maintains the processing efficiency of discrete units while incorporating continuous non-semantic features, achieving both computational tractability and comprehensive feature representation.
Data Source
Figure 1
Figure 2
AI summary
The method comprises obtaining at least one non-semantic feature including prosodic information of original speech data by computing a difference between an encoded unit sequence of the original speech data and an encoded unit sequence of normalized speech data; encoding said at least one non-semantic feature to produce a quantized representation of the at least one non-semantic feature; and inputting the quantized representation and discrete phoneme-related units into a deep learning model to generate a speech sequence representing the discrete phoneme-related units and the at least one non-semantic feature.