Spoken Language Model With Quantized Prosody Feature Encoding

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing generative spoken language models struggle to capture non-semantic features such as prosody, loudness, and timbre comprehensively, limiting their effectiveness in tasks like text-to-speech synthesis and voice conversion.

Innovation Solution

A method that enhances generative spoken language models by computing the difference between encoded unit sequences of original and normalized speech data to extract non-semantic features, encoding these features quantizedly, and integrating them with discrete phoneme-related units in a deep learning model.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If generative spoken language models use only discrete phoneme-related units, then the model structure remains simple, but the ability to capture non-semantic features such as prosody, loudness, and timbre is insufficient

Engineering Contradiction:

Improvecapability to capture non-semantic featuresVSAvoidmodel structure complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The speech representation is segmented into two distinct components: discrete phoneme-related units for semantic content and quantized non-semantic features for prosodic information. This segmentation allows the model to process semantic and non-semantic aspects separately while maintaining overall simplicity

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The model uses a composite representation combining discrete phoneme tokens and quantized non-semantic feature vectors. This composite structure integrates multiple types of information (semantic and non-semantic) in a unified framework without significantly increasing complexity

Inventive Principle:

Principle #40Composite materials

2Reliability

If continuous non-semantic features are used directly in the model, then the representation is comprehensive, but the efficiency for next-token prediction tasks is reduced

Engineering Contradiction:

Improverepresentation comprehensivenessVSAvoidnext-token prediction efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The continuous non-semantic features are transformed into discrete quantized representations through vector quantization. This parameter change from continuous to discrete domain makes the features more suitable for efficient next-token prediction while preserving the comprehensive representation of prosodic information

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

A vector quantization module serves as an intermediary between continuous non-semantic features and the discrete token-based model input. This intermediary transforms continuous features into discrete quantized vectors that can be efficiently processed by the language model

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20250391403A1Method for enhancing a generative spoken language model

Publication Date: 2025.12.25 ORANGE SA

AI summary

A method for enhancing a generative spoken language model. The method includes: obtaining at least one non-semantic feature including prosodic information of original speech data by computing a difference between an encoded unit sequence of the original speech data and an encoded unit sequence of normalized speech data; encoding the at least one non-semantic feature to produce a quantized representation of the at least one non-semantic feature; and inputting the quantized representation and discrete phoneme-related units into a deep learning model to generate a speech sequence representing the discrete phoneme-related units and the at least one non-semantic feature.