Generative AI Training Data for ASR Punctuation Restoration

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Automatic Speech Recognition (ASR) systems generate unpunctuated text, which affects readability and performance in downstream tasks, due to the lack of effective training data for punctuation restoration models, as written punctuated texts differ significantly from ASR-generated texts.

Innovation Solution

Improving the training data for punctuation restoration models by using sequence labeling language models and generative AI models like GPT to generate text similar to ASR outputs, fine-tuning the models with gradients from development datasets to align with the style of spoken speech, and augmenting training data with noise simulations to better reflect ASR text characteristics.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If written punctuated texts are used for training punctuation restoration models, then training data availability is improved, but the text style does not match ASR-generated texts

Engineering Contradiction:

Improvetraining data availabilityVSAvoidtext style match

Core Design Contradiction:

Quantity of substanceVSAdaptability or versatility

Solution Approach 1:

The patent uses a generative AI model to copy and synthesize training data that replicates the specific characteristics of ASR-generated texts. The generative model creates synthetic punctuated text samples that mirror the stylistic features, errors, and patterns of actual ASR outputs, enabling the PR model to be trained on data that accurately reflects its input distribution without requiring manual annotation of ASR texts.

Inventive Principle:

Principle #26Copying

2Adaptability or versatility

If ASR-generated texts are used for training, then text style match is improved, but training data quantity is insufficient

Engineering Contradiction:

Improvetext style matchVSAvoidtraining data quantity

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The system employs a generative AI model that serves itself by automatically generating training data from the ASR text distribution. The generative model learns from existing ASR outputs and synthesizes additional training samples, effectively multiplying the available data quantity while maintaining stylistic accuracy. This self-service approach eliminates the need for human annotators to create training data from ASR texts.

Inventive Principle:

Principle #25Self-service

3Measurement precision

If manual annotation of ASR texts is used, then training data quality is improved, but processing time and cost increase

Engineering Contradiction:

Improvetraining data qualityVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent replaces the mechanical process of manual human annotation with an automated generative AI model. The generative model automatically produces punctuated training data from ASR text inputs, substituting human labor with an automated system that operates continuously without time loss. This mechanical substitution maintains high training data quality while dramatically reducing processing time and operational costs.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20250022459A1Improved training methods for language models using data generation and reinforcement learning

Publication Date: 2025.01.16 ADOBE INC

AI summary

The disclosed method generates helpful training data for a language model, for example, a model implementing a punctuation restoration task, for real-world ASR texts. The method uses a reinforcement learning method using a generative AI model to generate additional data to train the language model. The method allows the generative AI model to learn from real-world ASR text to generate more effective training examples based on gradient feedback from the language model.