Parallel Expert Model Averaging for Speech Recognition

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Continual learning for end-to-end automatic speech recognition faces challenges such as catastrophic forgetting, where models perform poorly on original data after fine-tuning on new data domains, and existing solutions either add complexity by maintaining multiple parameter sets or require unavailable original data.

Innovation Solution

The approach involves parallelizing the fine-tuning process by training multiple domain expert models in parallel and averaging their parameters to create an average-of-domain-experts (AoDE) model, which maintains a single model for inference while avoiding catastrophic forgetting, using techniques like linear or non-linear interpolation of model parameters.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If models are fine-tuned on new data domains serially, then performance on new domains is improved, but performance on original domains deteriorates due to catastrophic forgetting

Engineering Contradiction:

Improveperformance on new domainsVSAvoidperformance on original domains

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The training process is segmented into parallel fine-tuning tasks where multiple domain expert models are trained simultaneously on different domain datasets, each maintaining specialized knowledge without interfering with others

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Multiple domain expert models trained in parallel are merged through parameter averaging to create a single generalized model that combines knowledge from all domains while preventing catastrophic forgetting

Inventive Principle:

Principle #5Merging (Combining)

2Adaptability or versatility

If multiple parameter sets are maintained for different domains, then performance on all domains is improved, but system complexity increases

Engineering Contradiction:

Improveperformance on all domainsVSAvoidnumber of parameter sets

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

Multiple domain-specific parameter sets are merged through parameter averaging into a single unified model, reducing complexity while maintaining multi-domain performance capabilities

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

A single generalized model is created that serves multiple domains universally through parameter averaging, eliminating the need to maintain separate parameter sets for different domains

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20240290319A1Systems and methods for continual learning for end to-end automatic speech recognition

Publication Date: 2024.08.29 JPMORGAN CHASE BANK NA

AI summary

In some aspects, the techniques described herein relate to a method including: providing, to a parallel model training platform, a plurality of domain datasets; training, by the parallel model training platform, a plurality of generalist models in parallel, wherein each generalist model of the plurality of generalist models is trained in parallel using a corresponding one of the plurality of domain datasets, and wherein training the plurality of generalist models in parallel generates a corresponding expert model for each generalist model in the plurality of generalist models; executing, by the parallel model training platform, a model parameter averaging process, wherein the model parameter averaging process take each corresponding expert model as input; and generating, by the parallel model training platform and as output of the model parameter averaging process, an average-of-domain-experts (AoDE) model.