Multilingual ASR with Language-Dependent Adapters for Low Latency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing automatic speech recognition (ASR) models are primarily designed for single languages and struggle to maintain accuracy and low latency when recognizing multiple languages simultaneously.

Innovation Solution

A streaming and non-streaming multilingual ASR model is developed, incorporating a causal encoder with an initial stack of multi-head attention layers, a non-causal encoder with a final stack of multi-head attention layers, and language-dependent adapter (LDA) modules. These LDA modules are configured to receive a language ID vector, activating specific language-dependent weights for each native language, thereby enabling the model to process multiple languages effectively.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a single-language ASR model is used, then recognition accuracy is high, but multilingual capability is lost

Engineering Contradiction:

Improvemultilingual capabilityVSAvoidrecognition accuracy

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The model segments language processing by inserting separate LDA modules for different languages at specific positions within the encoder stack. Each LDA module contains language-specific weights that are activated based on the input language, allowing the model to process multiple languages while maintaining language-specific accuracy characteristics.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies local quality by making different parts of the model language-specific. The LDA modules are inserted at specific positions (after certain encoder layers) and contain language-dependent weights that are tailored to specific languages. This allows different regions of the model to have different language processing characteristics, improving both multilingual capability and per-language accuracy.

Inventive Principle:

Principle #3Local quality

2Adaptability or versatility

If multiple languages are supported in a single model, then versatility improves, but model complexity increases

Engineering Contradiction:

Improvemultilingual capabilityVSAvoidmodel complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The base encoder-decoder model serves as a universal foundation that handles common speech recognition tasks across all languages. The LDA modules are added as optional, language-specific components that are only activated when needed. This multi-functionality approach allows the model to handle both general speech recognition and language-specific nuances without requiring completely separate models for each language.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The LDA modules are nested within the existing encoder-decoder architecture at specific positions. Rather than creating separate parallel models, the language-specific adapters are embedded within the universal model structure, with each LDA module containing language-specific weights nested within the broader multilingual model framework.

Inventive Principle:

Principle #7Nested doll (Nesting)

3Measurement precision

If language-specific adapters are added, then recognition quality improves, but computational overhead increases

Engineering Contradiction:

Improverecognition qualityVSAvoidcomputational overhead

Core Design Contradiction:

Measurement precisionVSPower

Solution Approach 1:

Instead of applying full language-specific processing to all inputs, the model uses partial action by selectively activating LDA modules based on the detected language. The language identification component determines which LDA modules to activate, applying language-specific processing only when and where needed, rather than uniformly across all processing paths.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20250078830A1Adapter Finetuning with Teacher Pseudo-Labeling for Tail Languages in Streaming Multilingual ASR

Publication Date: 2025.03.06 GOOGLE LLC

AI summary

A method includes receiving a sequence of acoustic frames characterizing a spoken utterance in a particular native language. The method also includes generating a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames by a causal encoder that includes an initial stack of multi-head attention layers. The method also includes generating a second higher order feature representation for a corresponding first higher order feature representation by a non-causal encoder that includes a final stack of multi-head attention layers. The method also includes receiving, as input at each corresponding language-dependent adapter (LDA) module, a language ID vector identifying the particular native language to activate corresponding language-dependent weights specific to the particular native language. The method also includes generating a first probability distribution over possible speech recognition hypotheses by a decoder.