Automated Assistant Language Detection via Entropy Scoring

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing speech recognition engines often fail to handle multilingual environments effectively, requiring users to manually switch languages or leading to excessive resource usage when encountering unsupported languages, and may not automatically determine the user's language for processing and responding.

Innovation Solution

An automated assistant uses multiple machine learning models trained in different languages to analyze audio inputs, calculate entropy scores based on candidate phrases, and select the most appropriate language for processing and responding, eliminating the need for manual language switching and reducing resource usage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If a speech recognition engine is configured to recognize speech in only a single language, then the engine's simplicity and lower computational resource requirements are maintained, but the system becomes unusable or provides erroneous output when a spoken utterance is received in an additional language

Engineering Contradiction:

Improvespeech recognition engine configurationVSAvoidlanguage support capability

Core Design Contradiction:

Device complexityVSAdaptability or versatility

Solution Approach 1:

The speech recognition system is designed to handle multiple languages through a universal architecture that can process utterances in any supported language. The system maintains a single speech recognition engine that can be dynamically configured to recognize speech in different languages based on the detected language of the incoming utterance, rather than requiring separate engines for each language.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The system dynamically determines the language of each spoken utterance in real-time and adapts the speech recognition engine's language configuration accordingly. This dynamic adaptation allows the system to switch between languages automatically without requiring manual user intervention or pre-configuration for each language scenario.

Inventive Principle:

Principle #15Dynamics

2Adaptability or versatility

If a speech recognition engine is configured to recognize speech in multiple languages, then language versatility is improved, but the system requires explicit user designation of the language to be utilized, leading to excess usage of computational and network resources

Engineering Contradiction:

Improvelanguage support capabilityVSAvoidcomputational resource usage

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary language identification on incoming spoken utterances before initiating full speech recognition processing. By detecting the language of the utterance first, the system can pre-configure the speech recognition engine with the appropriate language model, avoiding the need to process multiple languages simultaneously and reducing computational overhead.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system automatically detects and determines the language of each utterance without requiring explicit user designation or manual language selection. This self-service approach eliminates the need for users to interact with language selection interfaces and allows the system to autonomously optimize its processing for the detected language.

Inventive Principle:

Principle #25Self-service

3Measurement precision

If a user must manually designate a default language for speech recognition, then language recognition accuracy is improved, but the ease of operation deteriorates and leads to excess usage of computational resources for interface rendering and processing

Engineering Contradiction:

Improvelanguage recognition accuracyVSAvoidlanguage switching operation

Core Design Contradiction:

Measurement precisionVSEase of operation

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Inventive Principle:

Principle #25Self-service

4Measurement precision

If the automated assistant processes further spoken utterances in the single supported language after a failure, then language recognition accuracy is maintained, but productivity deteriorates due to additional usage of computational and network resources

Engineering Contradiction:

Improvelanguage recognition accuracyVSAvoidresource efficiency

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Inventive Principle:

Principle #15Dynamics

Data Source

PatentEP3642834B1Automatically determining language for speech recognition of spoken utterance received via an automated assistant interface

Publication Date: 2024.08.21 GOOGLE LLC

Figure 1
Figure 2
Figure 3A~3B

AI summary

Implementations relate to determining a language for speech recognition of a spoken utterance, received via an automated assistant interface, for interacting with an automated assistant. In various implementations, audio data indicative of a voice input that includes a natural language request from a user may be applied as input across multiple speech-to-text ("STT") machine learning models to generate multiple candidate speech recognition outputs. Each STT machine learning model may trained in a particular language. For each respective STT machine learning model of the multiple STT models, the multiple candidate speech recognition outputs may be analyzed to determine an entropy score for the respective STT machine learning model. Based on the entropy scores, a target language associated with at least one STT machine learning model of the multiple STT machine learning models may be selected. The automated assistant may respond to the request using the target language.