Word Boundary Detection for On-Device Speech Recognition Batching

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Modern automatic speech recognition (ASR) systems face challenges in balancing low latency and high accuracy while minimizing power consumption on-device, particularly in streaming speech recognition.

Innovation Solution

The method involves using a word boundary detection model to predict word boundaries in input audio frames, allowing for batching of frames into whole words, which are then processed in parallel by a speech recognition model to generate recognition results.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If speech recognition processes audio frames in a streaming fashion to achieve low latency, then response time is improved, but processing accuracy and power efficiency deteriorate due to frequent interruptions and inability to batch process

Engineering Contradiction:

ImprovelatencyVSAvoidspeech recognition accuracy

Core Design Contradiction:

Loss of timeVSReliability

Solution Approach 1:

The system performs preliminary voice activity detection and word boundary prediction on incoming audio frames before committing to processing decisions. This allows the system to anticipate when complete words will be available and prepare batching strategies in advance, resolving the contradiction by planning ahead rather than reacting to each frame individually

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The audio stream is segmented into word-level batches based on predicted word boundaries rather than being processed frame-by-frame or in fixed-size chunks. This segmentation allows the system to process complete semantic units (words) in parallel batches, improving both accuracy through complete context and efficiency through batched computation while maintaining low latency by processing segments as they become available

Inventive Principle:

Principle #1Segmentation

2Loss of time

If speech recognition processes audio frames individually in streaming mode, then latency is reduced, but power consumption increases due to inability to batch process operations

Engineering Contradiction:

ImprovelatencyVSAvoidpower consumption

Core Design Contradiction:

Loss of timeVSUse of energy by moving object

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Multiple audio frames containing portions of the same word or adjacent words are merged into single processing batches. By combining frames at the word boundary level rather than processing them individually, the system achieves parallel batched processing that reduces computational overhead and power consumption while maintaining the ability to output results as soon as complete words are detected

Inventive Principle:

Principle #5Merging (Combining)

3Productivity

If the system batches audio frames to improve power efficiency, then processing throughput is improved, but latency increases due to waiting for complete batches

Engineering Contradiction:

Improveprocessing throughputVSAvoidlatency

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The batching strategy is dynamic rather than static. Batch sizes and formation are adjusted in real-time based on voice activity detection results and word boundary predictions. When speech is continuous, larger batches are formed for efficiency; when speech pauses or new words begin, batches are formed and processed immediately, ensuring throughput optimization without excessive latency accumulation

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

Inventive Principle:

Principle #10Preliminary action

4Device complexity

If fixed-size batches are used for processing, then implementation simplicity is improved, but speech recognition quality deteriorates because batches may split words or combine multiple words

Engineering Contradiction:

Improvebatching implementation complexityVSAvoidspeech recognition quality

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12322383B2Predicting word boundaries for on-device batching of end-to-end speech recognition models

Publication Date: 2025.06.03 GOOGLE LLC

AI summary

A method includes receiving a sequence of input audio frames corresponding to an utterance captured by a user device, the utterance including a plurality of words. For each input audio frame, the method includes predicting, using a word boundary detection model configured receive the sequence of input audio frames as input, whether the input audio frame is a word boundary. The method includes batching the input audio frames into a plurality of batches based on the input audio frames predicted as word boundaries, wherein each batch includes a corresponding plurality of batched input audio frames. For each of the plurality of batches, the method includes processing, using a speech recognition model, the corresponding plurality of batched input audio frames in parallel to generate a speech recognition result.