Streaming Hotword Spotting With SVDF Memory-Constrained Decoding

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing hotword detection systems in streaming audio are inefficient due to separate training of signal processing, neural network acoustic encoder, and hand-designed decoder components, leading to increased complexity and resource consumption, and struggle with detecting hotwords across different languages and locales.

Innovation Solution

An end-to-end memorized neural network with sequentially-stacked Single Value Decomposition Filter (SVDF) layers is trained to jointly optimize both encoding and decoding components, incorporating a fixed memory capacity proportional to the time required to speak a hotword, allowing detection of multiple hotwords and variations across languages and locales.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If separate training of signal processing, neural network acoustic encoder, and hand-designed decoder components is used, then each component can be optimized independently, but the overall system complexity increases and resource consumption increases

Engineering Contradiction:

Improvecomponent optimizationVSAvoidsystem complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent merges the signal processing front end, neural network acoustic encoder, and decoder into a single jointly-trained end-to-end system. This integration allows the system to optimize the entire hotword detection pipeline simultaneously, reducing the complexity of managing multiple independently trained components while maintaining the ability to handle diverse linguistic variations through unified training.

Inventive Principle:

Principle #5Merging (Combining)

2Reliability

Engineering Contradiction:

Improvecomponent optimizationVSAvoidresource consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

Inventive Principle:

Principle #5Merging (Combining)

3Ease of manufacture

If hand-designed decoder component is used, then decoding can be performed with custom logic, but the system struggles with detecting hotwords across different languages and locales

Engineering Contradiction:

Improvecustom decoding logicVSAvoidlanguage and locale adaptability

Core Design Contradiction:

Ease of manufactureVSAdaptability or versatility

Solution Approach 1:

The patent employs parameter changes by training the decoder with diverse linguistic data representing multiple languages and locales. The decoder's parameters are adjusted during training to capture variations in pronunciation, phonetics, and language-specific patterns, enabling the system to adapt to different languages and regions without requiring manual reconfiguration of decoding logic.

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The jointly-trained end-to-end system achieves universality by processing multiple languages and locales through a single unified model. The system can detect hotwords across different linguistic contexts using the same architectural framework, eliminating the need for separate hand-designed decoders for each language while maintaining custom decoding capabilities through shared learning.

Inventive Principle:

Principle #6Universality (Multi-functionality)

4Loss of time

If fixed memory capacity proportional to hotword speaking time is used, then the neural network can focus on relevant time window, but the network must efficiently manage limited memory for accurate detection

Engineering Contradiction:

Improveprocessing time windowVSAvoiddetection accuracy

Core Design Contradiction:

Loss of timeVSMeasurement precision

Solution Approach 1:

The patent applies segmentation by dividing the audio input into discrete time frames and using a sliding window approach to process segments of the audio stream. The fixed memory capacity is allocated to store and process only the relevant time window corresponding to the expected hotword duration, while the segmentation allows the system to efficiently manage memory by discarding older frames outside the attention window, thus balancing processing efficiency with detection accuracy.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS20260088023A1End-to-end streaming keyword spotting

Publication Date: 2026.03.26 GOOGLE LLC

AI summary

A method for training hotword detection includes receiving a training input audio sequence including a sequence of input frames that define a hotword that initiates a wake-up process on a device. The method also includes feeding the training input audio sequence into an encoder and a decoder of a memorized neural network. Each of the encoder and the decoder of the memorized neural network include sequentially-stacked single value decomposition filter (SVDF) layers. The method further includes generating a logit at each of the encoder and the decoder based on the training input audio sequence. For each of the encoder and the decoder, the method includes smoothing each respective logit generated from the training input audio sequence, determining a max pooling loss from a probability distribution based on each respective logit, and optimizing the encoder and the decoder based on all max pooling losses associated with the training input audio sequence.