Open-Vocabulary Wake Word Embeddings for Edge Keyword Spotting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing keyword spotting systems at edge devices are limited by resource constraints, requiring fixed wake words and necessitating transmission of user-recorded audio for model retraining, which affects user experience and resource efficiency.
Innovation Solution
Implement an open-vocabulary keyword spotting system using an MLPMixer-based encoding model and a convolution-based model, allowing for a small footprint user-defined keyword spotting model that processes audio locally without retraining on a server.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional keyword spotting techniques are used with fixed wake words, then the system can be deployed on edge devices with limited resources, but the system cannot recognize custom user-defined wake words
Solution Approach 1:
The system creates an embedding representation (a compact numerical copy) of the user's custom wake word that can be stored and compared against incoming audio. Instead of retraining the entire model, the system copies the essential features of the wake word into an embedding vector that fits within edge device memory constraints.
Solution Approach 2:
The system performs preliminary processing of the user's wake word recording during an enrollment phase, converting it into an embedding representation before actual keyword spotting begins. This preliminary action prepares the custom wake word data in advance, so that during operation, the system only needs to compare incoming audio against the pre-computed embedding rather than performing full model retraining.
2Adaptability or versatility
If user-recorded audio is transmitted to a server for model retraining, then custom wake words can be recognized, but additional bandwidth, power, and time are consumed
Solution Approach 1:
The edge device performs the wake word embedding computation locally using its own processing capabilities and memory resources. The device serves itself by generating and storing the embedding representation without requiring external server assistance, thereby eliminating the need for audio transmission and reducing power consumption associated with communication and cloud processing.
Solution Approach 2:
The system extracts only the essential features of the wake word into a compact embedding representation, separating this essential information from the full audio recording. This extraction allows the system to work with a minimal data representation that fits in edge device memory, avoiding the need to transmit or process large amounts of audio data.
3Measurement precision
If the encoding model stores full audio recordings for wake word detection, then accurate recognition is achieved, but memory requirements exceed edge device capabilities
Solution Approach 1:
The system creates an embedding representation (a compact numerical copy) of the user's custom wake word that can be stored and compared against incoming audio. Instead of retraining the entire model, the system copies the essential features of the wake word into an embedding vector that fits within edge device memory constraints.
Solution Approach 2:
The system performs preliminary processing of the user's wake word recording during an enrollment phase, converting it into an embedding representation before actual keyword spotting begins. This preliminary action prepares the custom wake word data in advance, so that during operation, the system only needs to compare incoming audio against the pre-computed embedding rather than performing full model retraining.
4Adaptability or versatility
If model retraining is performed on a server, then open-vocabulary capabilities are achieved, but latency increases due to transmission and retraining time
Solution Approach 1:
The edge device performs the wake word embedding computation locally using its own processing capabilities and memory resources. The device serves itself by generating and storing the embedding representation without requiring external server assistance, thereby eliminating the need for audio transmission and reducing power consumption associated with communication and cloud processing.
Solution Approach 2:
The system performs preliminary processing of the user's wake word recording during an enrollment phase, converting it into an embedding representation before actual keyword spotting begins. This preliminary action prepares the custom wake word data in advance, so that during operation, the system only needs to compare incoming audio against the pre-computed embedding rather than performing full model retraining.
Data Source
AI summary
A computer-implemented method includes receiving enrollment audio from a user comprising a wake word to be enrolled for the device, preprocessing the enrollment audio to obtain a vector representation along at least a feature dimension and a temporal dimension, inputting the extracted vector representation to a trained encoding model to generate an embedding representation of the enrollment audio, wherein the encoding model includes a plurality of mixing blocks, and wherein the feature dimension and the temporal dimension of an output of a first layer of each mixing block are flipped for inputting to a second layer of the mixing block, and storing the generated embedding representation in a memory for use in detecting input of the enrolled wake word.