Embedded Sound Event Detection Using Audio Prototypes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing sound event detection (SED) models struggle with generalizability and computational complexity, making them unsuitable for real-time operation on embedded devices and adaptation to changing acoustic environments.
Innovation Solution
A lightweight parallel audio neural network architecture and contrastive language-audio pretraining model for real-time sound event detection, using offline multimodal query prototypes and few-shot learning to adapt to new sound events without extensive retraining.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If audio foundation models with large language models are used for sound event detection, then generalization capability to unseen sound classes is improved, but computational complexity and resource requirements increase
Solution Approach 1:
The system segments the audio foundation model into separate audio and text encoders that operate independently but share a common embedding space. This segmentation allows the model to process audio inputs without requiring the full complexity of a multimodal LLM, reducing computational overhead while maintaining generalization capabilities through the contrastively trained audio encoder
Solution Approach 2:
The invention extracts and utilizes only the audio encoder component from the contrastive language-audio pretraining model, separating it from the text encoder and LLM components. This extraction allows the system to leverage the generalization capabilities learned during contrastive pretraining while eliminating the computational burden of running large language models on resource-constrained embedded devices
2Device complexity
If existing SED models are used for closed-form recognition, then model structure is simple, but adaptability to new or unseen sound classes deteriorates
Solution Approach 1:
The system performs preliminary contrastive pretraining of the audio encoder on large-scale audio-text datasets before deployment. This preliminary action equips the encoder with generalization capabilities to handle unseen sound classes, while the actual inference stage maintains simplicity by only requiring embedding extraction and similarity computation without complex model structures
Solution Approach 2:
The invention changes the training objective parameters from traditional classification loss to contrastive loss during pretraining. This parameter change enables the audio encoder to learn robust representations that generalize to unseen classes, while the inference process remains computationally simple by only requiring similarity computation in the learned embedding space
3Adaptability or versatility
If continuously retraining models is performed to accommodate changing classes, then adaptability to new acoustic conditions is improved, but time consumption and computational cost increase
Solution Approach 1:
The system creates query prototypes that serve as compact representations of sound classes, which can be quickly generated and updated without retraining the entire model. These prototype copies allow the system to adapt to new acoustic conditions by simply adding new prototype vectors to the database, avoiding the time-consuming process of continuous model retraining
Data Source
AI summary
A method for real-time sound event detection on an embedded device includes pretraining a contrastive language-audio pretraining model as an audio foundation model and preparing offline multimodal query prototypes for sound events of interest. The pretrained model and query prototypes are deployed on an embedded device. The device receives an input audio stream and extracts audio embeddings using the pretrained model. Similarity scores are calculated between the extracted audio embeddings and the prepared query prototypes. The presence of a sound event is determined based on the calculated similarity scores, and a real-time sound event detection result is output. The system includes a memory storing the pretrained model and query prototypes, an audio input interface, and a processor configured to perform the extraction, calculation, determination, and output operations. A non-transitory computer-readable medium stores instructions that, when executed, cause a processor to perform the method.