Real-time Audio Classification via Sliding Window
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing audio segmentation and diarization systems suffer from delays in speech content classification, leading to poor user experience and inaccurate labeling, as they only output labels after completing the segmentation process, which is unsuitable for real-time applications like live streaming.
Innovation Solution
The approach involves applying classifiers directly to digital data from the audio stream for real-time classification, using a segmentation-by-classification method with deep neural networks to identify semantic classes and transitions, enabling immediate labeling of speech content, even in multi-speaker and multi-language scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If clustering techniques are used for speech segmentation and diarization, then the system can process audio signals, but the speech content classification is delayed until segmentation is complete
Solution Approach 1:
The patent divides the audio stream into fixed-size frames and processes each frame independently through the classifier, enabling parallel processing and real-time classification output without waiting for complete segmentation. This segmentation approach allows the system to output classification labels for each frame as it is processed, eliminating the delay inherent in traditional clustering-based systems that must complete entire segmentation before classification.
2Reliability
If classification is performed after complete segmentation, then accurate clustering can be achieved, but real-time labeling is not possible
Solution Approach 1:
The patent extracts acoustic features and applies classification models to each audio frame as it becomes available, performing classification actions preliminarily rather than waiting for complete segmentation. This preliminary classification enables real-time labeling while maintaining accuracy through the use of robust feature extraction and trained classifiers that can operate effectively on individual frames.
3Loss of information
If traditional segmentation approaches are used, then comprehensive audio analysis is achieved, but computational delays accumulate in live streaming
Solution Approach 1:
The patent implements continuous classification by processing audio frames in a continuous stream, where each frame is classified as it arrives without interruption or accumulation of delays. The system maintains continuous useful action by constantly analyzing incoming audio data through feature extraction and classification, ensuring that speech content analysis occurs in real-time without the computational delays that accumulate in traditional batch-processing approaches.
Data Source
AI summary
In an embodiment, the disclosed technologies include automatically recognizing speech content of an audio stream that may contain multiple different classes of speech content, by receiving, by an audio capture device, an audio stream; outputting, by one or more classifiers, in response to an inputting to the one or more classifiers of digital data that has been extracted from the audio stream, score data; where a score of the score data indicates a likelihood that a particular time segment of the audio stream contains speech of a particular class; where the one or more classifiers use one or more machine-learned models that have been trained to recognize audio of one or more particular classes to determine the score data; using a sliding time window process, selecting particular scores from the score data; using the selected particular scores, determining and outputting one or more decisions as to whether one or more particular time segments of the audio stream contain speech of one or more particular classes; where the one or more decisions are outputted within a real-time time interval of the receipt of the audio stream; where the one or more decisions are used by downstream processing of the audio stream to control any one or more of the following: labeling the audio stream, segmenting the audio stream, diarizing the audio stream.