Audio Object Separation Mask for Low-Latency Real-Time Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional audio object separation technologies based on deep learning face challenges in achieving real-time separation due to high latency and memory requirements, making them unsuitable for multimedia devices with strict latency constraints and varying memory capacities.
Innovation Solution
An electronic device employs a neural network model to process audio data in the frequency domain, utilizing fully connected and activation layers, query and value generation layers, and decoding layers to generate an audio object separation mask, enabling real-time separation without relying on past data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If self-attentive gated RNNs are used for audio object separation, then audio object can be separated from past audio data in real time, but the latency is 138 ms at minimum which is too high for multimedia devices
Solution Approach 1:
The patent segments the audio processing task into frequency domain transformation, encoding, attention mechanism, and decoding stages. By processing audio data in the frequency domain rather than time domain, and using a streamlined neural network architecture with separate encoding and attention layers, the system achieves real-time processing with reduced latency while maintaining separation performance.
Solution Approach 2:
The patent transforms audio data from time domain to frequency domain using Fourier transform, changing the representation parameters of the audio signal. This parameter transformation enables more efficient processing with reduced computational complexity and latency, while the frequency domain representation preserves the essential characteristics needed for audio object separation.
2Power
If local self-attention is used to reduce calculation amount, then voice can be reinforced in real time, but 29 to 32 past audio frames must be stored in real time which limits application to various multimedia devices
Solution Approach 1:
The patent extracts only the necessary historical information needed for audio object separation and processes it through the attention mechanism, rather than storing and processing all past audio frames. The encoding layer extracts essential features from the frequency domain audio data, and the attention mechanism selectively attends to relevant temporal patterns, reducing memory requirements while maintaining separation quality.
Solution Approach 2:
The patent moves the processing from time domain to frequency domain, changing the dimension of analysis. This dimensional transformation allows the system to process audio data more efficiently by operating on frequency components rather than temporal sequences, reducing the computational burden and memory requirements for real-time processing.
3Measurement precision
If non-casual system is used for audio object separation, then object separation performance on entire target audio is improved, but it is structurally impossible to separate audio object in real time since future audio data is necessary
Solution Approach 1:
The patent inverts the conventional approach by using a casual system that processes audio data in real-time from past to present, rather than requiring future data. The attention mechanism is designed to attend to past and current audio frames without needing future frames, achieving real-time processing while maintaining effective separation performance through the frequency domain transformation and efficient network architecture.
Data Source
AI summary
An electronic device comprising a memory storing a neural network model; and at least one processor to convert audio data to a frequency domain, obtain encoded data by inputting the audio data converted to the frequency domain into a first layer, and obtain query data, key data, and value data by inputting the encoded data into a second layer. The processor obtains scored query data by inputting the query data into a third layer, an attention weight by calculating the element-wise product of the scored query data and the key data, context data by calculating the element-wise product of the attention weight and the value data, an object separation mask by inputting the context data and the query data into a fourth layer, and an audio object included in the audio data by converting the object separation mask to a time domain.