Multi-Rate Learnable Audio Frontend for Hardware-Limited DNNs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current audio deep neural networks (DNN) architectures face challenges in deploying learnable frontends on vector processors due to kernel size constraints, leading to significant discrepancies in output between CPU and vector processor executions, especially with large 1D convolutional kernels.
Innovation Solution
A multi-rate learnable audio frontend model is designed, utilizing a multi-stage approach with learnable filterbanks and decimation filters, constrained by hardware limitations, to convert 1D audio waveforms into 2D spectrograms, enabling effective deployment on existing hardware accelerators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large 1D convolutional kernels are used to represent low-frequency components effectively, then the representation power is improved, but the device complexity and hardware compatibility deteriorate due to kernel size constraints on existing hardware accelerators
Solution Approach 1:
The patent divides the single large 1D convolutional kernel into multiple smaller 1D kernels applied in sequence across different stages. Each stage processes the audio waveform with a manageable kernel size that fits hardware constraints, while the cascade of stages collectively achieves the representation power of a larger kernel by processing different frequency bands and time scales.
Solution Approach 2:
The patent transforms the problem from using a single large 1D kernel to using multiple small 1D kernels across multiple stages, effectively adding a temporal dimension to the processing. This multi-stage approach allows each kernel to operate within hardware size limits while the cumulative effect across stages achieves the desired low-frequency representation capability.
2Speed
If audio DNNs are executed on vector processors, then the processing speed is improved, but the output accuracy deteriorates due to significant discrepancies between CPU and vector processor executions
Solution Approach 1:
The patent modifies the audio frontend architecture to use multiple small kernels with specific stride and padding parameters that are optimized for vector processor execution. By changing the kernel dimensions and processing parameters to match hardware capabilities, the model achieves consistent results across different processors while maintaining high processing speed on vector processors.
3Adaptability or versatility
If multi-stage learnable audio frontend is used, then the adaptability is improved, but the device complexity increases
Solution Approach 1:
The patent implements a multi-stage architecture where each stage can be independently configured with different kernel sizes, strides, and padding values. This dynamic configuration allows the system to adapt to different hardware platforms and audio processing requirements by adjusting parameters at each stage, while the modular structure prevents the overall complexity from becoming unmanageable.
Data Source
AI summary
Methods and systems of processing audio data with a multi-stage audio front end model is provided. A one-dimensional audio waveform is received as input and processed using a multi-stage audio frontend model to convert the one-dimensional waveform into a two-dimensional matrix representing features of the audio waveform. The multi-stage learnable audio frontend model is configured to apply a first filterbank to the audio waveform to generate a first time-frequency representation of the audio waveform; apply a first decimation filter to the audio waveform to generate a first decimated audio input; apply a second filterbank to the first decimated audio input to generate a second time-frequency representation of the audio waveform; and stack the first time-frequency representation and the second time-frequency representation together to generate the two-dimensional matrix.