Neural Network Audio Instrument Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Karaoke systems and audio processing technologies fail to effectively extract or morph specific instruments from audio mixes, especially when vocals are not panned to the center, leading to compromised sound quality and loss of original timbre.
Innovation Solution
A method and system utilizing an artificial neural network trained on a mixture spectrum, combining a target spectrum with non-target spectra, to extract or morph target instruments within an audio mix, allowing for independent processing of vocals and instruments without requiring special panning, and enabling real-time processing with preserved sound quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional voice extraction systems are used, then vocals can be removed from audio mix, but the system fails when vocals are not panned to center and cannot remove instruments
Solution Approach 1:
The system transforms the audio signal from time domain to frequency domain using Short-Time Fourier Transform, changing the representation parameters to enable spectral component separation. This allows extraction of specific instruments based on their spectral characteristics rather than spatial panning position, resolving the contradiction between adaptability to different panning configurations and extraction accuracy.
Solution Approach 2:
The audio spectrum is segmented into multiple frequency bins and time frames, allowing independent processing of different spectral components. The neural network processes each spectral component separately to identify and extract target instruments, enabling accurate extraction regardless of the instrument's panning position in the original mix.
2Adaptability or versatility
If data extraction and resynthesis is used, then audio can be processed and modified, but sound quality deteriorates and original timbre is lost
Solution Approach 1:
Instead of extracting and resynthesizing audio data, the system directly extracts the spectral components of target instruments from the frequency domain representation. The extracted spectral components are then processed and combined with remaining spectral components to produce the final output, preserving the original timbre and sound quality while enabling instrument-specific processing.
Solution Approach 2:
The system creates a spectral copy of the audio signal representing the target instrument, processes this copy independently, and combines it with the remaining spectral components. This copying approach allows precise control over the extracted instrument while maintaining the integrity of the original audio's timbral characteristics.
3Measurement precision
If complex processing is applied to extract instruments, then extraction accuracy improves, but computational complexity increases
Solution Approach 1:
The system performs preliminary spectral analysis using Short-Time Fourier Transform to convert the audio signal into the frequency domain before instrument extraction. This preliminary transformation simplifies subsequent processing by converting complex time-domain signals into separable frequency components, reducing the computational complexity of accurate instrument extraction.
Solution Approach 2:
The system replaces complex signal processing algorithms with a neural network model that has been trained to directly identify and extract instrument spectral components. This substitution of traditional signal processing with machine learning-based approach improves extraction accuracy while maintaining computational efficiency for real-time processing.
Data Source
AI summary
It is disclosed a method comprising obtaining a target spectrum, obtaining a set of non-target spectra, the set of non-target spectra comprising one or more non-target spectra, summing the target spectrum and the set of non-target spectra to obtain a mixture spectrum, and training an artificial neural network by using the mixture spectrum as input of the neural network and by using a spectrum which is based on the target spectrum as desired output of the artificial neural network.