Neural Accompaniment Extraction With Spectrogram CNNs for Mobile Karaoke
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for creating karaoke versions of songs often alter the timing of the accompaniment portion and require significant computational resources that exceed the capabilities of mobile devices, and there is a lack of sufficient ground truth data for training neural networks to accurately extract the accompaniment.
Innovation Solution
A neural network is trained using a combination of ground truth pairs and a loss network to determine the difference between the output and ground truth, supplemented by feature loss to improve convergence, and stochastic gradient descent is used to minimize the weighted combination of these values for effective accompaniment extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional methods are used to create karaoke versions, then the accompaniment can be extracted, but the timing of the accompaniment is altered and the audio mixture includes portions that should be removed
Solution Approach 1:
The patent replaces conventional signal processing methods with a neural network-based system. The neural network is trained to accurately separate vocals from accompaniment while preserving the original timing and structure of the audio, thereby improving both extraction accuracy and timing reliability simultaneously.
2Manufacturing precision
If conventional methods are used for accompaniment extraction, then the process can be completed, but the computational resources required exceed the capabilities of mobile devices
Solution Approach 1:
The patent transforms the audio input into a spectrogram representation, changing the parameter space from time-domain audio signals to frequency-time visual representations. This transformation enables the neural network to process the data more efficiently, reducing computational resource requirements while maintaining extraction quality, making it suitable for mobile device deployment.
3Manufacturing precision
If a neural network is trained with ground truth pairs, then accurate accompaniment extraction can be achieved, but there is a lack of sufficient ground truth data available for training
Solution Approach 1:
The patent introduces an intermediary approach by using spectrograms as a intermediate representation and employing a loss network that works with feature loss. This intermediary mechanism allows the system to achieve accurate accompaniment extraction even with limited ground truth training data, as the spectrogram transformation and feature-based loss function provide additional constraints that guide the learning process.
Data Source
AI summary
A messaging system that extracts accompaniment portions from songs. Methods of accompaniment extraction from songs includes receiving an input song that includes a vocal portion and an accompaniment portion, transforming the input song to an input image, where the input image represents the frequencies and intensities of the input song, processing the input image using a convolutional neural network (CNN) to generate an output image, and transforming the output image to an output accompaniment, where the output accompaniment includes the accompaniment of the input song.