Masked Spectrogram Reconstruction for General Sound Signal Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning techniques, such as those described in Non Patent Literature 1, are limited in their ability to solve general tasks like noise removal, super-resolution, and anomaly detection for sound signals.
Innovation Solution
A learning device and method that generates spectrograms from sound signals, divides them into patches, applies masking and reconstruction using a transformer model, and updates parameters to improve patch reconstruction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If Masked Language Modeling (MLM) and Masked Image Modeling (MIM) frameworks are applied to sound signals, then mask restoration and classification can be achieved, but the model cannot solve general tasks such as noise removal, super-resolution, style transfer, and anomaly detection
Solution Approach 1:
The patent applies universality by designing a masked spectrogram reconstruction model that can handle multiple general tasks (noise removal, super-resolution, style transfer, anomaly detection) through a single unified framework. The encoder-decoder transformer architecture with patch-based masking enables the model to learn universal representations that can be applied to various audio processing tasks without requiring task-specific models.
Solution Approach 2:
The patent utilizes parameter changes by adjusting the masking strategy and reconstruction objectives to adapt the same model architecture to different tasks. By varying the mask generation parameters, patch sizes, and loss functions during training, the model can be optimized for specific tasks while maintaining the same overall structure, enabling one model to serve multiple purposes.
2Manufacturing precision
If patches are selected as masked patches for reconstruction, then the encoder and decoder can learn to reconstruct audio information, but the model requires complex processing including spectrogram generation, patch division, and parameter updates
Solution Approach 1:
The patent applies segmentation by dividing the spectrogram into smaller patches, which allows the model to process and reconstruct audio information in manageable units. This patch-based approach simplifies the reconstruction task compared to working with the entire spectrogram at once, improving reconstruction accuracy while maintaining computational feasibility.
Solution Approach 2:
The patent uses mask tokens as intermediaries during the reconstruction process. These mask tokens placeholder representations of masked patches enable the encoder-decoder to focus on reconstructing only the masked regions using information from unmasked patches, simplifying the processing by directing attention to specific areas rather than processing the entire input.
Data Source
AI summary
A learning device includes: spectrogram generation circuitry 1 that generates a spectrogram from a first sound signal and generates a target spectrogram from a second sound signal; patch generation circuitry 2 that divides the spectrogram to generate a plurality of patches and divides the target spectrogram to generate a plurality of target patches; mask processing circuitry 3 that selects some patches as masked patches; reconstruction circuitry 4 that obtains a plurality of reconstructed patches by reconstructing the plurality of patches by processing of an encoder and a decoder by using visible patches other than some patches among the plurality of patches and mask tokens; and parameter update circuitry 5 that updates parameters such that target patches corresponding to the masked patches among the plurality of target patches approach reconstructed patches corresponding to the masked patches among the plurality of reconstructed patches.