Spectral Resynthesizer for Real-Time Polyphonic Audio Synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing audio synthesis systems struggle with real-time extraction of pitch features from polyphonic audio inputs and synchronizing synthesis with the tempo of the input audio, often resulting in latency and undesirable audio artifacts.
Innovation Solution
A method that involves analyzing digital audio streams by performing frequency transforms, blending spectra, and integrating peak frequencies to configure oscillators for real-time audio synthesis, allowing for the generation of synthesized audio that is synchronized with the tempo of the input audio.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If frequency domain transformation is used to extract pitch features, then pitch detection capability is improved, but latency increases
Solution Approach 1:
The audio signal is divided into overlapping segments that are processed independently and then combined. This allows the system to use shorter analysis windows for each segment, reducing latency while maintaining pitch detection accuracy through the overlapping structure that ensures continuous coverage.
Solution Approach 2:
Pitch detection and feature extraction are performed in advance on segmented audio data before synthesis. The system pre-processes audio segments to extract pitch features, stores them, and then uses these pre-extracted features for rapid synthesis, eliminating the need for real-time frequency domain transformation during playback.
2Adaptability or versatility
If inverse fast fourier transform is used to regenerate audio, then frequency domain modifications are achieved, but undesirable audio artifacts are generated
Solution Approach 1:
The system extracts only the essential pitch features from the frequency domain representation without performing a complete inverse transform. By taking out and using only the critical pitch information, the system avoids regenerating the full audio signal through IFFT, thereby eliminating the artifacts that would otherwise be introduced.
Solution Approach 2:
Instead of using IFFT to regenerate the audio signal, the system creates a simplified copy or representation of the pitch features and uses this copy for synthesis. This approach maintains the beneficial frequency domain processing while avoiding the harmful artifacts of full signal regeneration.
3Ease of operation
If monophonic pitch detection is used, then simple pitch tracking is achieved, but polyphonic audio inputs cannot be processed
Solution Approach 1:
The audio signal is segmented into multiple overlapping portions, each processed by a monophonic pitch detector. By segmenting the polyphonic signal and applying simple pitch detection to each segment independently, the system maintains the simplicity of monophonic tracking while handling polyphonic inputs through the combined results of multiple segment analyses.
Solution Approach 2:
The system transitions from analyzing the entire polyphonic signal simultaneously to analyzing segmented portions in a temporal dimension. This dimensional approach allows simple monophonic detectors to effectively process complex polyphonic material by examining smaller time windows where fewer simultaneous pitches are present.
Data Source
AI summary
A method of and system for generating audio having pitch attributes of an incoming audio stream. The method comprises receiving a digital audio input. The audio spectrum is analyzed and integrated over segments of digital audio data upon receiving analysis triggers which can be synced with the audio tempo. The integrated spectrum is processed to find peak frequencies in the spectrum and their associated gain stored in a peaks array. The peak frequencies are used to program the oscillators controllable attributes and characteristics. The synthesis is performed upon receiving an analysis clock. A number of digital oscillators are configured with the associated frequency parameters and gain parameters from a peaks array. The oscillators are configured according to the audio pitch analysis and generate an oscillator output at the frequency and gain specified in the peaks array. These oscillator outputs are summed together generating synthesized audio.