Sound Generation Device Using Trained Model for Real-Time Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autoregressive (AR) type generative models face delays in generating sound signals according to user instructions, making it difficult to produce sound signals in accordance with user intentions due to delays in following control values such as pitch or volume.
Innovation Solution
A sound generation method and device that uses a trained model to process control values and acoustic feature values at each time point, allowing for immediate updates to the acoustic feature value sequence based on user instructions, generating alternative acoustic feature values if a mandatory instruction is received, and updating the sequence accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If an AR type generative model is used to generate sound signals, then the model can process control values sequentially, but there will be delays in following the control values such as pitch or volume
Solution Approach 1:
The system pre-calculates and stores acoustic feature values for multiple future time points before they are actually needed. When a control value is received at time t, the system can immediately retrieve pre-computed acoustic features for t, t+1, t+2, etc., eliminating the sequential processing delay inherent in traditional AR models. This allows the sound generation to keep up with real-time control value changes without waiting for step-by-step computation.
Solution Approach 2:
The system dynamically adjusts the look-ahead window size and update frequency based on the current playback state and control value change rate. When control values change rapidly, the system increases the number of pre-computed future frames; when changes are gradual, it reduces computation. This dynamic adaptation optimizes the balance between responsiveness and computational efficiency, ensuring accurate following of control values without unnecessary processing overhead.
2Stability of the object's composition
If the acoustic feature value sequence is updated immediately with generated values, then the model maintains consistency, but it cannot respond to mandatory instructions from users
Solution Approach 1:
The system implements a dynamic update mechanism where the acoustic feature value sequence can switch between two modes: normal incremental updates that maintain consistency, and mandatory jumps that immediately apply user instructions. A flag or state variable tracks whether the system is in consistency-maintenance mode or user-instruction mode, allowing seamless transition between these conflicting requirements based on real-time user input.
Solution Approach 2:
The system introduces an intermediary control mechanism that sits between the generated acoustic feature values and the final sound output. This intermediary layer can detect mandatory user instructions and override the normally generated values with user-specified values at specific time points, while still maintaining the overall consistency of the acoustic feature value sequence through proper interpolation and transition handling.
Data Source
AI summary
A sound generation method includes receiving a control value at each of a plurality of time points on a time axis, accepting a mandatory instruction, generating an acoustic feature value of a specific time point, by using a trained model to process the control value and an acoustic feature value sequence, and updating the acoustic feature value sequence. The acoustic feature value sequence is updated by using the generated acoustic feature value, as the mandatory instruction has not been received for the specific time point. As the mandatory instruction has been received for the specific time point, one or more alternative acoustic feature values of one or more time points, which includes at least the specific time point, in accordance with the control value for the specific time point is generated, and the acoustic feature value sequence is updated by using the one or more alternative acoustic feature values.