Audio Packet Loss Compensation Using Time-Domain Semantic Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional audio packet loss compensation solutions in live streaming services suffer from information loss and poor accuracy due to frequency domain feature extraction, leading to inadequate voice recovery in complex network environments.
Innovation Solution
Directly perform feature extraction on the original time domain audio signal, incorporating semantic understanding assistance, and use a neural network model with one-dimensional convolutional and gated recurrent units to predict lost audio frames, enhancing accuracy and naturalness of voice restoration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If frequency domain feature extraction is performed on audio signal, then processing complexity is reduced, but information loss increases and prediction accuracy deteriorates
Solution Approach 1:
Instead of converting audio signals to frequency domain for feature extraction, the patent inverts the approach by performing feature extraction directly in the time domain on raw audio waveforms. This avoids the information loss associated with Fourier transformation while still enabling effective feature representation through convolutional neural networks.
Solution Approach 2:
The patent replaces traditional signal processing mechanisms (Fourier transform, spectral analysis) with a deep learning-based time domain processing system. The mechanical/mathematical transformation process is substituted with neural network-based feature extraction that operates directly on time domain signals, preserving original information while achieving comparable or superior performance.
2Speed
If frequency domain feature extraction is performed on audio signal, then processing speed is improved, but prediction accuracy deteriorates
Solution Approach 1:
The patent inverts the conventional processing pipeline by eliminating the frequency domain conversion step and performing end-to-end learning directly on time domain audio signals. This approach maintains processing efficiency while significantly improving prediction accuracy by preserving all original audio information.
3Device complexity
If conventional audio packet loss compensation is used, then system simplicity is maintained, but voice recovery quality is insufficient
Solution Approach 1:
The patent replaces conventional audio packet loss compensation mechanisms with a deep learning-based prediction system. Instead of using traditional signal processing techniques for interpolation and reconstruction, the system uses a neural network trained on time domain audio data to predict and reconstruct lost frames, achieving superior voice recovery quality.
Solution Approach 2:
The patent fundamentally changes the processing parameters by operating in the time domain rather than frequency domain. This parameter change enables the model to capture temporal relationships and audio characteristics more effectively, leading to improved voice recovery quality while maintaining reasonable system complexity.
4Device complexity
If semantic understanding assistance is not incorporated, then model simplicity is maintained, but prediction accuracy for lost audio frames is insufficient
Solution Approach 1:
The patent implements a multi-functional model that simultaneously performs acoustic feature extraction, semantic understanding, and audio frame prediction. By integrating these functions into a unified deep learning system, the model achieves high prediction accuracy while avoiding the complexity of separate processing pipelines.
Solution Approach 2:
The patent merges acoustic processing and semantic understanding components into a single integrated model. This combination allows the system to leverage both acoustic patterns and semantic context for improved audio frame prediction, achieving higher accuracy without requiring multiple separate systems.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the present invention provide an audio packet loss compensation processing method and apparatus and an electronic device, where the method includes: acquiring time domain signals of a first number of past existing audio frames; carrying out acoustic feature extraction on the time domain signals of the first number of the existing audio frames to generate acoustic features of the existing audio frames; carrying out semantic feature extraction on the time domain signals of the first number of the existing audio frames, and carrying out semantic understanding based on the extracted semantic features to generate semantic expressions corresponding to the existing audio frames; and predicting a time domain signal of a current audio frame according to the acoustic features and the semantic expressions. In an embodiment of the present invention, information loss of audio features is avoided by carrying out feature extraction of an original timing signal of audio frames during an audio frame prediction process. Furthermore, a semantic understanding assistance is introduced during the audio frame prediction process, so that a lost audio frame can be predicted more accurately.