Deep Learning Audio Packet Loss Concealment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing packet loss concealment techniques in audio processing, such as those used in VoIP and audio streaming, are inadequate in providing a smooth listening experience due to packet loss and delay over asynchronous networks like IP networks, as they often rely on receiver-based methods that are not effective in real-time scenarios.
Innovation Solution
Implementing machine learning models, specifically deep learning, to infer and recover lost audio frames by splitting previous frames into high-band and low-band signals, inferring the low-band signal, and combining it with the high-band signal to reconstruct the frame, while also adding noise and smoothing the frame in the time or frequency domain to enhance voice quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional receiver-based packet loss concealment techniques are used, then implementation simplicity is maintained, but voice quality and listening experience deteriorate due to inadequate compensation for packet loss and delay
Solution Approach 1:
The patent segments the audio signal processing into distinct frequency bands (low-band and high-band). The machine learning model specifically infers only the low-band signal components, while the high-band components are handled separately through traditional interpolation or copying methods. This segmentation allows the complex ML approach to focus on the most critical frequency range, improving voice quality without requiring the entire signal processing chain to be overly complex.
Solution Approach 2:
The patent performs preliminary splitting of the audio signal into frequency bands before the inference process. By pre-processing the signal to separate low-band and high-band components, the system prepares the data in an optimal format for the machine learning model to work with, reducing the computational burden during real-time packet loss concealment and improving overall processing efficiency.
2Reliability
If complex machine learning models are applied to infer lost frames, then voice quality improves, but computation complexity and time delay increase
Solution Approach 1:
The patent divides the audio spectrum into low-band and high-band segments, applying the computationally intensive machine learning inference only to the low-band portion. This segmentation strategy reduces the dimensionality of the problem the ML model must solve, thereby lowering computation complexity while still achieving significant packet loss mitigation through accurate low-band reconstruction.
Solution Approach 2:
The patent applies machine learning inference selectively only to the low-band signal components rather than processing the entire audio spectrum. This partial action approach focuses computational resources on the frequency range that contributes most to voice quality, achieving effective packet loss mitigation without the excessive computation that would result from processing all frequency bands with ML.
3Reliability
If machine learning inference is performed for lost frame recovery, then listening experience improves, but time delay increases due to processing requirements
Solution Approach 1:
The patent segments the audio processing task to apply ML inference only to low-band components, which reduces the amount of computation required compared to full-band processing. This segmentation enables faster inference times, thereby reducing the time delay introduced by packet loss concealment while still achieving effective audio stream recovery.
Solution Approach 2:
The patent performs partial inference by focusing machine learning processing only on the low-band signal portion rather than the complete audio signal. This partial processing approach significantly reduces the computational load and inference time, minimizing time delay while maintaining effective audio recovery quality.
4Productivity
If simple frame copying or interpolation is used, then computation time is reduced, but voice quality and smoothness deteriorate
Solution Approach 1:
The patent segments the audio signal into low-band and high-band components, applying machine learning inference specifically to the low-band portion to improve voice quality. Meanwhile, the high-band components are reconstructed using simpler and faster methods such as copying or interpolation, thereby maintaining acceptable processing speed while achieving superior overall audio quality compared to uniform simple processing.
Data Source
AI summary
A method for recovering a current frame of an audio stream includes detecting that a current packet is lost, the current packet including an audio signal; splitting one or more frames into respective high-band signals and respective low-band signals, the one or more frames precede the current frame in the audio stream; inferring a current low-band signal of the current frame using, as inputs to a machine-learning model, the respective low-band signals; combining the inferred current low-band signal with the respective high-band signals to obtain the current frame; and adding the current frame to a playout buffer.