Neural Spatial Audio Coding via Two-Branch Codec
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional spatial audio codecs struggle to achieve high-quality, perceptually accurate reconstruction of multi-channel audio at extremely low bitrates, especially when using microphone array spatial recording systems.
Innovation Solution
A neural spatial audio coding method that utilizes a two-branch codec framework, where the first branch encodes a reference channel using a neural sub-band codec, and the second branch encodes spatial information to reconstruct multi-channel audio, leveraging short-time Fourier transforms and neural networks for encoding and decoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional spatial audio codecs are used to compress multi-channel audio, then coding bitrate is reduced, but spatial reconstruction quality deteriorates significantly
Solution Approach 1:
The patent segments the multi-channel audio coding task into two independent branches: a first branch that codes a reference channel using a neural sub-band codec, and a second branch that codes spatial information using a spatial codec. This segmentation allows each branch to be optimized independently, with the reference channel capturing temporal-spectral content and the spatial branch capturing directional information, achieving high reconstruction quality at low bitrate through efficient division of coding responsibilities.
Solution Approach 2:
The patent introduces an intermediary representation called the spatial covariance matrix, which captures inter-channel correlations and spatial information in a compressed form. This intermediary serves as a bridge between the reference channel coding and the spatial reconstruction, enabling the system to recover multi-channel audio from a low-bitrate reference channel by leveraging the spatial covariance matrix as a mediator that encodes directional relationships between channels.
2Device complexity
If channel-independent coding is applied to microphone array speech, then coding complexity is reduced, but inter-channel and inter-band correlations are not fully exploited, requiring larger bands to be coded separately
Solution Approach 1:
The patent merges the coding of multiple channels into a unified spatial codec that processes the spatial covariance matrix as a single input. Instead of coding each channel independently, the spatial codec combines spatial information from all channels into a compact representation, fully exploiting inter-channel correlations. This merging approach reduces the total number of separate coding operations while capturing the full spatial structure of the audio signal.
Solution Approach 2:
The patent transforms the coding approach by changing from channel-domain parameters to spatial covariance domain parameters. The spatial covariance matrix serves as a new parameter representation that captures the essential spatial relationships between channels. By coding in this transformed parameter space rather than in the original channel domain, the system achieves more efficient compression while maintaining spatial information integrity.
Data Source
AI summary
A method performed for performing neural spatial audio coding, comprises: receiving an audio signal comprising a plurality of channels; selecting a channel from the plurality of channels as a reference channel; performing a STFT on the reference channel to generate a frequency domain reference channel; inputting the frequency domain reference channel into a first codec; performing the STFT on the plurality of channels minus the channel selected as the reference channel to generate a spatial covariance matrix; inputting the spatial covariance matrix and the frequency domain reference channel into a second codec; reconstructing the audio signal based on an output of the first codec and an output of the second codec to generate a reconstructed audio signal; and training the first and second codecs.