Stereo Audio Upmixing with PCA and DNN Envelope Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for upmixing downmix stereo audio signals to multi-channel audio signals, such as those using Independent Component Analysis (ICA), Primary Component Analysis (PCA), and Nonnegative Matrix Factorization (NMF), often result in spatial distortion by incorrectly positioning audio objects due to difficulties in matching separated components to the original multi-channel signal.
Innovation Solution
A signal processing device and image display apparatus that utilize a converter, primary component analyzer, feature extractor, model learning predictor, envelope adjustor, and inverse converter, along with a filter bank and masking unit, to perform primary component analysis and deep neural network-based prediction, enabling accurate channel separation and envelope adjustment to improve spatial distortion during upmixing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If Blind Upmix or Source Separation methods (ICA, PCA, NMF) are used to upmix downmix stereo audio signals to multi-channel audio signals, then the audio signal can be separated into individual channels or sources, but spatial distortion occurs because the separated components cannot be accurately matched to the original multi-channel signal
Solution Approach 1:
The patent transforms the audio signal from time domain to frequency domain using Short-Time Fourier Transform (STFT), adding a frequency dimension to the analysis. This allows the upmixing process to operate in the time-frequency domain, where the signal can be represented as a three-dimensional tensor (time, frequency, channel). By working in this extended dimensionality, the system can better preserve spatial characteristics and accurately match separated components to original multi-channel positions, resolving the spatial distortion problem while maintaining upmix capability.
2Adaptability or versatility
If Primary Component Analysis (PCA) is used to separate audio signals into primary and ambient components, then the signal can be decomposed, but the separated components appear in incorrect spatial positions (front, rear, upstream channels) different from the original content creator's intention
Solution Approach 1:
The patent employs a feedback mechanism where the upmixed multi-channel signal is compared with the original downmix signal, and the difference information is used to iteratively refine the separation and positioning of audio components. The system continuously adjusts the spatial positioning of separated components by analyzing the residual error and correcting the mapping between primary/ambient components and target channel positions, thereby resolving the spatial positioning inaccuracy while preserving the signal decomposition capability.
3Adaptability or versatility
If Independent Component Analysis (ICA) or Nonnegative Matrix Factorization (NMF) is used to decompose audio signals into independent or base components, then the signal can be separated, but it is difficult to match the decomposed components to specific multi-channel positions (front, center, woofer, rear, upstream)
Solution Approach 1:
The patent changes the parameter representation of the audio signal by transforming it into the frequency domain using STFT, and represents the signal as a three-dimensional tensor with explicit time, frequency, and channel dimensions. This parameter transformation makes the channel dimension observable and measurable, allowing the system to accurately match decomposed components to specific multi-channel positions. The frequency-domain representation provides additional structural information that facilitates channel identification and component mapping, resolving the channel matching difficulty while maintaining separation capability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed are a signal processing device and an image display apparatus including the same. The signal processing device and an image display apparatus including the same include: a converter configured to convert of frequency of an input stereo audio signal; a primary component analyzer configured to perform primary component analysis based on a signal from the converter; a feature extractor configured to extract a feature of a primary component signal based on a signal from the primary component analyzer; an envelope adjustor configured to perform envelope adjustment based on prediction performed on the basis of a deep neural network model; and an inverse converter configured to inversely convert a signal from the envelope adjustor to output an upmix audio signal of multi-channel. Accordingly, when upmixing the downmix stereo audio signal to a multichannel audio signal, spatial distortion can be improved.