On-Device Stem Separation Modules for Lower Compute and Power
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional stem separation algorithms require significant computational resources and are typically implemented in high-resource environments, making them unsuitable for resource-constrained devices.
Innovation Solution
Development of compact, lightweight stem separation modules that can be implemented on user electronic devices, utilizing condensed neural processing units and reduced feature sizes to perform audio stem separation efficiently in resource-constrained environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional stem separation algorithms are used, then stem separation performance is achieved, but computational resources and power consumption become excessive
Solution Approach 1:
The patent applies parameter changes by reducing the computational complexity of stem separation algorithms through optimized neural network architectures, quantization, and pruning techniques. This allows the system to maintain effective stem separation performance while significantly reducing power consumption and computational resource requirements, enabling deployment on resource-constrained devices.
Solution Approach 2:
The patent creates a simplified copy or variant of conventional stem separation algorithms that is optimized for resource-constrained environments. By using condensed neural processing units and reduced feature sizes, the system produces a lighter version of the original algorithm that maintains core functionality while reducing computational burden.
2Reliability
If conventional stem separation algorithms are used, then stem separation performance is achieved, but device complexity and resource requirements become excessive
Solution Approach 1:
The patent changes key parameters of the computational system including reducing the number of neural network layers, decreasing feature vector dimensions, and optimizing data types. These parameter changes result in a simplified algorithm implementation that maintains stem separation effectiveness while reducing device complexity and computational resource requirements.
Solution Approach 2:
The patent extracts only the essential components needed for effective stem separation, removing unnecessary computational elements from conventional algorithms. By taking out redundant processing steps and retaining only critical neural processing units, the system achieves reduced device complexity while preserving core separation functionality.
3Ease of operation
If stem separation is performed in real-time on user devices, then accessibility and usability improve, but computational resources become constrained
Solution Approach 1:
The patent optimizes algorithm parameters specifically for real-time execution on consumer devices, adjusting computational precision, processing batch sizes, and neural network configurations to balance real-time performance requirements with limited computational resources. This enables stem separation to run natively on user devices without requiring excessive hardware power.
Data Source
AI summary
A method comprises: (i) presenting a user interface frontend that displays a control for activating a system-level stem separation mode; and (ii) after receiving user input activating the system-level stem separation mode, and when a system generates a system output audio signal based on audio associated with one or more applications executing on the system: (a) refraining from sending the system output audio signal to one or more playback devices over a system audio output signal path; (b) identifying a set of one or more segments of the system output audio signal; (c) processing the set of one or more segments using a selected stem separation module to generate a set of stem-separated segments comprising a plurality of audio stems; and (d) sending at least one audio stem from the plurality of audio stems to the one or more playback devices via the system audio output signal path.