On-Device Audio Separation and Classification for Low-Compute Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine-learned sound separation models are resource-intensive and impractical for implementation on devices with lower compute capacity, such as smartphones and smartwatches, leading to long processing times and unsatisfactory user experiences.
Innovation Solution
A computing pipeline that includes a machine-learned audio separation model and classification model, optimized for efficient execution on mobile devices, utilizing audio-visual analysis and a graph-based execution framework to separate and classify sounds from different sources in media inputs, with features like keyframe decoding and parallel processing to minimize latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large machine-learned sound separation models are used, then sound separation quality is improved, but processing time increases and device feasibility deteriorates
Solution Approach 1:
The patent segments the audio processing pipeline into distinct stages: audio decoding, sound separation, audio classification, and video decoding. Each stage can be optimized independently and executed in parallel where possible, reducing overall processing time while maintaining quality. The graph-based execution framework enables efficient scheduling of these segmented tasks.
Solution Approach 2:
The system performs preliminary actions by decoding audio and video frames in advance and caching them for subsequent processing. This allows the machine-learned models to process pre-prepared data, reducing computation time during actual sound separation and classification operations.
2Reliability
If machine-learned sound separation models are implemented on mobile devices, then user privacy is improved, but compute capacity requirements worsen
Solution Approach 1:
The patent optimizes model parameters and execution parameters to enable efficient on-device processing. By adjusting computational parameters and using optimized graph execution, the system reduces the compute capacity burden on mobile devices while maintaining privacy through local processing.
Solution Approach 2:
The system introduces an intermediary graph-based execution framework that mediates between the machine-learned models and mobile device hardware. This framework optimizes the execution of computationally intensive tasks, making them feasible on devices with limited compute capacity while maintaining privacy through on-device processing.
3Productivity
If audio-visual analysis is performed using keyframe decoding, then processing speed is improved, but video quality may deteriorate
Solution Approach 1:
The system applies partial action by decoding only keyframes rather than every video frame. This reduces processing time and computational load while providing sufficient visual information for accurate audio classification, accepting that intermediate frame quality is not fully preserved but maintaining overall functionality.
Data Source
AI summary
Aspects of the disclosed technology include computer-implemented systems and methods for automatically separating sounds associated with different sources in media such as video. More particularly, a machine-learned system is configured to separate sounds in media and provide an interface for users to easily manipulate the separated sounds during playback of the media. The system can obtain media, provide decoded audio from the media to a machine-learned audio separation model, generate a plurality of separated sound components from the decoded audio using the machine-learned audio separation model, provide decoded video from the media and the plurality of separated sound components to a machine-learned audio classification model, and generate a class label for each of the plurality of separated sound components using the machine-learned audio classification model.