Mobile Hearing Audio Pipeline With Neural Speech Enhancement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hearing devices face challenges in efficiently enhancing speech quality and reducing noise interference due to practical limitations on power, processing capability, and memory storage, while also requiring flexible and adaptable sound enhancement techniques for various environments.
Innovation Solution
The implementation of simplified deep neural networks (DNNs) in hearing devices that dynamically switch between different models based on sound classification, utilizing feature extraction and pruning techniques to optimize resource usage and adapt to changing auditory environments, with support from mobile devices for enhanced processing capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks are used for speech enhancement in hearing devices, then speech quality enhancement is improved, but device complexity and resource requirements increase
Solution Approach 1:
The system segments the DNN processing into two parts: complex DNN inference performed on the mobile device cloud, and simpler audio signal processing performed on the hearing device. This segmentation allows high-precision speech enhancement without requiring the hearing device to have complex processing capabilities.
Solution Approach 2:
The mobile device serves as an intermediary between the audio source and the hearing device. It receives audio signals, performs complex DNN-based speech enhancement, and transmits the enhanced audio to the hearing device, which only needs to handle simpler audio processing tasks.
2Adaptability or versatility
If multiple DNN models are implemented for different acoustic environments, then adaptability is improved, but memory storage and processing resources increase
Solution Approach 1:
The system dynamically switches between different DNN models based on the acoustic environment detected by the hearing device. Instead of storing multiple complete DNN models on the hearing device, it maintains a library of model identifiers and loads only the required model from the mobile device when needed, making the system adaptable without requiring large storage capacity.
3Power
If cloud-based processing is used for DNN inference, then processing capability is improved, but loss of time in signal transmission increases
Solution Approach 1:
The hearing device performs preliminary audio signal processing locally, including noise floor estimation, audio signal acquisition, and preliminary feature extraction, before transmitting data to the mobile device for DNN inference. This preliminary action reduces the amount of data that needs to be transmitted and processed in the cloud, minimizing transmission time.
Solution Approach 2:
The system maintains continuous audio streaming from the hearing device to the mobile device, with overlapping processing stages where the hearing device continuously captures and pre-processes audio while the mobile device performs DNN inference on previous audio frames. This continuous processing reduces overall latency compared to batch processing approaches.
Data Source
AI summary
A system includes a mobile device that receives an audio signal from a microphone of the mobile device. The mobile device processes the audio signal via a neural network to obtain a speech-enhanced audio signal. The system includes an ear-wearable device comprising a data interface operable to communicate with the external data interface of the mobile device. The ear-wearable device includes an audio processing path coupled to the data interface and is operable to receive the speech-enhanced audio signal and reproduce the speech-enhanced audio in an ear of a user.