Apparatus and method for locating sound sources
The apparatus and method on edge devices separate and classify audio streams to determine sound source locations using interaural measurements and HRTF responses, addressing the limitations of existing techniques and enabling accurate sound source detection in all directions.
Patent Information
- Application Number
- PCT/KR2025/021285
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-12
- Filing Date
- 2025-12-10
- Publication Date
- 2026-11-19
AI Technical Summary
Existing sound source localization techniques require multiple microphones, precise synchronization, and are computationally expensive, making them unsuitable for ultra-edge devices like earbuds, and cannot detect sound sources in all directions simultaneously.
An apparatus and method using a listening device with a processor and memory that separates audio streams into sub-streams, classifies them by frequency, computes interaural measurements, generates HRTF responses, and determines sound source locations using a sound localization model, calibrated by user orientation.
Enables accurate sound source localization on edge devices without multiple microphones, detecting sources in all directions, and providing precise location estimates.
Smart Images

Figure KR2025021285_19112026_PF_FP_ABST
Abstract
Description
APPARATUS AND METHOD FOR LOCATING SOUND SOURCES
[0001] The present disclosure relates to sound sources localization, and more particularly relates to an apparatus and method for locating sound sources on a listening device being used by a user.
[0002] Sound sources localization is the determination of location of sound sources present around. Human can locate the sound sources with the help of two ears. A human brain determines the location of sound by measuring the difference of sound that reaches to each ear. Identification and localization of the sound sources is important for different audio application such as but not limited to audio sound separation, three-dimensional (3D) audio recording, siren identification, direction identification in hearing aid, directionality in case of the spatial (3D) audio hearing.
[0003] There are multiple existing methods for sound source localization. One such technique is a time delay estimation for sound source localization which estimates the source direction using the time delay of sound reaching multiple earphones. Another technique for sound source localization is beamforming where the direction of sound source is estimated using the microphone array by finding the direction that the beam is strongest by summing the signals of different microphones.
[0004] Another technique for sound source localization is a high-resolution spectral method which uses advance signal processing to separates signal and noise subspaces to estimate sound direction. However, all these existing techniques require multi microphone step up, precise microphone synchronization and are highly computationally expensive making it difficult to apply on ultra-edge listening devices such as earbuds. This leads to on cloud solution or implementing the solution over a phone. Further existing solution is unable to detect sound sources in all directions at once as localization of sound using two microphones using earbuds is very difficult. Also, there is no such prior art to detect all the sound sources in complete auditory sphere at the same time.
[0005] Therefore, there lies at least a need to provide a solution to one or more aforesaid problems or improve an existing solution, product, or process.
[0006] The information disclosed in this background of the disclosure section is only for enhancement of understanding of the general background of the disclosure and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.
[0007] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.
[0008] In an aspect, an apparatus for locating sound sources is provided. The apparatus may comprise memory storing instructions, and at least one processor operatively coupled to the memory and comprising processing circuitry. The at least one processor individually or collectively executes the instructions to cause the apparatus to receive an input audio stream. The at least one processor individually or collectively executes the instructions to cause the apparatus to separate the input audio stream into a plurality of sub audio streams using a trained model. The at least one processor individually or collectively executes the instructions to cause the apparatus to identify each sub audio stream as either a high frequency stream or a low frequency stream by frequency classification based on a frequency of the sub audio stream and a defined frequency threshold. The at least one processor individually or collectively executes the instructions to cause the apparatus to compute an interaural measurement based on the frequency classification of the sub audio stream. The at least one processor individually or collectively executes the instructions to cause the apparatus to generate Head-Related Transfer Function (HRTF) responses of the input audio stream received by the apparatus based on a personalized HRTF and the input audio stream. The at least one processor individually or collectively executes the instructions to cause the apparatus to determine, using a sound localization model, an estimated location of a sound source for the sub audio stream based on the computed interaural measurement and the generated HRTF responses, wherein the sound localization model is selected based on the frequency classification of the sub audio stream. The at least one processor individually or collectively executes the instructions to cause the apparatus to calibrate the estimated location of the sound source based on a directional orientation of a user to determine a location of the sound source.
[0009] In an aspect, a method for locating sound sources is provided. The method may comprise receiving an input audio stream by an apparatus. The method may comprise separating the input audio stream into a plurality of sub audio streams using a trained model. The method may comprise identifying each sub audio stream as either a high frequency stream or a low frequency stream by frequency classification based on a frequency of the sub audio stream and a defined frequency threshold. The method may comprise computing an interaural measurement based on the frequency classification of the sub audio stream. The method may comprise generating Head-Related Transfer Function (HRTF) responses of the input audio stream received by the apparatus based on a personalized HRTF and the input audio stream. The method may comprise determining, using a sound localization model, an estimated location of a sound source for the sub audio stream based on the computed interaural measurement and the generated HRTF responses. The method may comprise calibrating the estimated location of the sound source based on a directional orientation of a user to determine a location of the sound source.
[0010] In an aspect, a computer-readable storage medium storing instructions is provided. The instructions, when executed by at least one processor, may cause the at least one processor to perform the method corresponding.
[0011] To further clarify the advantages and features of the disclosure, a more particular description of the disclosure will be rendered by reference to specific embodiments thereof, which is illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the disclosure and are therefore not to be considered limiting of its scope. The disclosure will be described and explained with additional specificity and detail with the accompanying drawings.
[0012] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of at least one of apparatus and methods in accordance with embodiments of the present subject matter are now described, by way of example only, and with reference to the accompanying figures, in which:
[0013] Figure 1illustrates a block diagram of an apparatus for locating sound sources, in accordance with an embodiment of the disclosure;
[0014] Figure 2illustrates a block diagram of an implementation of various modules of the apparatus, for locating sound sources, in accordance with an embodiment of the disclosure;
[0015] Figure 3illustrates a block diagram of a sound source identification module, in accordance with an embodiment of the disclosure;
[0016] Figure 4illustrates a block diagram of a scoring module within the sound source identification module,in accordance with an embodiment of the disclosure;
[0017] Figure 5illustrates a block diagram of frequency classification of a selected set of sub audio streams among a plurality of sub audio streams,in accordance with an embodiment of the disclosure;
[0018] Figure 6 illustrates a flowchart to determine an estimated location of a sound source corresponding to the sub audio stream,in accordance with an embodiment of the disclosure;
[0019] Figure 7A, 7B, and 7C illustrate an example to generate a personalized HRTF of a user,in accordance with an embodiment of the disclosure;
[0020] Figure 8illustrates a block diagram of a calibration module,in accordance with an embodiment of the disclosure; and
[0021] Figure 9 illustrates a sequence flow of a method of locating sound sources, in accordance with an embodiment of the disclosure.
[0022] The figures depict embodiments of the disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the disclosure described herein.
[0023] It should be appreciated by those skilled in art that any block diagrams herein represent conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and executed by a computer or processor, whether or not such computer or processor is explicitly shown.
[0024] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.
[0025] While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described and will be described in detail below. It should be understood that, however it is not intended to limit the disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and the scope of the disclosure.
[0026] The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed in a device or system or apparatus proceeded by "comprises쪋a" does not, without more constraints, preclude the existence of other elements or additional elements in the device or system or apparatus.A function related to AI according to the disclosure is implemented through a processor and a memory. The processor may be configured as one or more processors. In this case, the one or more processors may be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), or a digital signal processor (DSP), a dedicated graphics processor such as a graphics processing unit (GPU) or a vision processing unit (VPU), or a dedicated AI processor such as a neural processing unit (NPU). The one or more processors may control input data to be processed according to predefined operation rules or an AI model stored in a memory. Alternatively, when the one or more processors are AI-dedicated processors, they may be designed with a hardware architecture specialized for processing specific AI models.
[0027] The predefined operation rules or AI model may be generated via training. When the predefined operation rules or AI model are generated via training, it means that the predefined operation rules or AI model set to perform desired characteristics (or purposes) is generated by training a basic AI model with a learning algorithm that utilizes a large number of training data. The training may be performed by a device itself (e.g., the cleaner body 1000) where AI according to the disclosure is being performed, or performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.
[0028] An AI model may include a plurality of neural network layers. Each of the neural network layers includes a plurality of weight values, and performs a neural network arithmetic operation via an arithmetic operation between an arithmetic operation result of a previous layer and the plurality of weight values. A plurality of weight values in each of the neural network layers may be optimized by a result of training the AI model. For example, the plurality of weight values may be refined to reduce or optimize a loss value or a cost value obtained by the Al model during the training. An artificial neural network may include, for example, but is not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network.
[0029] Definitions for other certain words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many if not most instances, such definitions apply to prior as well as future uses of such defined words and phrases.
[0030] In the following detailed description of the embodiments of the disclosure, reference is made to the accompanying drawings that form a part hereof, and which are shown by way of illustration specific embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the disclosure. The following description is, therefore, not to be taken in a limiting sense.
[0031] Figure 1 illustrates a block diagram of an apparatus 100 for locating sound sources, in accordance with an embodiment of the disclosure. The apparatus 100 may comprise a processing unit 102 including at least one processor, an input / output (I / O) interface 104, and memory 106, but not limited thereto.
[0032] The processing unit 102 may comprise specialized processing units such as integrated system (bus) controllers, memory management control units, digital signal processing units, etc. The at least one processor included in the processing unit 102 may execute instructions stored in the memory, individually or collectively. The processing unit 102 may include one or a plurality of processors, is a general-purpose processor such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Artificial Intelligence (AI) dedicated processor such as a neural processing unit (NPU).
[0033] The processor may include various processing circuitry and / or multiple processors. For example, as used herein, including the claims, the term "processor" may include various processing circuitry, including at least one processor, wherein one or more of at least one processor, individually and / or collectively in a distributed manner, may be configured to perform various functions described herein. As used herein, when "a processor", "at least one processor", and "one or more processors" are described as being configured to perform numerous functions, these terms cover situations, for example and without limitation, in which one processor performs some of recited functions and another processor(s) performs other of recited functions, and also situations in which a single processor may perform all recited functions. Additionally, the at least one processor may include a combination of processors performing various of the recited / disclosed functions, e.g., in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.
[0034] The processing unit 102 may be disposed in communication with one or more input / output (I / O) devices via I / O interface 104. The I / O interface 104 may employ communication protocols / methods such as, without limitation, audio, analog, digital, stereo, IEEE-1394, serial bus, Universal Serial Bus (USB), infrared, PS / 2, BNC, coaxial, component, composite, Digital Visual Interface (DVI), high-definition multimedia interface (HDMI), Radio Frequency (RF) antennas, S-Video, Video Graphics Array (VGA), IEEE 802.n / b / g / n / x, Bluetooth, cellular (e.g., Code-Division Multiple Access (CDMA), High-Speed Packet Access (HSPA+), Global System For Mobile Communications (GSM), Long-Term Evolution (LTE) or the like), etc. Using the I / O interface 104, the apparatus 100 may communicate with one or more I / O devices.
[0035] The memory 106 may be a component for storing various programs or data, and may consist of a storage medium, such as read-only memory (ROM), random access memory (RAM), a hard disk, compact disc ROM (CD-ROM), a digital video disc (DVD), etc., or a combination of storage media. The memory 106 may exist separately but may also be configured to be included in the processor. The memory 106 may consist of volatile memory, non-volatile memory, or a combination of volatile memory and non-volatile memory. The memory 106 may store a program or at least one instruction for performing operations according to one or more embodiments of the disclosure as described below. The memory 106 may provide stored data to the processor according to a request from the processor.
[0036] The apparatus 100 may also include suitable logic, circuitry, and interfaces that may be configured to locate sound sources. In an embodiment, the apparatus 100 may be implemented on a listening device such as but not limited to earbuds, headphones etc. Moving on, a detailed explanation of the working of the apparatus 100 is explained in forthcoming paragraphs.
[0037] In an embodiment, the disclosure relates to the apparatus 100 for locating sound sources where the at least one processor of the apparatus 100 may be configured to receive an input audio stream using at least one listening device being used by a user. In an embodiment, the input audio stream may include sounds generated by a plurality of sound sources present in an environment. In one implementation, the apparatus 100 may receive the input audio stream through the I / O interface 104. In an embodiment, the input audio stream may be stored as data 108 within memory 106.
[0038] In an embodiment, the received input audio stream stored in the memory 106 may be processed by modules 110 of the apparatus 100. The modules 110 may be stored within the memory 106 as shown in Figure 1. As an example, the modules 110, communicatively coupled to the processing unit 102, may also be present outside the memory 106.
[0039] In one implementation, the modules 110 may comprise, for example, a sound source identification module 112, a switching module 114, a feature detection module 116, a personalized Head-Related Transfer Function (HRTF) module 118, a sound source localization module 120, and a calibration module 122. It will be appreciated that such aforementioned modules may be represented as a single module or a combination of different modules. The modules may be implemented in any suitable hardware, software, firmware, or combination thereof. Further, the modules 110 may be implemented by various techniques comprising but not limited to computer programs, one or more neural networks, machine learning algorithms, embedded systems design and cloud computing architectures. The implementation of the various modules 110 to locate sound sources shall now be explained in subsequent paragraphs.
[0040] Figure 2 illustrates a block diagram of an implementation of various modules 110 of the apparatus 100, for locating sound sources, in accordance with an embodiment of the disclosure. In an embodiment of the disclosure, the sound source identification module 112 upon receiving the input audio stream comprising sounds generated by the plurality of sound sources present in an environment may be configured to separate the input audio stream into a plurality of sub audio streams using a trained model. In an embodiment, each sub audio stream of the plurality of sound sources corresponds to a respective sound source among the plurality of sound sources. The subsequent paragraphs shall now explain in detail the sound source identification module 112.
[0041] Figure 3 illustrates a block diagram of the sound source identification module 112, in accordance with an embodiment of the disclosure. The sound source identification module 112 may comprise at least one of a sound source classification module 302, a sound source separation module 304 and a scoring module 306. In an embodiment of the disclosure, the sound source classification module 302 may be configured to receive the input audio stream and may convert the input audio stream into a spectrogram which is a frequency domain representation of the input audio stream. Further a convolutional neural network (CNN) architecture may be used for the feature extraction of the audio spectrogram to generate feature maps corresponding to the audio spectrogram as an output from the CNN architecture. Further, the generated feature maps may be fed to a linear classifier for classifying the input audio stream. For example, in a non-limiting scenario, the sound source classification module 302 may classify the input audio stream as siren, baby crying, microwave timer, doorbell, running water etc. Further, the classified audio stream may be fed as input to the trained model such as the sound source separation module 304 for the purpose of separation of the classified audio stream into a plurality of sub audio streams, as discussed below.
[0042] In an embodiment of the disclosure, the sound source separation module 304 may be configured to separate the input audio stream into a plurality of sub audio streams. In an embodiment, each sub audio stream of the plurality of sound sources may correspond to a respective sound source among the plurality of sound sources. The sound source separation module 304 may be a hybrid time-frequency domain-based sound separation model. All the sound sources may be first identified by the hybrid time-frequency domain-based sound separation model and then separated into multiple different sub-streams, as shown below in Table 1.
[0043]
[0044] The hybrid time-frequency domain-based sound separation model may be separately trained on a single sound stream and corresponding multiple sound streams as output, to be further fed to the scoring module 306 as explained below in subsequent paragraphs.
[0045] Figure 4 illustrates a block diagram of the scoring module 306 within the sound source identification module 112, in accordance with an embodiment of the disclosure. In an embodiment of the disclosure, the scoring module 306 upon receiving the plurality of sub audio streams may be configured to determine a score for each of the received plurality of sub audio streams.
[0046] The scoring module 306 may be configured to select a set of sub audio streams among the plurality of sub audio streams for the classification. In an embodiment, the set of sub audio streams may be selected based on a comparison of the determined score for each of the plurality of sub audio streams with a defined (e.g., predefined, predetermined) threshold score. In an embodiment, the scoring module 306 may determine the score for each of the plurality of sub audio streams based on at least one of: similarity within the sub audio streams, historical response of the user corresponding to each sub audio streams, location of the user, and defined (e.g., pre-defined, predetermined) priority of the sub audio streams, as explained in Figure 4.
[0047] In an embodiment, the scoring module 306 may determine the score based on the similarity within the sub audio streams. For example, for a ring bell, a usual ring bell may be given more priority than a ring bell of new house. Table 2 below shows the scoring module 306 to determine the score based on the similarity within the sub audio streams.
[0048]
[0049] In an embodiment, the scoring module 306 may determine the score based on the historical response of the user corresponding to each sub audio streams. For example, the user's response to siren, microwave timer, doorbell etc. Table 3 below shows the scoring module 306 to determine the score based on the historical response of the user corresponding to each sub audio streams.
[0050]
[0051] In an embodiment, the scoring module 306 may determine the score based on the location of the user. For example, the scoring module 306 using the one or more sensors embedded on the listening device may track the location of the user and may be triggered based on the location of the user location accordingly. Table 4 below shows the scoring module 306 to determine the score based on the location of the user.
[0052]
[0053] In an embodiment, the scoring module 306 may determine the score based on the defined (e.g., pre-defined, predetermined) priority of the sub audio streams. For example, the priority may be set by the user. Table 5 below shows the scoring module 306 to determine the score based on the defined priority of the sub audio streams.
[0054]
[0055] Referring back to Figure 3, in an embodiment, the scoring module 306 after determining the score for each of the audio streams may then select the set of sub audio streams among the plurality of sub audio streams based on the comparison of the determined score for each of the plurality of sub audio streams with a defined (e.g., predefined, predetermined) threshold score. For example, the determined score for each of the audio streams greater than the defined (e.g., predefined, predetermined) threshold score for each of the plurality of sub audio streams may be selected for further frequency classification, as explained in below paragraphs and the rest sub audio streams may be filtered out. In a non-limiting embodiment, the defined (e.g., predefined, predetermined) threshold score may vary according to the user. Further, the selected of set of sub audio streams among the plurality of sub audio streams may be frequency classified as explained below in subsequent paragraphs.
[0056] Figure 5 illustrates a block diagram of frequency classification of the selected set of sub audio streams among the plurality of sub audio streams, in accordance with an embodiment of the disclosure. Referring back to Figure 2, the switching module 114 may be configured to classify the frequency of the selected set of sub audio streams and further switch the selected set of sub audio streams based on a defined (e.g., predefined, predetermined) frequency threshold.
[0057] In an embodiment, the switching module 114 may separate the set of sub audio streams based on the frequency of each selected sub audio streams. For example, each sub audio stream may be analysed by the switching module 114 using a Moving Average Based Algorithm to compute a Weighted Moving Average (WMA) frequency for each of the sub audio stream. In an embodiment, the switching module 114 may divide the set of sub audio streams into a plurality of time windows. Further, within each window, the Moving Average Based Algorithm may identify the frequency components for each set of the sub audio streams and may assign weights to each data point for the set of sub audio streams based on the amplitude of the corresponding sub audio stream. For example, the smallest amplitude within the window may be used as a reference, and the weights for other amplitudes may be calculated in comparison with the reference.
[0058] Further, the WMA may be then computed by taking a weighted average of the frequencies in each window for each set of the sub audio streams. This WMA may be calculated using the Equation 1:
[0059] (Equation 1)
[0060] Here, P1, P2, ... are the frequencies at each sample space for each set of the sub audio streams, and W1, W2, ... are the corresponding weights for each set of the sub audio streams based on their corresponding amplitudes. Hence, each sub audio stream may be assigned a unique WMA classified frequency (e.g., f1, f2, f3...), which may reflect the dominant frequency behaviour of that particular sub audio stream, as shown in Figure 5.
[0061] Further, the switching module 114 may classify each sub audio stream as either a high frequency stream or a low frequency stream based on the classified frequency of the sub audio stream as explained in a flowchart of Figure 6.
[0062] Figure 6 illustrates a flowchart to determine an estimated location of a sound source corresponding to the sub audio stream, in accordance with an embodiment of the disclosure. After calculating the WMA frequency for each sub-audio stream (at operation 600), the switching module 114 as explained in the flowchart of Figure 6 may compare the WMA frequency for each sub-audio stream against a defined (e.g., predefined, predetermined) threshold or a reference value (at operation 610). If the frequency exceeds the threshold, the audio stream may be categorized as the high-frequency (such as claps or sirens) and if the frequency falls below the threshold, the sub audio stream may be classified as the low-frequency (such as bass or certain vocal tones). In an embodiment of the disclosure the defined (e.g., predefined, predetermined) threshold frequency may be but not limited to 1500 Hz. Hence if the WMA frequency of the sub-audio stream is greater than 1500 Hz, the audio stream may be categorized as the high-frequency (at operation 620) and if the WMA frequency of the sub-audio stream is lower than 1500 Hz, then the audio stream may be categorized as the low-frequency (at operation 630). The switching module 114 may switch the frequency classified sub audio stream based on their frequency categorisation for further calculation, as explained below.
[0063] In an embodiment, for each classified sub audio stream, the at least one processor may be configured to compute one of an Interaural Level Difference (ILD) or an Interaural Phase Difference (IPD) based on the frequency classification of the sub audio stream. In an embodiment, for each of the low frequency classified sub audio stream, the at least one processor may be configured to compute the IPD (at operation 630). In an embodiment, for each of the high frequency classified sub audio stream, the at least one processor may be configured to compute the ILD (at operation 620).
[0064] In an embodiment, the ILD is the difference in intensity of the sub audio stream arriving at the listening device corresponding to the left ear and the right ear of the user. The ILD may be computed in decibels (dB) using the below Equation 2:
[0065] (Equation 2)
[0066] Here, are the sound pressures at the left and right ears of the listening device used by the user, respectively due to the incoming sub audio streams.
[0067] In an embodiment, the IPD is the difference in phase of the sub audio stream arriving at the listening device corresponding to the left ear and the right ear of the user, and may be computed using the below Equation 3
[0068] (Equation 3)
[0069] Here, d is the distance between the ears (typically ~0.2 meters in humans), is the angle of the incoming sub audio stream relative to the midline of the user, is the wavelength of the sub audio stream and 2 represents a full wave cycle in radians.
[0070] Further, the switching module 114 may send the output of the any one of the IPD or ILD responses to a corresponding sound localization model selected based on the frequency classification of the sub audio stream.In an embodiment,thecorresponding sound localization model may be separatesound localization models used to determine location of the sound sources for each of the computed ILD (at operation 640) or the computed IPD (at operation 650), as shown in the flowchart of Figure 6. In an embodiment,the corresponding sound localization model may determine the estimated location of the sound sources (at operation 660) based on the computed ILD or the computed IPD, and a generated Head-Related Transfer Function (HRTF) responses of the input audio stream received by the listening device corresponding to a left ear and a right ear of the user. In an embodiment, the generated HRTF responses may be based on a personalized HRTF of the user and the input audio stream, as explained in below paragraphs.
[0071] Figure 7A, 7B, and 7C illustrate an example to generate the personalized HRTF of the user, in accordance with an embodiment of the disclosure.
[0072] Referring to Figure 7A, in an embodiment to generate the personalized HRTF of the user, the at least one processor may be configured to generate a three-dimensional (3D) mesh of the user's face using facial features of the user. In an embodiment, the feature detection module 116 may capture the facial features of the user from the front, left, right side so that a complete feature is captured from each side. In an embodiment the process for extracting facial features of the user may begin with the user capturing their images from the front, left or the right side. In an embodiment, the feature detection module 116 may extract the facial features of the user using a recorded video of the user's face, to generate the facial 3D mesh of the user. In an embodiment, the feature detection module 116 may crop out the overlapping part by similarity matching. In an embodiment, the feature detection module 116 may combine the images except the cropped part. In an embodiment, the mesh representation of the user's face may be a graphical representation as shown below, which helps to get larger details of the face with its representation as set of vertices (V) and Faces (F). In an embodiment, mesh may be a graph structure where nodes of the graph are: 1) edges of the combined facial image and 2) points of change of facial features (e.g., nose, lips).
[0073] M = (V, F)
[0074] Here, M may represent the mesh containing all the information of nodes and the adjacency metrics corresponding to it. V may be a set of 3D points (x, y, z coordinates). F may be a set of polygonal faces connecting the vertices.
[0075] Referring again to Figure 7A,the at least one processor may be configured to process the generated 3D mesh to determine a numerical representation of facial data of the user where each point on the 3D mesh (vertex) may be encoded in terms of its coordinates. These numerical representations may form a high-dimensional dataset that may represent the shape and features of the user's face. Further, data standardization may be applied to the numerical facial data. Standardization may ensure that each feature contributes equally by scaling the data to have a mean of zero and a standard deviation of one.
[0076] Referring to Figure 7B, once standardized, the at least one processor may be configured to generate a distance map based on the numerical representation of the facial data. For example, the distance between vertices may be computed using Euclidean distance which helps in identifying the relative spatial relationships across the facial structure, forming the basis for generating a distance map.
[0077] The distance map may highlight the variation in facial geometry and may be analysed by the at least one processor to determine the principal components. The principal components may capture the most significant variations in the data and may be used in the dimensionality reduction step via a Principal Component Analysis (PCA) technique. This technique may transform the high-dimensional data into a lower-dimensional space, preserving only the most important features.
[0078] Referring to Figure 7C, using the generated distance map, the personalized HRTF module 118may be configured to calculate the user's first HRTF by solving an acoustic equation iteratively as shown below Equation 4:
[0079] (Equation 4)
[0080] Here,pis the acoustic pressure,kis the wave number given by . Herefis the frequency of the sound stream andcis the speed of sound stream. Further, by solving this equation across different frequencies, angles (azimuth and elevation), and responses for each ear (left and right), the personalized HRTF module 118 may generate the first HRTF of the user.
[0081] In an embodiment of the disclosure, the personalized HRTF module 118 may be configured to obtain a second HRTF of the user based on a feedback assisted differentiator technique using a plurality of sound waves of a defined (e.g., predefined, predetermined) frequency range. In an embodiment, for the feedback-assisted differentiator technique for calculating the HRTF of the user, the user may wear a listening device such as but not limited to earbuds and may listen to sound waves emitted from the user's device for example using the user's phone speakers. In an embodiment, these sound waves may be generated to span a defined (e.g., predefined, predetermined) frequency range for example but not limited from 20 Hz to 20,000 Hz. incremented in steps of 1000 Hz. However, the scope of the disclosure is not confined to this frequency range and may be varied.
[0082] Further, the earphones equipped with microphones may record the sound waves as received by the left and right ears, respectively to provide at least three crucial information: the original input audio stream received by the listening device x(t), the left ear received audio signal yL(t) and the right ear received audio signal yR(t). The personalized HRTF module 118may be configured to convert the above three crucial information in time domain to frequency domain by computing the Fourier transforms of the respective time domain signals resulting in X(f), YL(f), and YR(f) respectively.
[0083] Further, the personalized HRTF module 118may be configured to calculate a transfer function for the left ear HL(f) and the transfer function for the right ear HR(f) using the Equation 5:
[0084] (Equation 5)
[0085] In an embodiment, the above calculation for the left and right ear may be repeated across the entire set of frequencies in the defined (e.g., predefined, predetermined) range to obtain a detailed frequency-dependent profile of how the user's ears perceive the input audio stream.
[0086] Further, after calculating the HRTFs for all the frequencies, the personalized HRTF module 118may be configured to obtain the second HRTF of the user by averaging the results over the entire frequency set. Further, the personalized HRTF module 118may normalize the generated first HRTF based on the facial features of the user with the obtained second HRTF to obtain the personalized HRTF of the user.
[0087] In an embodiment, the sound source localization module 120 may determine, using the corresponding sound localization model, the estimated location of the sound source corresponding to the sub audio stream based on at least one of the computed ILD or the computed IPD, and the generated HRTF responses of the user, as explained in subsequent paragraphs.
[0088] In an embodiment, the sound source localization module 120 may feed multiple inputs to the sound localization model. In an embodiment, the sound source localization module 120 may consolidated the multiple inputs to form a tensor further to be fed into the sound localization model. In a non-limiting embodiment, the tensor may be four-dimensional tensor formed using the multiple inputs. However, the scope of the disclosure is not confined to four-dimensional tensor and may be varied.
[0089] In an embodiment, the first input among the multiple input may include the computed ILD or the IPD, which are crucial auditory information that the human brain uses for sound localization. In an embodiment, the sound source localization module 120 may be configured to normalizethe calculated ILD or the IPD for the fast performance of the sound localization model across a plurality of dataset and inputs. In an embodiment, the normalized ILD or the IPD may form the first dimension of the sound source localization module model as input.
[0090] In an embodiment, the second input may be obtained from the HRTF responses of the input audio stream received by the listening device corresponding to the left ear and the right ear of the user based on the personalized HRTF of the user and the input audio stream. In an embodiment, the at least one processor may be configured to multiply the input audio stream S(f) with the personalized HRTF obtained for each ear of the user to obtain the HRTF responses for each of the user's ear. For example, for the left ear, the at least one processor may be configured to obtain the HRTF response for the left ear using the below Equation 6:
[0091] (Equation 6)
[0092] where is the left-ear personalised HRTF of the user. The at least one processor may repeat the same process for the right ear, obtaining SR(f) (HRTF response for the right ear). Further, these obtained HRTF responses for both the ears may form the second and third dimensions of the sound source localization module model as input.
[0093] Hence, these obtained inputs ILD or the IPD, SL(f), SR(f) may be combined by the sound source localization module 120 to form the four-dimensional tensor to be further fed as the input to the sound localization model. Referring back to Figure 6, in an embodiment, the switching module 114 may send the output of the any one of the IPD or ILD responses to the corresponding sound localization model to determine the estimated location of the sound sources based on the computed ILD or the computed IPD, and the generated HRTF responses of the input audio stream received by the listening device corresponding to the left ear and the right ear of the user. In an embodiment, the four-dimensional tensor formed may be used as input to a Deep Neural Network (DNN) based architectures for the corresponding sound localization model to obtain the location of the sound source. In an embodiment, the sound source localization module 120 may structure the dimensions as follows:
[0094] - Batch size (commonly set to 64),
[0095] - ILD or the IPD values,
[0096] - HRTF responses for the left ear,
[0097] - HRTF responses for the right ear.
[0098] In an embodiment, the sound source localization module 120 using the sound localization model may output the location of the sound source in the angular output format in the three-dimensional space. In an embodiment, for all the sound sources, the sound source localization module 120 may feed input raw sub-streams to the sound localization model to obtain the source location for all of the sources.
[0099] In an embodiment, the sound source localization module 120 may train the sound localization model from a publicly available dataset, using a non-limiting batch size of 64 typically trained over 360 epochs. Once trained, the soundlocalization model may accurately estimate the location of sound sources in real-time. For example, for each audio sub-stream (like bass, clap, or siren), the sound localization model may predict its unique spatial location, as shown below in Table 6.
[0100] Table 6 below shows the obtained output from the sound localization model to determine the estimated location of the sound source corresponding to the sub audio stream based on at least one of the computed ILD or the computed IPD, and the generated HRTF responses.
[0101]
[0102] In an embodiment, the calibration module 122may calibrate the determined estimated location of the sound source based on a directional orientation of the user to determine a precise location of the corresponding sound source as explained in Figure 8 below.
[0103] Figure 8 illustrates a block diagram of the calibration module 122, in accordance with an embodiment of the disclosure. In an embodiment, the calibration module 122 may be configured to determine the directional orientation of the user using one or more sensors embedded in the listening device. In an embodiment, after the sound localization model has estimated the location of the sound source position in terms of angular coordinates , this directional information must be adjusted or calibrated based on the direction the user is facing using the calibration module 122.
[0104] In an embodiment, the calibration module 122may utilize one or more sensors embedded in the listening device. For example, the one or more sensors may be but not limited to gyroscopes and accelerometers, to determine the precise orientation of the user's head. Additionally, the calibration module 122 may incorporate compass data to provide absolute orientation of the user with respect to cardinal directions (north, south, east, west). Further, the calibration module 122may combine the obtained orientation data with the orientation of the user's head to recalculates the sound source directions relative to the user's facing direction, as shown in Figure 8.
[0105] For instance, if the sound localization model detects a sound source at a particular angular position, but the user has turned to face a different direction, the calibration module 122 may adjust the coordinates accordingly so that the displayed or perceived direction remains accurate. As a result, the final output known as the calibrated output may translate angular coordinates into directions like East, Southeast, Southwest, North or Northwest, which aligns with the user's actual orientation to determine the precise location of the corresponding sound source.
[0106] Figure 9 illustrates a sequence flow of a method 900 of locating sound sources, in accordance with an embodiment of the disclosure. At operation 902, the method may comprise receiving an input audio stream by an apparatus. For example, the method may comprise receiving an input audio stream by at least one listening device being used by a user. In an embodiment, the input audio stream may include sounds generated by the plurality of sound sources present in the environment. In an embodiment, the received input audio stream may be stored in the memory 106 and further be processed by modules 110 of the apparatus 100 as explained above in Figure 1, and the same has not been repeated for the sake of brevity.
[0107] At operation 904, the method may comprise separating the input audio stream into a plurality of sub audio streams using a trained model such as the sound source identification module 112. In an embodiment, each sub audio stream of the plurality of sound sources may correspond to the respective sound source among the plurality of sound sources. The sound source identification module 112 may comprise at least one of the sound source classification module 302, the sound source separation module 304 and the scoring module 306. In an embodiment of the disclosure, the sound source classification module 302 may receive the input audio stream and may convert the input audio stream into the spectrogram which is the frequency domain representation of the input audio stream. Further the CNN architecture may be used for the feature extraction of the audio spectrogram to generate feature maps corresponding to the audio spectrogram as the output from the CNN architecture. Further, the generated feature maps may be fed to the linear classifier for classifying the input audio stream. For example, in a non-limiting scenario, the sound source classification module 302 may classify the input audio stream as siren, baby crying, microwave timer, doorbell, running water etc. Further, the classified audio stream may be fed as input to the trained model such as the sound source separation module 304 for the purpose of separation of the classified audio stream into a plurality of sub audio streams, as discussed below.
[0108] In an embodiment of the disclosure, the sound source separation module 304 may separate the input audio stream into the plurality of sub audio streams. In an embodiment, each sub audio stream of the plurality of sound sources may correspond to the respective sound source among the plurality of sound sources. The sound source separation module 304 may be the hybrid time-frequency domain-based sound separation model. All the sound sources may be identified by the hybrid time-frequency domain-based sound separation model the and separated into multiple different sub-streams, as shown above in Table 1 and has not been repeated for the sake of brevity. The hybrid time-frequency domain-based sound separation model may be separately trained on the single sound stream and corresponding multiple sound streams as output, to be further fed to the scoring module 306.
[0109] In an embodiment of the disclosure, the scoring module 306 upon receiving the plurality of sub audio streams may determine the score for each of the received plurality of sub audio streams. The scoring module 306 may select the set of sub audio streams among the plurality of sub audio streams for the classification. In an embodiment, the set of sub audio streams may be selected based on the comparison of the determined score for each of the plurality of sub audio streams with the defined (e.g., predefined, predetermined) threshold score. In an embodiment, the scoring module 306 may determine the score for each of the plurality of sub audio streams based on at least one of: similarity within the sub audio streams, historical response of the user corresponding to each sub audio streams, location of the user, and defined (e.g., pre-defined, predetermined) priority of the sub audio streams, as explained above in Figure 4 through Tables 2-5, and the same has not been repeated for the sake of brevity.
[0110] In an embodiment, the scoring module 306 after determining the score for each of the audio streams may select the set of sub audio streams among the plurality of sub audio streams based on the comparison of the determined score for each of the plurality of sub audio streams with a defined (e.g., predefined, predetermined) threshold score. For example, the determined score for each of the audio streams greater than the defined (e.g., predefined, predetermined) threshold score for each of the plurality of sub audio streams may be selected for further frequency classification and the rest sub audio streams may be filtered out. In a non-limiting embodiment, the defined (e.g., predefined, predetermined) threshold score may vary according to the user.
[0111] At operation 906, the method may comprise identifying each sub audio stream as either a high frequency stream or a low frequency stream by frequency classification based on a frequency of the sub audio stream and a defined frequency threshold. For example, the method may comprise classifying each sub audio stream based on a frequency of the sub audio stream and a defined frequency threshold. In an embodiment, the switching module 114 may separate the set of sub audio streams based on the frequency of each selected sub audio streams. For example, each sub audio stream may be analysed by the switching module 114 using the Moving Average Based Algorithm to compute the Weighted Moving Average (WMA) frequency for each of the sub audio stream. In an embodiment, the switching module 114 may divide the set of sub audio streams into a plurality of time windows. Further, within each window, the moving average-based algorithm may identify the frequency components for each set of the sub audio streams and may assign weights to each data point for the set of sub audio streams based on the amplitude of the corresponding sub audio stream. For example, the smallest amplitude within the window may be used as a reference, and the weights for other amplitudes may be calculated in comparison with the reference.
[0112] Further, the WMA may be computed by taking the weighted average of the frequencies in each window for each set of the sub audio streams. This WMA may be calculated using the Equation 7:
[0113] (Equation 7)
[0114] Here, P1, P2, ... are the frequencies at each sample space for each set of the sub audio streams, and W1, W2, ... are the corresponding weights for each set of the sub audio streams based on their corresponding amplitudes. Hence, each sub audio stream may be assigned the unique WMA classified frequency (e.g., f1, f2, f3...), which may reflect the dominant frequency behaviour of that particular sub audio stream, as shown in Figure 5.
[0115] Further, the switching module 114 may classify each sub audio stream as either the high frequency stream or the low frequency stream based on the classified frequency of the sub audio stream as explained above in the flowchart of Figure 6.
[0116] After calculating the WMA frequency for each sub-audio stream (at operation 600), the switching module 114 as explained in the flowchart of Figure 6 may compare the WMA frequency for each sub-audio stream against the defined (e.g., predefined, predetermined) threshold or the reference value (at operation 610). If the frequency exceeds the threshold, the audio stream may be categorized as the high-frequency (such as claps or sirens) and if the frequency falls below the threshold, the sub audio stream may be classified as the low-frequency (such as bass or certain vocal tones). In an embodiment of the disclosure, the defined threshold frequency may be but not limited to 1500 Hz. Hence if the WMA frequency of the sub-audio stream is greater than 1500 Hz, the audio stream may be categorized as the high-frequency (at operation 620) and if the WMA frequency of the sub-audio stream is lower than 1500 Hz, the audio stream may be categorized as the low-frequency (at operation 630). The switching module 114 may switch the frequency classified sub audio stream based on their frequency categorisation for further calculation.
[0117] At operation 908, the method may comprise computing an interaural measurement based on the frequency classification of the sub audio stream. For example, the method may comprise computing one of the Interaural Level Difference (ILD) or the Interaural Phase Difference (IPD) based on the frequency classification of the sub audio stream. In an embodiment, for each of the low frequency classified sub audio stream, the method 900 may compute the IPD (at operation 630). In an embodiment, for each of the high frequency classified sub audio stream, the method 900 may compute the ILD (at operation 620).
[0118] In an embodiment, the ILD is the difference in intensity of the sub audio stream arriving at the listening device corresponding to the left ear and the right ear of the user. The ILD may be computed in decibels (dB) using the below Equation 8:
[0119] (Equation 8)
[0120] Here, are the sound pressures at the left and right ears of the listening device used by the user, respectively due to the incoming sub audio streams.
[0121] In an embodiment, the IPD is the difference in phase of the sub audio stream arriving at the listening device corresponding to the left ear and the right ear of the user, and may be computed using the below Equation 9:
[0122] (Equation 9)
[0123] Here, d is the distance between the ears (typically ~0.2 meters in humans), is the angle of the incoming sub audio stream relative to the midline of the user, is the wavelength of the sub audio stream and 2 represents a full wave cycle in radians.
[0124] In an embodiment, the switching module 114 may send the output of the any one of the IPD or ILD responses to the corresponding sound localization model selected based on the frequency classification of the sub audio stream. In an embodiment, the corresponding sound localization model may be separate sound localization models used to determine location of the sound sources for each of the computed ILD (at operation 640) or the computed IPD (at operation 650), as shown in the flowchart of Figure 6. In an embodiment, the corresponding sound localization model determines the estimated location of the sound sources (at operation 660) based on the computed ILD or the computed IPD, and the generated HRTF responses of the input audio stream received by the listening device corresponding to a left ear and a right ear of the user.
[0125] At operation 910, the method may comprise generating HRTF responses of the input audio stream received by the apparatus based on a personalized HRTF and the input audio stream. For example, the method may comprise generating HRTF responses of the input audio stream received by the listening device corresponding to a left ear and a right ear of the user based on a personalized HRTF of the user and the input audio stream. Figure 7A, 7B, and 7C illustrate the example to generate the personalized HRTF of the user, in accordance with an embodiment of the disclosure.
[0126] In an embodiment to generate the personalized HRTF of the user, the method 900 may comprise generating the 3D mesh of the user's face using facial features of the user, as shown in Figure 7A. In an embodiment, the feature detection module 116 may capture the facial features of the user from the front, left, right side so that a complete feature is captured from each side. In an embodiment the process for extracting facial features of the user may begin with the user capturing their images from the front, left or the right side. In an embodiment, the feature detection module 116 may extract the facial features of the user using a recorded video of the user's face, to generate the facial 3D mesh of the user. In an embodiment, the feature detection module 116 may crop out the overlapping part by similarity matching. In an embodiment, the feature detection module 116 may combine the images except the cropped part. In an embodiment, the mesh representation of the user's face may be a graphical representation as shown below, which helps to get larger details of the face with its representation as set of vertices (V) and Faces (F). In an embodiment, mesh may be a graph structure where nodes of the graph are: 1) edges of the combined facial image and 2) points of change of facial features (e.g., nose, lips).
[0127] M = (V, F)
[0128] Here, M may represent the mesh containing all the information of nodes and the adjacency metrics corresponding to it. V may be a set of 3D points (x, y, z coordinates). F may be a set of polygonal faces connecting the vertices.
[0129] Referring again to Figure 7A, the method 900 may comprise processing the generated 3D mesh to determine a numerical representation of facial data of the user where each point on the 3D mesh (vertex) may be encoded in terms of its coordinates. These numerical representations may form the high-dimensional dataset that may represent the shape and features of the user's face. Further, data standardization may be applied to the numerical facial data. Standardization may ensure that each feature contributes equally by scaling the data to have a mean of zero and a standard deviation of one.
[0130] Referring again to Figure 7B, once standardized, the method 900 may comprise generating the distance map based on the numerical representation of the facial data. For example, the distance between vertices is computed using Euclidean distance which helps in identifying the relative spatial relationships across the facial structure, forming the basis for generating a distance map.
[0131] The distance map may highlight the variation in facial geometry and may be analysed to determine the principal components. The principal components may capture the most significant variations in the data and may be used in the dimensionality reduction step via a Principal Component Analysis (PCA) technique. This technique may transform the high-dimensional data into the lower-dimensional space, preserving only the most important features.
[0132] Referring again to Figure 7C, using the generated distance map, the personalized HRTF module 118 may calculate the user's first HRTF by solving the acoustic equation iteratively as shown below Equation 10:
[0133] (Equation 10)
[0134] Here,pis the acoustic pressure,kis the wave number given by . Herefis the frequency of the sound stream andcis the speed of sound stream. Further, by solving this equation across different frequencies, angles (azimuth and elevation), and responses for each ear (left and right), the personalized HRTF module 118 may generate the first HRTF of the user.
[0135] In an embodiment of the disclosure, the personalized HRTF module 118 may obtain the second HRTF of the user based on the feedback assisted differentiator technique using the plurality of sound waves of the predefined frequency range. In an embodiment, for the feedback-assisted differentiator technique for calculating the HRTF of the user, the user may wear the listening device such as but not limited to earbuds and may listen to sound waves emitted from the user's device for example using the user's phone speakers. In an embodiment, these sound waves may be generated to span the predefined frequency range for example but not limited from 20 Hz to 20,000 Hz. incremented in steps of 1000 Hz. However, the scope of the disclosure is not confined to this frequency range and may be varied.
[0136] Further, the earphones equipped with microphones may record the sound waves as received by the left and right ears, respectively to provide at least three crucial information: the original input audio stream received by the listening device x(t), the left ear received audio signal yL(t) and the right ear received audio signal yR(t). The personalized HRTF module 118 may convert the above three crucial information in time domain to frequency domain by computing the Fourier transforms of the respective time domain signals resulting in X(f), YL(f), and YR(f) respectively.
[0137] Further, the personalized HRTF module 118 may calculate the transfer function for the left ear HL(f) and the transfer function for the right ear HR(f) using the Equation 11:
[0138] (Equation 11)
[0139] In an embodiment, the above calculation for the left and right ear may be repeated across the entire set of frequencies in the predefined range to obtain the detailed frequency-dependent profile of how the user's ears perceive the input audio stream.
[0140] Further, after calculating the HRTFs for all the frequencies, the personalized HRTF module 118 may obtain the second HRTF of the user by averaging the results over the entire frequency set. Further, the personalized HRTF module 118 may normalize the generated first HRTF based on the facial features of the user with the obtained second HRTF to obtain the personalized HRTF of the user.
[0141] At operation 912, the method may comprise determining an estimated location of a sound source for the sub audio stream based on the computed interaural measurement and the generated HRTF responses using a sound localization model. For example, the method may comprise determining, using a corresponding sound localization model, an estimated location of a sound source corresponding to the sub audio stream based on at least one of the computed ILD or the computed IPD, and the generated HRTF responses.
[0142] In an embodiment, the sound source localization module 120 may feed multiple inputs to the sound localization model. In an embodiment, the sound source localization module 120 may consolidated the multiple inputs to form the tensor further to be fed into the sound localization model. In a non-limiting embodiment, the tensor may be four-dimensional tensor formed using the multiple inputs. However, the scope of the disclosure is not confined to four-dimensional tensor and may be varied.
[0143] In an embodiment, the first input among the multiple input may include the computed ILD or the IPD, which may be crucial auditory information that the human brain uses for sound localization. In an embodiment, the sound source localization module 120 may normalize the calculated ILD or the IPD for the fast performance of the sound localization model across the plurality of dataset and inputs. In an embodiment, the normalized ILD or the IPD may form the first dimension of the sound source localization module model as input.
[0144] In an embodiment, the second input may be obtained from the HRTF responses of the input audio stream received by the listening device corresponding to the left ear and the right ear of the user based on the personalized HRTF of the user and the input audio stream. In an embodiment, the method 900 may comprise multiplying the input audio stream S(f) with the personalized HRTF obtained for each ear of the user to obtain the HRTF responses for each of the user's ear. For example, for the left ear, the method 900 may comprise obtaining the HRTF response for the left ear using the below Equation 12:
[0145] (Equation 12)
[0146] where is the left-ear personalised HRTF of the user. The method 900 may comprise repeating the same process for the right ear, obtaining SR(f) (HRTF response for the right ear). Further, these obtained HRTF responses for both the ears may form the second and third dimensions of the sound source localization module model as input.
[0147] Hence, these obtained inputs ILD or the IPD, SL(f), SR(f) may be combined by the sound source localization module 120 to form the four-dimensional tensor to be further fed as the input to the sound localization model. Referring back to Figure 6, in an embodiment, the switching module 114 may send the output of the any one of the IPD or ILD responses to the corresponding sound localization model to determine the estimated location of the sound sources based on the computed ILD or the computed IPD, and the generated HRTF responses of the input audio stream received by the listening device corresponding to the left ear and the right ear of the user. In an embodiment, the four-dimensional tensor formed may be used as input to the Deep Neural Network (DNN) based architectures for the corresponding sound localization model to obtain the location of the sound source. In an embodiment, the sound source localization module 120 may structure the dimensions as follows:
[0148] - Batch size (commonly set to 64),
[0149] - ILD or the IPD values,
[0150] - HRTF responses for the left ear,
[0151] - HRTF responses for the right ear.
[0152] In an embodiment, the sound source localization module 120 using the sound localization model may output the location of the sound source in the angular output format in the three-dimensional space. In an embodiment, for all the sound sources, the sound source localization module 120 may feed input raw sub-streams to the sound localization model to obtain the source location for all of the sources.
[0153] In an embodiment, the sound source localization module 120 may train the sound localization model from the publicly available dataset, using the non-limiting batch size of 64 typically trained over 360 epochs. Once trained, the sound localization model may accurately estimate the location of sound sources in real-time. For example, for each audio sub-stream (like bass, clap, or siren), the sound localization model may predict its unique spatial location, as shown above in Table 6, and the same has not been repeated for the sake of brevity.
[0154] At operation 914, the method may comprise calibrating the estimated location of the sound source based on the directional orientation of a user to determine a location of the sound source. For example, the method may comprise calibrating the estimated location of the sound source based on the directional orientation of the user to determine a precise location of the corresponding sound source. In an embodiment, the calibration module 122 may determine the directional orientation of the user using one or more sensors embedded in the listening device. In an embodiment, after the sound localization model has estimated the location of the sound source position in terms of angular coordinates , the directional information must be adjusted or calibrated based on the direction the user is facing using the calibration module 122.
[0155] In an embodiment, the calibration module 122 may utilize one or more sensors embedded in the listening device. For example, the one or more sensors may be but not limited to gyroscopes and accelerometers, to determine the precise orientation of the user's head. Additionally, the calibration module 122 may incorporate compass data to provide absolute orientation of the user with respect to cardinal directions (north, south, east, west). Further, the calibration module 122 may combine the obtained orientation data with the orientation of the user's head to recalculates the sound source directions relative to the user's facing direction, as shown in Figure 8.
[0156] For instance, if the sound localization model detects the sound source at a particular angular position, but the user has turned to face a different direction, the calibration module 122 may adjust the coordinates accordingly so that the displayed or perceived direction remains accurate. As a result, the final output known as the calibrated output may translate angular coordinates into directions like East, Southeast, Southwest, North or Northwest, which aligns with the user's actual orientation to determine the precise location of the corresponding sound source.
[0157] In an embodiment of the disclosure, an apparatus for locating sound sources is provided. The apparatus may comprise memory storing instructions, and at least one processor operatively coupled to the memory and comprising processing circuitry.
[0158] In an embodiment of the disclosure, the apparatus, wherein the at least one processor individually or collectively executes the instructions to cause the apparatus to determine a score for each of the plurality of sub audio streams for the frequency classification of each sub audio stream. The at least one processor individually or collectively executes the instructions to cause the apparatus to select a set of sub audio streams among the plurality of sub audio streams for the frequency classification. wherein the set of sub audio streams is selected based on a comparison of the determined score for each of the plurality of sub audio streams with a defined threshold score.
[0159] In an embodiment of the disclosure, the apparatus, wherein the at least one processor individually or collectively executes the instructions to cause the apparatus to determine the score for each of the plurality of sub audio streams based on at least one of: similarity within the sub audio streams, historical response of the user corresponding to each sub audio streams, location of the user, or defined priority of the sub audio streams.
[0160] In an embodiment of the disclosure, the apparatus, wherein the at least one processor individually or collectively executes the instructions to cause the apparatus to generate a three-dimensional (3D) mesh of face of the user using facial features of the user. The at least one processor individually or collectively executes the instructions to cause the apparatus to process the 3D mesh to determine a numerical representation of facial data of the user. The at least one processor individually or collectively executes the instructions to cause the apparatus to generate a distance map based on the numerical representation of the facial data. The at least one processor individually or collectively executes the instructions to cause the apparatus to generate a first HRTF based on the generated distance map.
[0161] In an embodiment of the disclosure, the apparatus, wherein the at least one processor individually or collectively executes the instructions to cause the apparatus to obtain a second HRTF based on a feedback assisted differentiator technique using a plurality of sound waves of a defined frequency range. The at least one processor individually or collectively executes the instructions to cause the apparatus to normalize the generated first HRTF based on the facial features of the user with the obtained second HRTF to obtain the personalized HRTF of the user.
[0162] In an embodiment of the disclosure, the apparatus, wherein the interaural measurement may comprise an interaural level difference (ILD) or an interaural phase difference (IPD). The apparatus, wherein the IPD may be the difference in phase of the sub audio stream arriving at the apparatus corresponding to a left ear and a right ear of the user. The apparatus, wherein the ILD may be the difference in intensity of the sub audio stream arriving at the apparatus corresponding to the left ear and the right ear of the user.
[0163] In an embodiment of the disclosure, the apparatus, wherein the trained model may be a hybrid time-frequency domain-based sound separation model. The apparatus, wherein the hybrid time-frequency domain-based sound separation model may be separately trained on a single sound stream and corresponding multiple sound streams as output.
[0164] In an embodiment of the disclosure, the apparatus, wherein the at least one processor individually or collectively executes the instructions to cause the apparatus todetermine the directional orientation of the user using one or more sensors embedded in the apparatus.
[0165] In an embodiment of the disclosure, the apparatus, wherein the input audio stream may include sounds generated by a plurality of sound sources present in an environment.
[0166] In an embodiment of the disclosure, the apparatus, wherein each sub audio stream of the plurality of sound sources may correspond to a respective sound source among the plurality of sound sources.
[0167] In an embodiment of the disclosure, a method for locating sound sources is provided.
[0168] In an embodiment of the disclosure, the method may comprise determining a score for each of the plurality of sub audio streams for the frequency classification of each sub audio stream. The method may comprise selecting a set of sub audio streams among the plurality of sub audio streams for the frequency classification, wherein the set of sub audio streams is selected based on a comparison of the determined score for each of the plurality of sub audio streams with a defined threshold score.
[0169] In an embodiment of the disclosure, the method, wherein the score for each of the plurality of sub audio streams is determined based on at least one of: similarity within the sub audio streams, historical response of the user corresponding to each sub audio streams, location of the user, or defined priority of the sub audio streams.
[0170] In an embodiment of the disclosure, the method, wherein the personalized HRTF of the user may be generated by generating a three-dimensional (3D) mesh of face of the user using facial features of the user. The method, wherein the personalized HRTF of the user may be generated by processing the 3D mesh to determine a numerical representation of facial data of the user. The method, wherein the personalized HRTF of the user may be generated by generating a distance map based on the numerical representation of the facial data. The method, wherein the personalized HRTF of the user may be generated by generating a first HRTF based on the generated distance map. The method, wherein the personalized HRTF of the user may be generated by obtaining a second HRTF based on a feedback assisted differentiator technique using a plurality of sound waves of a defined frequency range. The method, wherein the personalized HRTF of the user may be generated by normalizing the generated first HRTF based on the facial features of the user with the obtained second HRTF to obtain the personalized HRTF of the user.
[0171] In an embodiment of the disclosure, the method, wherein the interaural measurement may comprise an interaural level difference (ILD) or an interaural phase difference (IPD). The method, wherein the IPD may be the difference in phase of the sub audio stream arriving at the listening device corresponding to the left ear and the right ear of the user. The method, wherein the ILD may be the difference in intensity of the sub audio stream arriving at the listening device corresponding to the left ear and the right ear of the user.
[0172] In an embodiment of the disclosure, the method, wherein the trained model may be a hybrid time-frequency domain-based sound separation model. The method, wherein the hybrid time-frequency domain-based sound separation model may be separately trained on a single sound stream and corresponding multiple sound streams as output.
[0173] In an embodiment of the disclosure, the method, wherein the directional orientation of the user may be determined using one or more sensors embedded in the apparatus.
[0174] In an embodiment of the disclosure, the method, wherein the input audio stream may include sounds generated by a plurality of sound sources present in an environment.
[0175] In an embodiment of the disclosure, the method, wherein each sub audio stream of the plurality of sound sources may correspond to a respective sound source among the plurality of sound sources.
[0176] Hence the disclosure proposes the apparatus 100 and method 900 which allows the user to achieve low-latency, computationally inexpensive sound source localization on the listening device for numerous applications on the listening device such as hearing aid, alarm and siren detection etc. The order in which the various operations of the methods are described is not intended to be construed as a limitation, and any number of the method described blocks can be combined in any order to implement the method. Additionally, individual blocks may be deleted from the methods without departing from the spirit and scope of the subject matter described herein. Furthermore, the methods can be implemented in any suitable hardware, software, firmware, or combination thereof. It may be noted here that the subject matter of some or all embodiments described with reference to Figures 1-8 may be relevant for the methods and the same is not repeated for the sake of brevity.
[0177] The various operations of methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software component(s) and / or module(s), including, but not limited to a circuit, an application specific integrated circuit (ASIC), or processor. Generally, where there are operations illustrated in Figures 1-9, those operations may be performed by any suitable corresponding counterpart means-plus-function components.
[0178] Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the disclosure. A computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to perform steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible items and exclude carrier waves and transient signals, e.g., non-transitory. Examples include Random Access Memory (RAM), Read-Only Memory (ROM), volatile memory, nonvolatile memory, hard drives, Compact Disc (CD) ROMs, Digital Video Disc (DVDs), flash drives, disks, and any other known physical storage media.
[0179] Certain aspects may comprise a computer program for performing the operations presented herein. For example, such a computer program product may comprise a computer readable media having instructions stored (and / or encoded) thereon, the instructions being executable by one or more processors to perform the operations described herein. For certain aspects, the computer program product may include packaging material.
[0180] Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.
[0181] The terms "an embodiment", "embodiment", "embodiments", "the embodiment", "the embodiments", "one or more embodiments", "some embodiments", and "one embodiment" mean "one or more (but not all) embodiments of the disclosure(s)" unless expressly specified otherwise.
[0182] The terms "including", "comprising", "having" and variations thereof mean "including but not limited to", unless expressly specified otherwise. The enumerated listing of items does not imply that any or all of the items are mutually exclusive, unless expressly specified otherwise. The terms "a", "an" and "the" mean "one or more", unless expressly specified otherwise.
[0183] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary a variety of optional components are described to illustrate the wide variety of possible embodiments of the disclosure.
[0184] When a single device or article is described herein, it will be readily apparent that more than one device / article (whether or not they cooperate) may be used in place of a single device / article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device / article may be used in place of the more than one device or article, or a different number of devices / articles may be used instead of the shown number of devices or programs. The functionality and / or the features of a device may be alternatively embodied by one or more other devices which are not explicitly described as having such functionality / features. Thus, other embodiments of the disclosure need not include the device itself.
[0185] Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art.
Claims
1.A method (900) for locating sound sources, the method (900) comprising:receiving (902), an input audio stream by an apparatus;separating (904), the input audio stream into a plurality of sub audio streams using a trained model;identifying (906), each sub audio stream as either a high frequency stream or a low frequency stream by frequency classification based on a frequency of the sub audio stream and a defined frequency threshold;computing (908), an interaural measurement based on the frequency classification of the sub audio stream;generating (910), head-related transfer function (HRTF) responses of the input audio stream received by the apparatus based on a personalized HRTF and the input audio stream;determining (912), an estimated location of a sound source for the sub audio stream based on the computed interaural measurement and the generated HRTF responses using a sound localization model, wherein the sound localization model is selected based on the frequency classification of the sub audio stream; andcalibrating (914), the estimated location of the sound source based on a directional orientation of a user to determine a location of the sound source.2.The method (900) of claim 1, further comprising:determining a score for each of the plurality of sub audio streams for the frequency classification of each sub audio stream; andselecting a set of sub audio streams among the plurality of sub audio streams for the frequency classification, wherein the set of sub audio streams is selected based on a comparison of the determined score for each of the plurality of sub audio streams with a defined threshold score.3.The method (900) of any one of claims 1 and 2, wherein the score for each of the plurality of sub audio streams is determined based on at least one of: similarity within the sub audio streams, historical response of the user corresponding to each sub audio streams, location of the user, or defined priority of the sub audio streams.4.The method (900) of any one of claims 1 to 3, wherein the personalized HRTF of the user is generated by:generating a three-dimensional (3D) mesh of face of the user using facial features of the user;processing the 3D mesh to determine a numerical representation of facial data of the user;generating a distance map based on the numerical representation of the facial data;generating a first HRTF based on the generated distance map;obtaining a second HRTF based on a feedback assisted differentiator technique using a plurality of sound waves of a defined frequency range; andnormalizing the generated first HRTF based on the facial features of the user with the obtained second HRTF to obtain the personalized HRTF of the user.5.The method (900) of any one of claims 1 to 4, wherein the interaural measurement comprises an interaural level difference (ILD) or an interaural phase difference (IPD), wherein the IPD is the difference in phase of the sub audio stream arriving at the apparatus corresponding to a left ear and a right ear of the user, and wherein the ILD is the difference in intensity of the sub audio stream arriving at the apparatus corresponding to the left ear and the right ear of the user.6.The method (900) of any one of claims 1 to 5, wherein the trained model is a hybrid time-frequency domain-based sound separation model, and wherein the hybrid time-frequency domain-based sound separation model is separately trained on a single sound stream and corresponding multiple sound streams as output.7.The method (900) of any one of claims 1 to 6, wherein the directional orientation of the user is determined using one or more sensors embedded in the apparatus.8.The method (900) of any one of claims 1 to 7, wherein the input audio stream includes sounds generated by a plurality of sound sources present in an environment.9.The method (900) of any one of claims 1 to 8, wherein each sub audio stream of the plurality of sound sources corresponds to a respective sound source among the plurality of sound sources.10.An apparatus for locating sound sources, the apparatus (100) comprising:memory (106) storing instructions; andat least one processor (102) operatively coupled to the memory (106) and comprising processing circuitry,wherein the at least one processor (102) individually or collectively executes the instructions to cause the apparatus to:receive an input audio stream;separate the input audio stream into a plurality of sub audio streams using a trained model;identify each sub audio stream as either a high frequency stream or a low frequency stream by frequency classification based on a frequency of the sub audio stream and a defined frequency threshold;compute an interaural measurement based on the frequency classification of the sub audio stream;generate head-related transfer function (HRTF) responses of the input audio stream received by the apparatus based on a personalized HRTF and the input audio stream;determine an estimated location of a sound source for the sub audio stream based on the computed interaural measurement and the generated HRTF responses using a sound localization model, wherein the sound localization model is selected based on the frequency classification of the sub audio stream; andcalibrate the estimated location of the sound source based on a directional orientation of a user to determine a location of the sound source.11.The apparatus (100) of claim 10, wherein the at least one processor (102) individually or collectively executes the instructions to cause the apparatus to:determine a score for each of the plurality of sub audio streams for the frequency classification of each sub audio stream; andselect a set of sub audio streams among the plurality of sub audio streams for the frequency classification, wherein the set of sub audio streams is selected based on a comparison of the determined score for each of the plurality of sub audio streams with a defined threshold score.12.The apparatus (100) of any one of claims 10 and 11, wherein the at least one processor (102) individually or collectively executes the instructions to cause the apparatus to determine the score for each of the plurality of sub audio streams based on at least one of: similarity within the sub audio streams, historical response of the user corresponding to each sub audio streams, location of the user, or defined priority of the sub audio streams.13.The apparatus (100) of any one of claims 10 to 12, wherein the at least one processor (102) individually or collectively executes the instructions to cause the apparatus to:generate a three-dimensional (3D) mesh of face of the user using facial features of the user;process the 3D mesh to determine a numerical representation of facial data of the user;generate a distance map based on the numerical representation of the facial data;generate a first HRTF based on the generated distance map;obtain a second HRTF based on a feedback assisted differentiator technique using a plurality of sound waves of a defined frequency range; andnormalize the generated first HRTF based on the facial features of the user with the obtained second HRTF to obtain the personalized HRTF of the user.14.The apparatus (100) of any one of claims 10 to 13, wherein the interaural measurement comprises an interaural level difference (ILD) or an interaural phase difference (IPD), wherein the IPD is the difference in phase of the sub audio stream arriving at the apparatus corresponding to a left ear and a right ear of the user, and wherein the ILD is the difference in intensity of the sub audio stream arriving at the apparatus corresponding to the left ear and the right ear of the user.15.A computer-readable storage medium storing instructions, wherein the instructions, when executed by at least one processor (102), cause the at least one processor (102) to perform the method of any one of claims 1 to 9.
