Audio processing method, terminal, storage medium, and program product

By configuring an irregular microphone array in a portable device and employing spherical harmonic beamforming and temporal alignment strategies, HOA coefficients are generated, solving the problem of high-order coding performance degradation caused by insufficient microphone quantity, and achieving high-precision audio signal processing and sound field reconstruction.

WO2026232289A1PCT designated stage Publication Date: 2026-11-12BEIJING XIAOMI MOBILE SOFTWARE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2025/092975
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-05-06
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

In existing technologies, the application of HOA encoding methods in portable devices is limited, especially due to insufficient microphones leading to a decline in high-order encoding performance and an inability to effectively reconstruct complex sound fields.

Method used

By configuring irregular microphone arrays and combining spherical harmonic beamforming and time-domain alignment strategies, HOA coefficients are generated to achieve high-order audio coding, breaking through the limitation of the number of microphones and optimizing the high-frequency phase response.

Benefits of technology

It improves the signal fidelity of audio signals and the accuracy of sound field reconstruction, enhances the immersive listening experience, and is suitable for audio processing in complex sound field environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2025092975_12112026_PF_FP_ABST
    Figure CN2025092975_12112026_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to an audio processing method, a terminal, a storage medium, and a program product. The method comprises: a terminal acquires a first audio signal collected by a microphone array, the microphone array comprising at least one microphone device, and on the basis of a first higher order ambisonics (HOA) coefficient, perform order upscaling and encoding on the first audio signal to generate a second audio signal. Therefore, on the basis of the microphone array configuration of the terminal, HOA encoding of the audio signal is realized, the signal fidelity of an output audio is improved, and the sound field reconstruction accuracy is significantly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Audio processing methods, terminals, storage media, and software products Technical Field

[0001] This disclosure relates to the technical field of audio signal processing, and more particularly to an audio processing method, terminal, storage medium, and program product. Background Technology

[0002] Higher Order Ambisonics (HOA) technology is one of the mainstream technologies for spatial audio, with advantages including device independence and head motion compensation capabilities. Among related technologies, HOA encoding relies on regular spherical microphone arrays (SMA). However, how HOA encoding methods can be applied to portable devices such as terminals requires further research. Summary of the Invention

[0003] To improve the output quality of audio signals, embodiments of this disclosure provide an audio processing method, a terminal, a storage medium, and a program product.

[0004] According to a first aspect of the embodiments of this disclosure, an audio processing method is provided, executed by a terminal, the method comprising:

[0005] Acquire a first audio signal collected by the microphone array of the terminal, wherein the microphone array includes at least one microphone device;

[0006] The first audio signal is up-encoded according to the first high-order background sound effect HOA coefficient to generate the second audio signal.

[0007] According to a second aspect of the present disclosure, an audio processing apparatus is provided for use in a terminal, the apparatus comprising:

[0008] A transceiver module is used to acquire a first audio signal collected by the microphone array of the terminal, wherein the microphone array includes at least one microphone device;

[0009] The processing module is used to perform up-order encoding on the first audio signal according to the first high-order background sound effect HOA coefficient to generate a second audio signal.

[0010] According to a third aspect of the present disclosure, a terminal is provided for performing the audio processing method described in any one of the first aspects of the present disclosure.

[0011] According to a fourth aspect of the present disclosure, a storage medium is provided that stores instructions that, when executed on a terminal, cause the terminal to perform an audio processing method as described in any one of the first aspects of the present disclosure.

[0012] According to a fifth aspect of the present disclosure, a program product is provided, comprising at least one of a program and instructions, wherein when the program and instructions are executed by a terminal, the program and instructions implement the steps of the audio processing method described in any of the first aspects of the present disclosure.

[0013] By adopting the above technical solution, at least the following beneficial technical effects can be achieved:

[0014] The terminal acquires a first audio signal collected by a microphone array, which includes at least one microphone device. The first audio signal is then up-encoded according to a first high-order background noise (HOA) coefficient to generate a second audio signal. Thus, based on the terminal's microphone array configuration, HOA encoding of the audio signal is achieved, improving the signal fidelity of the output audio and significantly enhancing the accuracy of sound field reconstruction. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings required for the description of the embodiments are introduced below. The following drawings are only some embodiments of this disclosure and do not impose specific limitations on the protection scope of this disclosure.

[0016] Figure 1 is a schematic diagram of a microphone array configuration of a terminal according to an embodiment of the present disclosure.

[0017] Figure 2A is a schematic flowchart illustrating an audio processing method according to an embodiment of the present disclosure.

[0018] Figure 2B is a flowchart illustrating a method for determining HOA coefficients according to an embodiment of the present disclosure.

[0019] Figure 2C is a schematic diagram illustrating the audio simulation experiment results according to an embodiment of the present disclosure.

[0020] Figure 2D is a schematic diagram illustrating real audio experimental results according to an embodiment of the present disclosure.

[0021] Figure 3A is a schematic flowchart illustrating an audio processing method according to an embodiment of the present disclosure.

[0022] Figure 3B is a schematic flowchart illustrating an audio processing method according to an embodiment of the present disclosure.

[0023] Figure 4 is a schematic flowchart illustrating an audio processing method according to an embodiment of the present disclosure.

[0024] Figure 5 is a schematic diagram of the structure of a terminal according to an embodiment of the present disclosure.

[0025] Figure 6 is a structural schematic diagram of an electronic device 600 according to an embodiment of the present disclosure.

[0026] Figure 7 is a schematic diagram of the structure of a chip 700 according to an embodiment of the present disclosure. Detailed Implementation

[0027] This disclosure provides an audio processing method, a terminal, a storage medium, and a program product.

[0028] In a first aspect, embodiments of this disclosure propose an audio processing method, executed by a terminal, the method comprising:

[0029] Acquire a first audio signal collected by the microphone array of the terminal, wherein the microphone array includes at least one microphone device;

[0030] The first audio signal is up-encoded according to the first high-order background sound effect HOA coefficient to generate the second audio signal.

[0031] In the above embodiments, based on the microphone array configuration of the terminal, HOA encoding of the audio signal is realized, which improves the signal fidelity of the output audio and significantly improves the accuracy of sound field reconstruction.

[0032] In conjunction with some embodiments of the first aspect, in some embodiments, the number of the at least one microphone device and the audio order of the second audio signal do not need to satisfy the first limiting condition.

[0033] In the above embodiments, by limiting the conditions for up-order encoding of audio signals in this embodiment, higher-order audio signals can be processed without increasing the number of microphone devices, thus improving the flexibility of audio processing.

[0034] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes:

[0035] Based on the geometric coordinates of the microphone array, determine the frequency response of the microphone array to the plane wave sound source in K discrete directions, where K is a positive integer;

[0036] Based on the frequency response, generate the array manifold matrix of the microphone array;

[0037] Determine the values ​​of multiple spherical harmonic functions corresponding to the K discrete directions respectively;

[0038] Based on the multiple spherical harmonic function values, the array manifold matrix, and the first audio order, a discrete spherical harmonic transformation (DSHT) matrix is ​​generated, where the first audio order is a set order of the terminal.

[0039] The first HOA coefficient is determined based on the array manifold matrix and the DSHT matrix.

[0040] In the above embodiments, by transforming the microphone sampling point constraint into a sound source direction constraint, the display of the number of microphones is bypassed, and the terminal achieves HOA encoding of the first audio signal.

[0041] In conjunction with some embodiments of the first aspect, in some embodiments, determining the first HOA coefficient based on the array manifold matrix and the DSHT matrix includes:

[0042] Based on the array manifold matrix and the DSHT matrix, determine the constraints on the first HOA coefficients;

[0043] The first HOA coefficient is determined based on the constraints.

[0044] In the above embodiments, the error between the reconstructed sound field and the original sound field can be minimized by using constraints, thereby improving the accuracy of sound field reconstruction and ensuring that the omnidirectional components of the sound field conform to physical rules.

[0045] In conjunction with some embodiments of the first aspect, in some embodiments, determining the multiple spherical harmonic function values ​​corresponding to the K discrete directions respectively includes:

[0046] The value of the spherical harmonic function is determined by the following formula:

[0047] Among them, the Let θ be the spherical harmonic function value in any discrete direction, φ be the polar angle, φ be the azimuth angle, n be a natural number less than or equal to the first audio order, and m be an integer greater than or equal to -n and less than or equal to n. For the related Legendre polynomial.

[0048] In the above embodiments, the sound field is decomposed into the coefficients of spherical harmonic functions, thereby enabling the encoding, reconstruction, sound source localization, and beamforming of the sound field, thus improving the immersive auditory experience.

[0049] In conjunction with some embodiments of the first aspect, in some embodiments, generating the Discrete Spherical Harmonic Transform (DSHT) matrix based on the plurality of spherical harmonic function values ​​and the array manifold matrix includes:

[0050] The DSHT matrix is ​​determined using the following formula:

[0051] Among them, the Let D(ω) be the spherical harmonic function value in any discrete direction, and let D(ω) be the array manifold matrix.

[0052] In the above embodiments, the discrete signals on the sphere can be converted into spherical harmonic coefficients through the DSHT matrix, thereby enabling the encoding, reconstruction, analysis and processing of the sound field, providing data support for creating an immersive audio experience.

[0053] In conjunction with some embodiments of the first aspect, in some embodiments, the constraints include:

[0054] Among them, the The first HOA coefficient, D(ω) is the array manifold matrix, S is the DSHT matrix, and e n,m It is a unit vector.

[0055] In the above embodiments, by optimizing the HOA coefficients through constraints, the error between the reconstructed sound field and the original sound field can be minimized, thereby improving the accuracy of sound field reconstruction and meeting specific sound field reconstruction requirements.

[0056] In conjunction with some embodiments of the first aspect, in some embodiments, determining the first HOA coefficient using regularized least squares based on the constraints includes:

[0057] The first HOA coefficient is determined using the following formula:

[0058] Among them, the The first HOA coefficient, D(ω) is the array manifold matrix, S is the DSHT matrix, and e n,m Let λ be a unit vector, and let λ be the regularization coefficient.

[0059] In the above embodiments, by determining the HOA coefficients from discrete sampling points on the sphere, an accurate description and processing of the three-dimensional sound field environment is achieved, thereby improving the accuracy of the output audio.

[0060] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes:

[0061] If the signal simulation degree of the second audio signal is determined to be lower than a set threshold, the second audio order is obtained. The second audio order is greater than the first audio order, and the first audio order is the initial set order of the terminal.

[0062] The second HOA coefficient is determined based on the second audio order and the first HOA coefficient;

[0063] The first audio signal is encoded according to the second HOA coefficient to generate the third audio signal.

[0064] In the above embodiments, there is no limit to the number of order increases of the audio signal. The number of order increases is determined by measuring the signal fidelity of the audio signal generated after the order increase. This avoids incorrect order increases that could reduce the signal fidelity of the audio signal and enhances the realism of the output audio.

[0065] In conjunction with some embodiments of the first aspect, in some embodiments, the step of encoding the first audio signal according to the first high-order background sound effect HOA coefficient to generate the second audio signal includes:

[0066] Based on the first HOA coefficient, the third HOA coefficient of the fourth audio signal is determined, wherein the fourth audio signal is an audio signal in the first audio signal whose audio frequency is higher than a set frequency threshold.

[0067] The fourth audio signal is encoded according to the third HOA coefficient to generate the fifth audio signal;

[0068] The sixth audio signal is encoded according to the first HOA coefficient to generate a seventh audio signal. The sixth audio signal is an audio signal in the first audio signal whose audio frequency is lower than or equal to the set frequency threshold.

[0069] The second audio signal is generated based on the fifth audio signal and the seventh audio signal.

[0070] In the above embodiments, by separating phase and amplitude processing, directional information is preserved in the aliasing frequency band, thereby improving the high-frequency robustness of audio processing.

[0071] Secondly, embodiments of this disclosure provide an audio processing apparatus for use in a terminal, the apparatus comprising:

[0072] A transceiver module is used to acquire a first audio signal collected by the microphone array of the terminal, wherein the microphone array includes at least one microphone device;

[0073] The processing module is used to perform up-order encoding on the first audio signal according to the first high-order background sound effect HOA coefficient to generate a second audio signal.

[0074] Thirdly, embodiments of this disclosure provide a terminal for executing the audio processing method described in any one of the first aspects of this disclosure.

[0075] Fourthly, embodiments of this disclosure provide a storage medium storing instructions that, when executed on a terminal, cause the terminal to perform the audio processing method as described in any one of the first aspects of this disclosure.

[0076] Fifthly, embodiments of this disclosure provide a program product comprising at least one of a program and instructions, wherein when the program or instructions are executed by a terminal, they implement the steps of the audio processing method described in any of the first aspects of this disclosure.

[0077] Sixthly, embodiments of this disclosure provide a chip or chip system. The chip or chip system includes processing circuitry configured to perform the method described according to an optional implementation of the first aspect above.

[0078] It is understood that the aforementioned audio processing apparatus, terminal, storage medium, program product, etc., are all used to execute the methods proposed in the embodiments of this disclosure. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0079] This disclosure is not exhaustive, but merely illustrative of some embodiments, and is not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined, for example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments. In all embodiments of this disclosure, unless otherwise specified or logically conflicting, the terminology and / or descriptions between the embodiments are consistent and can be mutually referenced. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0080] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure.

[0081] In this embodiment of the disclosure, unless otherwise stated, elements expressed in the singular form, such as "a," "an," "the," "the," "the," "the," "the," "the," "this," etc., can mean "one and only one," or "one or more," "at least one," etc. For example, when using articles such as "a," "an," "the," etc. in translation, the noun following the article can be understood as either a singular expression or a plural expression.

[0082] In the embodiments of this disclosure, "multiple" refers to two or more.

[0083] In some embodiments, the terms "at least one of A or B, at least one of A and B", "one or more", "a plurality of", "multiple" and the like can be used interchangeably.

[0084] In some embodiments, the notation "at least one of A and B", "A and / or B", "A in one case, B in another", "in response to one case A, in response to another case B", etc., may include the following technical solutions depending on the situation: in some embodiments, A (execute A regardless of whether there is a branch B); in some embodiments, B (execute B regardless of whether there is a branch A); in some embodiments, execution is selected from A and B (A and B are selectively executed); in some embodiments, both A and B are executed. The same applies when there are more branches such as A, B, C, etc.

[0085] In some embodiments, the notation "A or B" may include the following technical solutions, depending on the situation: in some embodiments, A (execute A regardless of whether a branch B exists); in some embodiments, B (execute B regardless of whether a branch A exists); in some embodiments, execution is selected from A and B (A and B are selectively executed). The same applies when there are more branches such as A, B, and C.

[0086] The prefixes "first," "second," etc., used in the embodiments of this disclosure are merely for distinguishing different descriptive objects and do not impose restrictions on the position, order, priority, quantity, or content of the descriptive objects. The description of the descriptive objects is found in the claims or the context of the embodiments, and the use of prefixes should not constitute unnecessary restrictions. For example, if the descriptive object is a "field," the ordinal numbers preceding "field" in "first field" and "second field" do not restrict the position or order of the "fields." "First" and "second" do not restrict whether the "fields" they modify are in the same message, nor do they restrict the order of "first field" and "second field." Similarly, if the descriptive object is a "level," the ordinal numbers preceding "level" in "first level" and "second level" do not restrict the priority between "levels." Furthermore, the number of descriptive objects is not limited by ordinal numbers and can be one or more. For example, in "first device," the number of "devices" can be one or more. Furthermore, the objects modified by different prefixes can be the same or different. For example, if the object being described is "device", then "first device" and "second device" can be the same device or different devices, and their types can be the same or different. Similarly, if the object being described is "information", then "first information" and "second information" can be the same information or different information, and their content can be the same or different.

[0087] In some embodiments, “including A,” “containing A,” “for indicating A,” and “carrying A” can be interpreted as directly carrying A or indirectly indicating A.

[0088] In some embodiments, terms such as "time / frequency" and "time-frequency domain" refer to the time domain and / or frequency domain.

[0089] In some embodiments, terms such as “in response to…”, “in response to determining…”, “in the case of…”, “when…”, “when…”, “if…”, etc. can be used interchangeably. These descriptions all refer to the device making a corresponding action under certain objective circumstances. They do not necessarily limit the time, nor do they require the device to make a judgment action when implementing it, nor do they mean that there must be other limitations.

[0090] In some embodiments, the terms “greater than,” “greater than or equal to,” “not less than,” “more than,” “more than or equal to,” “not less than,” “higher than,” “higher than or equal to,” “not lower than,” and “above” can be used interchangeably, as can the terms “less than,” “less than or equal to,” “not greater than,” “less than,” “less than or equal to,” “not more than,” “lower than,” “lower than or equal to,” “not higher than,” and “below”.

[0091] In some embodiments, devices, etc., may be interpreted as physical or virtual, and their names are not limited to those described in the embodiments. Terms such as “device,” “equipment,” “circuit,” “network element,” “network function,” “network device,” “function,” “node,” “unit,” “section,” “system,” “network,” “chip,” “chip system,” “entity,” and “subject” are interchangeable.

[0092] In some embodiments, the terms "terminal", "terminal device", "user equipment (UE)", "user terminal", "mobile station (MS)", "mobile terminal (MT)", "subscriber station", "mobile unit", "subscriber unit", "wireless unit", "remote unit", "mobile device", "wireless device", "wireless communication device", "remote device", "mobile subscriber station", "access terminal", "mobile terminal", "wireless terminal", "remote terminal", "handset", "user agent", "mobile client", and "client" can be used interchangeably.

[0093] In some embodiments, the acquisition of data, information, etc., may comply with the laws and regulations of the country where the location is situated.

[0094] In some embodiments, data, information, etc., may be obtained with the user's consent.

[0095] Furthermore, each element, each row, or each column in the table of this disclosure can be implemented as an independent embodiment, and any combination of any element, any row, or any column can also be implemented as an independent embodiment.

[0096] In some embodiments, implementing neural HOA coding using a data-driven approach requires training the model with a large amount of real microphone signals, and its generalization ability is limited. Model-driven methods include sound field decomposition and least squares coding. Sound field decomposition relies on direction-of-arrival (DOA) estimation and source separation, and requires high accuracy. Least squares coding directly constructs a linear equation between the HOA coefficients and the array manifold, but is limited by the number of microphones and cannot overcome the order limitation of HOA coding.

[0097] In some embodiments, the least squares-based HOA coding method and the beamforming-based HOA coding method need to satisfy Q≥(N+1). 2 This approach cannot be applied to terminals that implement HOA encoding with four microphones, and for high-frequency audio signals, insufficient microphone spacing can easily lead to a decrease in HOA encoding performance. Methods that rely on DOA estimation or sound field models are highly complex and have limited generalization ability.

[0098] For example, construct a system of linear equations: AB = p by arraying manifold matrices, and solve for the pseudo-inverse matrix. As an encoding matrix, it only supports low-order HOA encoding due to limitations in the number of microphones. Underdetermined equations lead to the failure of high-order encoding and deterioration of the condition number in the high-frequency band. A 2.5D spherical harmonic beamforming mode is used to directly output the HOA coefficients. However, this relies on a regular array and does not solve the problem of irregular microphone arrangement at the terminal.

[0099] In some embodiments, a terminal microphone array HOA encoding and order-up method based on spherical harmonic beamforming is disclosed. Fourth-order HOA encoding is achieved using four microphones through spherical harmonic beamforming. A time-domain alignment strategy is introduced to optimize the high-frequency phase response, and the beamformer is designed directly based on the array manifold, avoiding the complex preprocessing steps in HOA encoding.

[0100] For example, this embodiment is applied to a HOA audio acquisition and processing scenario in a complex sound field environment. Figure 1 is a schematic diagram of the microphone array configuration of the terminal according to an embodiment of this disclosure. As shown in Figure 1, the terminal is configured with four irregular microphones: Mic A, Mic B, Mic C, and Mic D. Among them, Mic A is the main microphone located at the bottom of the terminal, Mic B is the secondary microphone located at the bottom of the terminal, Mic C is the microphone located at the top of the terminal, and Mic D is the microphone located at the back of the terminal. The maximum spacing between the microphones in the terminal is d. max =8cm. The terminal includes: an audio processing module for running the beamforming algorithm of this disclosure to generate a HOA signal of up to the fourth order; and an output interface for directly using the processed HOA signal for local audio recording on the terminal, real-time call noise reduction, or transmission to an external headset via Bluetooth. This embodiment overcomes the limitation on the number of microphones by using spherical harmonic beamforming to achieve fourth-order HOA encoding with four microphones, suppressing high-frequency aliasing, introducing a time-domain alignment strategy, optimizing high-frequency phase response, eliminating the need for DOA estimation, and allowing direct beamformer design based on array manifolds, avoiding complex audio preprocessing.

[0101] In some embodiments, the signal transmission process of this disclosure includes:

[0102] Step 1: The microphone array collects the sound pressure signals in the current sound field environment of the terminal: p1(ω), p2(ω), p3(ω), p4(ω). For example, p1(ω) is the sound pressure signal collected by Mic A, p2(ω) is the sound pressure signal collected by Mic B, p3(ω) is the sound pressure signal collected by Mic C, and p4(ω) is the sound pressure signal collected by Mic D.

[0103] Step 2: Generate HOA coefficients using beamforming algorithm Where n≤4, |m|≤n;

[0104] Step 3: The HOA signal is used for terminal storage (e.g., spatial audio storage in video recording) or real-time communication (e.g., directional noise reduction).

[0105] In some embodiments, this embodiment is applicable to various application scenarios. For example, in a video recording scenario, when a user records video through a terminal, the terminal collects the audio signal of the ambient sound field through a microphone array, encodes it into a HOA signal, and can later reconstruct the 360° spatial audio through headphones. In a voice call scenario, in noisy environments, HOA encoding separates the direction of the target speaker, improving call clarity. In a multimedia playback scenario, when the terminal plays music, the HOA signal is combined with the headphone's virtual surround sound algorithm to enhance the stereo experience.

[0106] Figure 2A is a flowchart illustrating an audio processing method according to an embodiment of the present disclosure. As shown in Figure 2A, the embodiment of the present disclosure relates to an audio processing method executed by a terminal, the method including:

[0107] Step S2101: Acquire the first audio signal collected by the microphone array of the terminal.

[0108] For example, this embodiment is executed by a terminal equipped with a microphone array. The terminal uses this microphone array to collect a first audio signal from the current sound field environment. The terminal collects ambient audio from the sound field environment, obtains the first audio signal, and then encodes it to generate a second audio signal for output. The audio effect of the second audio signal is the same as or similar to that of the first audio signal, thereby achieving high-precision reproduction of the first audio signal and improving the accuracy of sound field reconstruction.

[0109] In some embodiments, the first audio signal is an audio signal of the sound field environment collected by the terminal through a microphone array.

[0110] For example, in this embodiment, the microphone array includes Q microphone devices, where Q is a positive integer. These microphone devices are positioned at different locations on the terminal, and since each microphone device has a different location, the audio signals collected by each microphone device are different. That is, the first audio signal represents the ambient audio signal collected by each microphone device. For example, if the microphone array includes 4 microphone devices, then the first audio signal is p(ω) = [p1(ω), p2(ω), p3(ω), p4(ω)]. T Where p1(ω) is the audio signal acquired by Mic A, p2(ω) is the audio signal acquired by Mic B, p3(ω) is the audio signal acquired by Mic C, and p4(ω) is the audio signal acquired by Mic D.

[0111] In some embodiments, the name of the first audio signal is not limited, and may be, for example, "microphone acquisition signal", "ambient sound field signal", "ambient audio signal", "background audio signal", etc.

[0112] In some embodiments, the terminal achieves high-precision spatial audio capture by reconstructing the acquired audio signals. For example, the terminal's use cases may include at least one of the following: Virtual Reality (VR) applications, Augmented Reality (AR) applications, smart home applications, remote conferencing system applications, and drone acoustic monitoring applications.

[0113] In some embodiments, the output end of the second audio signal can be a terminal. For example, in a scenario where video recording is performed based on a terminal, the terminal's microphone array is used to collect audio from the ambient sound field to obtain a first audio signal. The first audio signal is encoded to generate a second audio signal, which is then associated with and saved with the recorded video data. Later, when the video data is played back through the terminal's speaker, the speaker is the output end of the second audio signal. The second audio signal is played back through the speaker, thereby restoring 360° spatial audio.

[0114] Optionally, in some embodiments, the output terminal of the second audio signal can also be other electronic devices. For example, in a voice call environment, the terminal's microphone array collects the audio signal of the ambient sound field (including user voice and ambient noise) to obtain a first audio signal. After the terminal encodes the first audio signal to generate a second audio signal, it sends the second audio signal to other electronic devices via wireless network communication. The second audio signal is then played using the speaker of the other electronic device. This achieves a high degree of fidelity in the voice call and improves the realism of the voice call.

[0115] In some embodiments, the microphone array includes at least one microphone device.

[0116] For example, in this embodiment, the arrangement of the microphone devices in the terminal can be regular or irregular. This embodiment does not impose any limitations. The terminal can set the number and arrangement of the microphone devices based on assembly requirements.

[0117] Optionally, in some embodiments, the number of at least one microphone device and the audio order of the second audio signal do not need to satisfy the first constraint condition.

[0118] For example, in this embodiment, the microphone array includes Q microphone devices, with the first constraint being: Q ≥ (N+1). 2 Where N represents the audio order of the second audio signal. That is, in this embodiment, the number of microphone devices included in the microphone array is not limited during the implementation of high-order audio encoding. For example, fourth-order audio encoding of the audio signal can be achieved by configuring four microphone devices; third-order audio encoding of the audio signal can be achieved by configuring two microphone devices; and fourth-order audio encoding of the audio signal can be achieved by configuring nine microphone devices.

[0119] It should be noted that in this embodiment, the terminal uses HOA technology to perform up-order encoding on the first audio signal to generate the second audio signal. The goal of HOA technology is to reproduce sound scenes in three-dimensional space. Compared to stereo or surround sound, HOA technology can more accurately describe the direction, distance, and spatial distribution of sound, allowing sound to move freely in three-dimensional space and achieving an immersive sound experience. The order in HOA technology refers to the highest order of the spherical harmonic function used to describe the three-dimensional sound field. This spherical harmonic function represents the sound signals distributed in three-dimensional space. The HOA order determines the detail and precision of the sound scene. The higher the order, the more refined the description of the sound scene, and the more sound details can be captured and reproduced.

[0120] For example, in related technologies, at least 25 microphone devices need to be configured in the terminal to implement fourth-order HOA encoding. This embodiment can achieve high-order HOA encoding using fewer microphone devices. For instance, the microphone array includes four microphone devices. In related technologies, these four microphone devices need to meet a first constraint, meaning they can only perform the highest-order encoding processing. However, in this embodiment, fourth-order HOA encoding can be implemented based on these four microphone devices.

[0121] Step S2102: The first audio signal is encoded in an up-order manner according to the first HOA coefficient to generate the second audio signal.

[0122] For example, in this embodiment, the terminal collects the first audio signal through the microphone array as: pQ (ω)=[p1(ω),p2(ω),p3(ω),…,p Q (ω)] T The first audio signal is HOA encoded based on the first HOA coefficient to obtain the second audio signal. During the terminal's testing phase, the terminal measures the plane wave sound source in a sound field simulation or anechoic chamber based on the arrangement of each microphone device in the microphone array. The constraint of the microphone sampling points is converted into a sound source direction constraint, transforming the limitation on the number of microphones into a limitation on the measurement direction, thereby generating the first HOA coefficient. During the terminal's application phase, the second audio signal is obtained by multiplying the first HOA coefficient by the acquired first audio signal.

[0123] Figure 2B is a flowchart illustrating a method for determining HOA coefficients according to an embodiment of the present disclosure. As shown in Figure 2B, this embodiment of the present disclosure relates to a method for determining HOA coefficients, executed by a terminal, and the method includes:

[0124] Step S2201: Determine the frequency response of the microphone array to the plane wave sound source in K discrete directions based on the geometric coordinates of the microphone array.

[0125] For example, in this embodiment, a three-dimensional coordinate system is constructed based on the terminal. The geometric coordinates of the microphone array are determined according to the position of each microphone device in the microphone array relative to the origin of this three-dimensional coordinate system. For instance, the center of gravity of the terminal is taken as the origin of the three-dimensional coordinate system. The extension direction of the long side of the terminal corresponding to the display device is the X-axis direction of the three-dimensional coordinate system, the extension direction of the short side of the display device perpendicular to the long side is the Z-axis direction of the three-dimensional coordinate system, and the direction perpendicular to the plane formed by the X-axis and Z-axis is the Y-axis direction of the three-dimensional coordinate system. After determining the three-dimensional coordinate system, the geometric coordinates (x, y, z) of the microphone array are obtained according to the position of each microphone device relative to the origin. Q y Q , z Q ).

[0126] In some embodiments, K is a positive integer. For example, this embodiment is applied to the experimental phase of the terminal, in K discrete directions (θ) k φ k On the other hand, through sound field simulation or anechoic chamber measurement, the frequency response d(ω,θ) of each microphone device in the microphone array to the plane wave sound source is obtained. k ,φ kWhere θ is the polar angle (the angle between the vector from the positive Z-axis to the point), ranging from [0, π]; φ is the direction angle (the angle between the projection of the vector from the positive X-axis to the point onto the XY plane), ranging from [0, 2π]. By measuring the frequency response of the microphone array in K discrete directions, the response characteristics of the microphone array to different frequencies and incident angles are determined, reflecting the sensitivity of the microphone array in different directions and the time delay of the sound wave arriving at the microphone array.

[0127] It should be noted that, to ensure the accuracy and regularity of the frequency response, the K discrete directions in this embodiment are uniformly sampled covering the sphere. This allows the microphone array to fully capture the sound field in all directions, avoiding the loss of sound field information in certain directions, ensuring that the sound field is smoothly reconstructed in all directions, reducing spectral leakage, and improving the accuracy of the spherical harmonic coefficients. For example, in this embodiment, directions can be selected at equal intervals on the sphere, or directions can be selected on the sphere such that the spherical area corresponding to each direction is the same, thereby achieving uniform sampling on the sphere.

[0128] Step S2202: Generate the array manifold matrix of the microphone array based on the frequency response.

[0129] For example, in this embodiment, the frequency responses of Q microphone devices in the microphone array are measured separately in K discrete directions. The frequency response d of each microphone device in each discrete direction is obtained. Q (ω,θ K ,φ K Based on this frequency response, an array manifold matrix is ​​constructed, resulting in the following array manifold matrix:

[0130] Where ω is the frequency response, Q is the microphone device number, and K is the direction number corresponding to the discrete direction.

[0131] Step S2203: Determine the values ​​of multiple spherical harmonic functions corresponding to the K discrete directions.

[0132] For example, in this embodiment, the spherical harmonic function values ​​in K discrete directions are determined to construct a spherical harmonic basis matrix, and the spherical harmonic basis matrix is ​​applied to encode and reconstruct the sound field.

[0133] Optionally, in some embodiments, the step of determining the multiple spherical harmonic function values ​​corresponding to the K discrete directions includes:

[0134] The value of the spherical harmonic function is determined by the following formula:

[0135] in, Let be the spherical harmonic function value in the discrete direction (θ, φ), where θ is the polar angle, φ is the azimuth angle, n is the order of the spherical harmonic function (where n is a natural number less than or equal to the first audio order), and m is an integer greater than or equal to -n and less than or equal to n. For the related Legendre polynomial.

[0136] Step S2204: Generate the Discrete Spherical Harmonic Transform (DSHT) matrix based on multiple spherical harmonic function values, the array manifold matrix, and the first audio order.

[0137] For example, the first audio order is the initial audio order set in the terminal, typically set to 3. Based on the first HOA coefficients, the terminal encodes the first audio signal into a second audio signal with an audio order of 3. In this embodiment, the spherical harmonic function values ​​at all discrete sampling points are constructed into a matrix, resulting in a DSHT (Discrete Spherical Harmonic Transform) matrix S. Each column of matrix S corresponds to a spherical harmonic function value. Each row corresponds to the discrete direction (θ, φ) of a discrete sampling point, and the matrix S can be represented as:

[0138] Where, l = n 2 The matrix has dimensions K×(N+1), where +n+m+1, 0≤n≤N, -n≤m≤n. 2 Calculate the matrix in This indicates a pseudo-inverse operation.

[0139] Optionally, in some embodiments, the above step "generating a Discrete Spherical Harmonic Transform (DSHT) matrix based on multiple spherical harmonic function values ​​and an array manifold matrix" includes:

[0140] The DSHT matrix is ​​determined using the following formula:

[0141] in, Let ω be the spherical harmonic function value in any discrete direction, and D(ω) be the array manifold matrix.

[0142] For example, this embodiment introduces a DSHT matrix as an intermediate mapping to project the irregular array response into the spherical harmonic space, avoiding dependence on the geometric symmetry of the microphone array and improving the flexibility and applicability of sound field reconstruction. By determining the mapping relationship between the response vector corresponding to the irregular microphone array and the spherical harmonic coefficient vector, the conversion from irregular sampling points to the spherical harmonic space is achieved. In this embodiment, the DSHT matrix S can be solved using the least squares method or other optimization methods.

[0143] Step S2205: Determine the first HOA coefficient based on the array manifold matrix and the DSHT matrix.

[0144] For example, in this embodiment, by combining the array manifold matrix D(ω) and the DSHT matrix S, the response of the microphone array can be converted into spherical harmonic coefficients. The HOA coefficients are elements of the spherical harmonic coefficient vector and can be extracted using the following formula:

[0145] Where e1 is a unit vector, representing a vector that is 1 at the first position and 0 at the other positions. The microphone array response is converted into spherical harmonic coefficients by the above extraction formula, and the HOA coefficients are determined based on these spherical harmonic coefficients.

[0146] Optionally, in some embodiments, step S2206 above includes:

[0147] Based on the array manifold matrix and the DSHT matrix, determine the constraints for the first HOA coefficient;

[0148] Determine the first HOA coefficient based on the constraints.

[0149] For example, in this embodiment, constraints are constructed for each HOA coefficient of the microphone array. These constraints ensure that the reconstruction and processing of the sound field meets application requirements. Based on these constraints, the HOA coefficients are optimized, thereby achieving more accurate, efficient, and robust sound field processing.

[0150] Optionally, in some embodiments, the constraint includes:

[0151] in, Let ω be the first HOA coefficient, D(ω) be the array manifold matrix, S be the DSHT matrix, and e be the first HOA coefficient. n,m It is a unit vector.

[0152] For example, in this embodiment It is (N+1) 2 A unit vector of dimension (n, m), where the (n, m)th element is 1 and all other elements are 0. Each HOA coefficient is converted into a unit vector using an arrayed manifold matrix and a DSHT matrix. This indicates that at frequency ω, the (n, m)th HOA coefficient is precisely extracted, while the other coefficients are suppressed.

[0153] For example, in this embodiment, the regularized least squares method minimizes the error between the reconstructed sound field and the original sound field while satisfying the constraints, thereby improving the stability and accuracy of the coefficients and ensuring the accuracy and stability of the first HOA coefficients.

[0154] Optionally, in some embodiments, the above step "determining the first HOA coefficient by regularized least squares method according to the constraints" includes:

[0155] The first HOA coefficient is determined using the following formula:

[0156] in, Let ω be the first HOA coefficient, D(ω) be the array manifold matrix, S be the DSHT matrix, and e be the first HOA coefficient. n,m Let λ be a unit vector and λ be the regularization coefficient.

[0157] Optionally, in some embodiments, the above-described audio processing method may further include:

[0158] If the signal simulation degree of the second audio signal is determined to be lower than the set threshold, the second audio order is obtained. The second audio order is greater than the first audio order, and the first audio order is the initial set order of the terminal.

[0159] The second HOA coefficient is determined based on the second audio order and the first HOA coefficient;

[0160] The first audio signal is encoded based on the second HOA coefficient to generate the third audio signal.

[0161] For example, in this embodiment, during the initial HOA encoding of the first audio signal, the input first audio order is a preset audio order of the terminal. Based on this first audio order, the terminal generates a first HOA coefficient corresponding to the first audio order using the aforementioned HOA coefficient determination method. After encoding the first audio signal based on the first HOA coefficient, a second audio signal with upgraded order is obtained. The encoded second audio signal is compared with the first audio signal to determine the signal fidelity of the second audio signal. This signal fidelity indicates the degree of difference between the second audio signal and the actual sound field audio received by the microphone. If the signal fidelity is lower than a set threshold, it is determined that the currently used first audio order is mismatched. The terminal can obtain a second HOA coefficient based on the second audio order and the aforementioned HOA coefficient determination method. Then, the first audio signal is encoded based on the second HOA coefficient to generate a third audio signal.

[0162] It should be noted that in this embodiment, the terminal measures the audio signal generated after each encoding to determine the signal fidelity of the audio signal. If the signal fidelity is lower than the set threshold, the order of the spherical harmonic function will be increased, new HOA coefficients will be generated, and the acquired audio signal will be re-encoded based on the newly generated HOA coefficients until the spherical harmonic function reaches the maximum order set by the terminal. If the signal fidelity is higher than or equal to the set threshold, the terminal will use the encoded audio signal as the output audio signal.

[0163] Optionally, in some embodiments, step S2102 above includes:

[0164] Based on the first HOA coefficient, the third HOA coefficient of the fourth audio signal is determined. The fourth audio signal is the audio signal in the first audio signal whose audio frequency is higher than a set frequency threshold.

[0165] The fourth audio signal is encoded based on the third HOA coefficient to generate the fifth audio signal;

[0166] The sixth audio signal is encoded according to the first HOA coefficient to generate the seventh audio signal. The sixth audio signal is an audio signal in the first audio signal whose audio frequency is lower than or equal to a set frequency threshold.

[0167] The second audio signal is generated based on the fifth and seventh audio signals.

[0168] For example, in this embodiment, by separating phase and amplitude processing, directional information is preserved in the aliasing frequency band, improving the robustness of high-frequency processing for high-frequency audio signals. The terminal performs frequency measurement on the acquired first audio signal to determine the frequency distribution of each type of audio in the first audio signal, and processes low-frequency audio signals and high-frequency audio signals separately. A frequency threshold is set as the spatial aliasing frequency value, which can be set as: ω th =2πc / (2d) max For example, d max =8cm, f th The first audio signal is defined as follows: (≈2125Hz), meaning the audio signals above 2125Hz in the first audio signal are the fourth audio signal, and the audio signals below 2125Hz in the first audio signal are the sixth audio signal. For the fourth audio signal, using the above-described HOA coefficient determination method, the array manifold matrix D(ω) is replaced with the amplitude response matrix |D(ω)|, thus ignoring phase information to avoid aliasing errors, resulting in the third HOA coefficient. The fourth audio signal is then encoded using this third HOA coefficient to generate the fifth audio signal. The sixth audio signal is then encoded using the first HOA coefficient to generate the seventh audio signal. The fifth and seventh audio signals are then mixed to form the second audio signal for output.

[0169] Figure 2C is a schematic diagram illustrating the audio simulation experiment results according to an embodiment of the present disclosure. Figure 2D is a schematic diagram illustrating the actual audio experiment results according to an embodiment of the present disclosure. As shown in Figures 2C and 2D, in the related art, HOA encoding is limited by Q≥(N+1). 2 In this embodiment, the limitation on the number of microphones is transformed into a limitation on the measurement direction vector. Experimental verification confirms that even with only four irregularly arranged microphone devices, fourth-order HOA coding and upscaling can be achieved.

[0170] In some embodiments, in simulation experiments, as the SNR (Signal to Noise ratio) decreased from 30dB to 0dB, the present method consistently outperformed the baseline method in terms of the εerror metric, especially under low SNR conditions (0dB), where the error of the present method (6.69) was significantly lower than that of the baseline method (10.90), indicating that it has excellent noise immunity.

[0171] In some embodiments, this method outperforms the baseline method at all SNR levels in terms of SDR metrics, indicating higher signal fidelity.

[0172] In some embodiments, simulation results show that system performance decreases with increasing RT60, mainly manifested as a significant increase in the error index εerror. However, compared to the baseline method, this method exhibits better robustness to reverberation.

[0173] In some embodiments, under mild reverberation conditions (RT60 = 0.2 s), the method reduces the error from 3.36 to 1.89, while under high reverberation conditions (RT60 = 2.0 s), the method maintains the error at 300.26, demonstrating a significant performance advantage compared to the baseline method's 1020.52.

[0174] In some embodiments, based on actual experimental results, this method significantly improves spatial correlation in the 2-5 kHz range while markedly reducing reconstruction error. At higher frequencies, the frequency-partitioned curves exhibit relatively stable performance, while the baseline method shows significant fluctuations.

[0175] In some embodiments, the above experiments confirmed the effectiveness of the frequency partitioning strategy of this method above 2kHz, demonstrating that maintaining frequency consistency is more effective than providing incorrect phase information when no phase information is provided. Spherical Harmonic Transform (SHT) also contributes to this high-frequency stability characteristic, which is fully reflected in the SDR metric.

[0176] In some embodiments, the accuracy of sound field reconstruction is significantly improved by upscaling to the 4th order, and the improvement from further upscaling is negligible. SDR metrics show that the baseline method is not only significantly lower than this method on the overall average, but also exhibits a more pronounced downward trend in the mid-to-high frequency range, indicating its deviation from the theoretical HOA coefficients.

[0177] The communication method involved in the embodiments of this disclosure may include at least one of steps S2101 to S2102. For example, step S2102 may be implemented as a standalone embodiment, but is not limited thereto.

[0178] In some embodiments, step S2102 is optional, and one or more of these steps may be omitted or substituted in different embodiments.

[0179] In some embodiments, the steps and their optional implementations in other embodiments described before or after this embodiment, as well as other related parts in the specification, can be referred to, and will not be repeated here.

[0180] Figure 3A is a schematic flowchart illustrating an audio processing method according to an embodiment of the present disclosure. As shown in Figure 3A, the present disclosure relates to an audio processing method, which includes:

[0181] Step S3101: Determine the frequency response of the microphone array to the plane wave sound source in K discrete directions based on the geometric coordinates of the microphone array.

[0182] The optional implementation of step S3101 can be found in the optional implementation of step S2201 in Figure 2B, and other related parts in the embodiments involved in Figure 2B, which will not be repeated here.

[0183] Step S3102: Generate the array manifold matrix of the microphone array based on the frequency response.

[0184] The optional implementation of step S3102 can be found in the optional implementation of step S2202 in Figure 2B, and other related parts in the embodiments involved in Figure 2B, which will not be repeated here.

[0185] Step S3103: Determine the values ​​of multiple spherical harmonic functions corresponding to the K discrete directions.

[0186] The optional implementation of step S3103 can be found in the optional implementation of step S2203 in Figure 2B, and other related parts in the embodiments involved in Figure 2B, which will not be repeated here.

[0187] Step S3104: Generate the Discrete Spherical Harmonic Transform (DSHT) matrix based on multiple spherical harmonic function values, the array manifold matrix, and the first audio order.

[0188] In some embodiments, the first audio order is the initial set order of the terminal.

[0189] The optional implementation of step S3104 can be found in the optional implementation of step S2204 in Figure 2B, and other related parts in the embodiments involved in Figure 2B, which will not be repeated here.

[0190] Step S3105: Determine the first HOA coefficient based on the array manifold matrix and the DSHT matrix.

[0191] The optional implementation of step S3105 can be found in the optional implementation of step S2205 in Figure 2B, and other related parts in the embodiments involved in Figure 2B, which will not be repeated here.

[0192] Step S3106: Acquire the first audio signal collected by the microphone array of the terminal.

[0193] In some embodiments, the microphone array includes at least one microphone device.

[0194] The optional implementation of step S3106 can be found in the optional implementation of step S2101 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0195] Step S3107: The first audio signal is encoded in an up-order manner according to the first high-order background sound effect HOA coefficient to generate the second audio signal.

[0196] The optional implementation of step S3107 can be found in the optional implementation of step S2102 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0197] The communication method involved in the embodiments of this disclosure may include at least one of steps S3101 to S3107. For example, steps S3106 and S3107 may be implemented as independent embodiments, but are not limited thereto.

[0198] In some embodiments, steps S3101-S3105 are optional, and one or more of these steps may be omitted or substituted in different embodiments.

[0199] In some embodiments, the steps and their optional implementations in other embodiments described before or after this embodiment, as well as other related parts in the specification, can be referred to, and will not be repeated here.

[0200] Figure 3B is a schematic flowchart illustrating an audio processing method according to an embodiment of the present disclosure. As shown in Figure 3B, the present disclosure relates to an audio processing method, which includes:

[0201] Step S3201: Acquire the first audio signal collected by the microphone array of the terminal.

[0202] In some embodiments, the microphone array includes at least one microphone device.

[0203] The optional implementation of step S3201 can be found in the optional implementation of step S2101 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0204] Step S3202: Determine the third HOA coefficient of the fourth audio signal based on the first HOA coefficient.

[0205] In some embodiments, the fourth audio signal is an audio signal in the first audio signal whose audio frequency is higher than a set frequency threshold.

[0206] The optional implementation of step S3202 can be found in the optional implementation of step S2102 in Figure 2A and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0207] Step S3203: Encode the fourth audio signal according to the third HOA coefficient to generate the fifth audio signal.

[0208] The optional implementation of step S3203 can be found in the optional implementation of step S2102 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0209] Step S3204: Encode the sixth audio signal according to the first HOA coefficient to generate the seventh audio signal.

[0210] In some embodiments, the sixth audio signal is an audio signal in the first audio signal whose audio frequency is lower than or equal to a set frequency threshold.

[0211] The optional implementation of step S3204 can be found in the optional implementation of step S2102 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0212] Step S3205: Generate a second audio signal based on the fifth and seventh audio signals.

[0213] The optional implementation of step S3205 can be found in the optional implementation of step S2102 in Figure 2A, and other related parts in the embodiments involved in Figure 2A, which will not be repeated here.

[0214] The communication method involved in the embodiments of this disclosure may include at least one of steps S3201 to S3205. For example, steps S3202+S3203+S3204+S3205 may be implemented as independent embodiments, but are not limited thereto.

[0215] In some embodiments, step S3201 is optional, and one or more of these steps may be omitted or substituted in different embodiments.

[0216] In some embodiments, the steps and their optional implementations in other embodiments described before or after this embodiment, as well as other related parts in the specification, can be referred to, and will not be repeated here.

[0217] Figure 4 is a flowchart illustrating an audio processing method according to an embodiment of the present disclosure. As shown in Figure 4, the embodiment of the present disclosure relates to an audio processing method executed by a terminal, the method including:

[0218] Step S4101: Input the geometric coordinates of the microphone array and construct the array manifold matrix.

[0219] For example, the geometric coordinates of the microphone array are (x... Q y Q , z Q ), where Q = 1, 2, 3, 4…K, in K discrete directions (θ) k φ k On the other hand, the frequency response d(ω,θ) of the microphone array to the plane wave sound source is obtained through sound field simulation or anechoic chamber measurement. k ,φ k ).

[0220] Construct an array manifold matrix D(ω) with dimension 4×K, i.e.:

[0221] Where, d q (ω,θ k ,φ k ) indicates that the q-th microphone is in the direction (θ) k ,φ k The complex response of ).

[0222] Step S4102: Input the target HOA order and direction sampling points to generate the discrete spherical harmonic transformation matrix.

[0223] For example, the input target HOA order N = 4, and the direction sampling point (θ) k φ k ), calculate the spherical harmonic function value at each sampling point to construct the spherical harmonic basis matrix. Its elements are:

[0224] Where 0≤n≤N, -n≤m≤n, the matrix dimension is K×(N+1). 2 Calculate the DSHT (Discrete Spherical Harmonic Transform) matrix. in This indicates a pseudo-inverse operation.

[0225] Step S4103: Input the array manifold matrix and the DSHT matrix, and solve the beamforming weights using the regularized least squares method.

[0226] For example, for each target HOA coefficient Construct constraint equations:

[0227] in, For unit vectors (l=n) 2 (+n+m+1 bits are 1).

[0228] Beamforming weights are solved using regularized least squares method.

[0229] Where λ is the Tikhnov regularization coefficient, used to suppress noise sensitivity.

[0230] Step S4104: Input the spatial aliasing frequency threshold and perform aliasing encoding on the audio signal.

[0231] For example, the spatial aliasing frequency threshold is: ωth=2πc / (2dmax), for example dmax=8cm, fth≈2125Hz.

[0232] In some embodiments, for high-frequency audio signals where ω > ωth, the array manifold D(ω) is replaced with the amplitude response matrix |D(ω)|, thereby ignoring phase information to avoid aliasing errors. The beamforming weights of the high-frequency audio signal are then recalculated. Used to optimize amplitude matching.

[0233] Step S4105: Based on the HOA coefficients, perform real-time HOA encoding on the audio signal to obtain the HOA audio signal for output.

[0234] For example, the input microphone signal p(ω) = [p1(ω), p2(ω), p3(ω), p4(ω)] T The microphone signal is multiplied by the HOA coefficient to obtain the HOA audio signal, which is calculated using the following formula:

[0235] In some embodiments, by transforming microphone sampling point constraints into sound source direction constraints, the limitation on the number of microphones is overcome, and fourth-order coding is achieved. Furthermore, a DSHT matrix S is introduced as an intermediate mapping to project the irregular array response onto the spherical harmonic space, avoiding reliance on array geometric symmetry.

[0236] In some embodiments, by separating phase and amplitude processing, directional information is preserved in the aliasing frequency band, thereby improving high-frequency robustness.

[0237] For example, in this embodiment, the spherical harmonic function is:

[0238] in, For Legendre polynomials.

[0239] In some embodiments, the steps and their optional implementations in other embodiments described before or after this embodiment, as well as other related parts in the specification, can be referred to, and will not be repeated here.

[0240] This disclosure also proposes an apparatus (also referred to as an electronic device, etc.) for implementing any of the above audio processing methods. For example, an apparatus is proposed that includes units or modules for implementing the steps performed by the electronic device in any of the above audio processing methods.

[0241] It should be understood that the division of units or modules in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the units or modules in the device can be implemented by a processor calling software: for example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above audio processing methods or to implement the functions of each unit or module of the above device. The processor can be, for example, a general-purpose processor, such as a Central Processing Unit (CPU) or a microprocessor, and the memory can be internal or external to the device. Alternatively, the units or modules in the device can be implemented in the form of hardware circuits. The functionality of some or all of the units or modules can be achieved through the design of these hardware circuits, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC). The functionality of some or all of the units or modules is achieved through the design of the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a programmable logic device (PLD). Taking a field-programmable gate array (FPGA) as an example, it can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files, thereby achieving the functionality of some or all of the units or modules. All units or modules of the above device can be implemented entirely through processor-called software, entirely through hardware circuits, or partially through processor-called software with the remaining parts implemented through hardware circuits.

[0242] In this embodiment, the processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a Central Processing Unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), or a digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. The logical relationships of the aforementioned hardware circuits are fixed or reconfigurable. For example, the processor is a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units or modules. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a Neural Network Processing Unit (NPU), a Tensor Processing Unit (TPU), or a Deep Learning Processing Unit (DPU).

[0243] It should be noted that the various embodiments / examples described above in this disclosure can be used in conjunction with the foregoing and / or subsequent embodiments, or they can be used independently. Whether used alone or in conjunction with the foregoing and / or subsequent embodiments, the implementation principle is similar. In this disclosure, some embodiments are described as implementations used together. Of course, such illustrative examples are not intended to limit the scope of this disclosure.

[0244] Figure 5 is a schematic diagram of a terminal according to an embodiment of the present disclosure. The terminal 500 is used to perform any of the above methods. In some embodiments, as shown in Figure 5, the terminal 500 may include at least one of a transceiver module 501, a processing module 502, etc.

[0245] In some embodiments, the transceiver module 501 is used to acquire a first audio signal collected by the microphone array of the terminal, the microphone array including at least one microphone device. The processing module 502 is used to perform up-order encoding on the first audio signal according to the first high-order background sound effect (HOA) coefficient to generate a second audio signal.

[0246] In some embodiments, the transceiver module may include a transmitting module and / or a receiving module, which may be separate or integrated. Optionally, the transceiver module may be interchangeable with a transceiver.

[0247] In some embodiments, the processing module may be a single module or may include multiple sub-modules. Optionally, the multiple sub-modules may each perform all or part of the steps required by the processing module.

[0248] In some embodiments, the processing module can be interchanged with the processor, and the transceiver module can be interchanged with the transceiver.

[0249] Figure 6 is a schematic diagram of the structure of an electronic device 600 according to an embodiment of the present disclosure. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0250] Referring to FIG6, the electronic device 600 may include one or more of the following components: processing component 602, memory 604, power supply component 606, multimedia component 608, audio component 610, input / output interface 612, sensor component 614, and communication component 616.

[0251] Processing component 602 typically controls the overall operation of electronic device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0252] In some embodiments, the processor 620 performs at least one of the processing steps.

[0253] Memory 604 is configured to store various types of data to support the operation of electronic device 600. Examples of this data include instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, videos, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0254] Power supply component 606 provides power to various components of electronic device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 600.

[0255] Multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0256] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0257] Input / output interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0258] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 can detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of electronic device 600, changes in position of electronic device 600 or a component of electronic device 600, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of electronic device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0259] Communication component 616 is configured to facilitate wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0260] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0261] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of an electronic device 600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0262] Figure 7 is a schematic diagram of a chip 700 according to an embodiment of the present disclosure. When the electronic device 600 can be a chip or a chip system, the schematic diagram of the chip 700 shown in Figure 7 can be referenced, but is not limited thereto.

[0263] Chip 700 includes one or more processors 701. Chip 700 is used to perform any of the above methods.

[0264] In some embodiments, chip 700 further includes one or more interface circuits 702. Optionally, terms such as interface circuit, interface, and transceiver pin can be used interchangeably. In some embodiments, chip 700 further includes one or more memories 703 for storing data and / or instructions. Optionally, all or part of the memories 703 may be located outside chip 700. Optionally, interface circuit 702 is connected to memory 703, and interface circuit 702 can be used to receive data and / or instructions from memory 703 or other devices, and interface circuit 702 can be used to send data and / or instructions to memory 703 or other devices. For example, interface circuit 702 can read data and / or instructions stored in memory 703 and send the data and / or instructions to processor 701.

[0265] In some embodiments, processor 701 performs at least one of the processing steps.

[0266] The modules and / or devices described in the various embodiments, such as virtual devices, physical devices, and chips, can be combined or separated arbitrarily as needed. Optionally, some or all steps can also be performed collaboratively by multiple modules and / or devices, which is not limited here.

[0267] This disclosure also proposes a storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform any of the above methods. Optionally, the storage medium is an electronic storage medium. Optionally, the storage medium is a computer-readable storage medium, but not limited thereto; it may also be a storage medium readable by other devices. Optionally, the storage medium may be a non-transitory storage medium, but not limited thereto; it may also be a temporary storage medium.

[0268] This disclosure also proposes a program product, including a program and / or instructions, which, when executed by an electronic device, cause the electronic device to perform any of the above methods. Optionally, the program product is a computer program product. Optionally, the program product is stored on the storage medium.

[0269] This disclosure also proposes a computer program that, when run on a computer, causes the computer to perform any of the above methods.

Claims

1. An audio processing method, executed by a terminal, characterized in that, The method includes: Acquire a first audio signal collected by the microphone array of the terminal, wherein the microphone array includes at least one microphone device; The first audio signal is up-encoded according to the first high-order background sound effect HOA coefficient to generate the second audio signal.

2. The method according to claim 1, characterized in that, The number of the at least one microphone device and the audio order of the second audio signal do not need to satisfy the first constraint condition.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Based on the geometric coordinates of the microphone array, determine the frequency response of the microphone array to the plane wave sound source in K discrete directions, where K is a positive integer; Based on the frequency response, generate the array manifold matrix of the microphone array; Determine the values ​​of multiple spherical harmonic functions corresponding to the K discrete directions respectively; Based on the multiple spherical harmonic function values, the array manifold matrix, and the first audio order, a discrete spherical harmonic transformation (DSHT) matrix is ​​generated, where the first audio order is a set order of the terminal. The first HOA coefficient is determined based on the array manifold matrix and the DSHT matrix.

4. The method according to claim 3, wherein determining the first HOA coefficient based on the array manifold matrix and the DSHT matrix comprises: Based on the array manifold matrix and the DSHT matrix, determine the constraints on the first HOA coefficients; The first HOA coefficient is determined based on the constraints.

5. The method according to claim 3 or 4, characterized in that, Determining the multiple spherical harmonic function values ​​corresponding to the K discrete directions includes: The value of the spherical harmonic function is determined by the following formula: Among them, the Let θ be the spherical harmonic function value in any discrete direction, φ be the polar angle, φ be the azimuth angle, n be a natural number less than or equal to the first audio order, and m be an integer greater than or equal to -n and less than or equal to n. For the related Legendre polynomial.

6. The method according to any one of claims 3-5, characterized in that, The step of generating the Discrete Spherical Harmonic Transform (DSHT) matrix based on the plurality of spherical harmonic function values, the array manifold matrix, and the first audio order includes: The DSHT matrix is ​​determined using the following formula: Among them, the Let D(ω) be the spherical harmonic function value in any discrete direction, and let D(ω) be the array manifold matrix.

7. The method according to claim 4, characterized in that, The constraints include: Among them, the The first HOA coefficient, D(ω) is the array manifold matrix, S is the DSHT matrix, and e n,m It is a unit vector.

8. The method according to claim 4 or 7, characterized in that, Determining the first HOA coefficient based on the constraints includes: The first HOA coefficient is determined using the following formula: Among them, the The first HOA coefficient, D(ω) is the array manifold matrix, S is the DSHT matrix, and e n,m Let λ be a unit vector, and let λ be the regularization coefficient.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: If the signal simulation degree of the second audio signal is lower than a set threshold, the second audio order is obtained. The second audio order is greater than the first audio order, and the first audio order is the set order of the terminal. The second HOA coefficient is determined based on the second audio order and the first HOA coefficient; The first audio signal is encoded according to the second HOA coefficient to generate the third audio signal.

10. The method according to any one of claims 1-9, characterized in that, The step of encoding the first audio signal according to the first high-order background sound effect HOA coefficient to generate the second audio signal includes: Based on the first HOA coefficient, the third HOA coefficient of the fourth audio signal is determined, wherein the fourth audio signal is an audio signal in the first audio signal whose audio frequency is higher than a set frequency threshold. The fourth audio signal is encoded according to the third HOA coefficient to generate the fifth audio signal; The sixth audio signal is encoded according to the first HOA coefficient to generate a seventh audio signal. The sixth audio signal is an audio signal in the first audio signal whose audio frequency is lower than or equal to the set frequency threshold. The second audio signal is generated based on the fifth audio signal and the seventh audio signal.

11. An audio processing apparatus, characterized in that, Applied to a terminal, the device includes: A transceiver module is used to acquire a first audio signal collected by the microphone array of the terminal, wherein the microphone array includes at least one microphone device; The processing module is used to perform up-order encoding on the first audio signal according to the first high-order background sound effect HOA coefficient to generate a second audio signal.

12. A terminal, characterized in that, The terminal is used to execute the audio processing method according to any one of claims 1-10.

13. A storage medium storing instructions, characterized in that, When the instruction is executed on the terminal, the terminal performs the audio processing method as described in any one of claims 1-10.

14. A program product comprising at least one of a program and instructions, characterized in that, When at least one of the programs or instructions is executed by the terminal, it implements the steps of the audio processing method according to any one of claims 1-10.