Method and apparatus for acquiring and processing multiple audio signals, and conference device

Abstract

Embodiments of the present application disclose a method and apparatus for acquiring and processing multiple audio signals, and a conference device. In the embodiments of the present application, an independent connection link and an independent audio acquisition link are provided for each audio input device, so that an application program can acquire a first audio signal from each first audio acquisition link by means of a corresponding first API interface of an operating system, and separately process the acquired first audio signal. The embodiments of the present application can separately acquire multiple audio signals, so that the application program can flexibly process each acquired audio signal, thereby satisfying requirements in different conference scenarios, and solving the technical problem in the prior art that a conference device cannot separately acquire each audio signal when simultaneously acquiring multiple audio signals.

Description

Multi-channel audio signal acquisition and processing methods, devices, and conference equipment Technical Field

[0001] This application relates to the field of data acquisition, and more particularly to a method, apparatus, and conferencing equipment for acquiring and processing multi-channel audio signals. Background Technology

[0002] Currently, the Android system only supports the acquisition of one HAL (Hardware Abstraction Layer) node. In scenarios that require the acquisition of multiple audio signals, the audio signals from multiple device nodes need to be mixed before they are acquired by the application layer of the Android system.

[0003] However, because this audio signal acquisition involves mixing the audio signals before transmitting them to the application layer, the application layer cannot play a single audio signal locally, nor can it perform individual data processing on each audio signal, such as selecting a specific audio signal for echo cancellation, noise reduction, gain control, or other digital signal processing. Furthermore, the mixing of multiple audio signals and subsequent processing by the application's algorithms can easily lead to audio distortion, thus degrading the user experience. Summary of the Invention

[0004] This application provides a method, apparatus, and conferencing equipment for acquiring and processing multiple audio signals, which solves the technical problem in the prior art that conferencing equipment cannot acquire each audio signal individually when acquiring multiple audio signals simultaneously.

[0005] In a first aspect, embodiments of this application provide a method for acquiring and processing multi-channel audio signals. The method is applicable to an application program in the application layer of a conferencing device. The conferencing device is used to connect to at least two audio input devices, and the connection links between the conferencing device and each audio input device are independent. An independent audio acquisition link is established between the application program and each audio input device. The method includes:

[0006] Obtain the link information corresponding to each audio acquisition link of the application, wherein the link information is used to identify whether the corresponding audio acquisition link is based on the operating system's standard audio acquisition framework settings;

[0007] Based on the link information corresponding to each of the audio acquisition links, a first audio acquisition link is determined in the audio acquisition links based on the audio acquisition framework set according to the operating system standard;

[0008] The first audio signal is acquired from the first audio acquisition link through the first API interface. Each first audio acquisition link corresponds to a first API interface. The first API interface is the interface for acquiring audio signals and is suitable for the case where the operating system is configured with the default data access permissions.

[0009] Each of the first audio signals is processed separately.

[0010] This application embodiment provides an independent connection link and audio acquisition link for each audio input device, enabling the application to acquire a first audio signal from each first audio acquisition link through the operating system's first API interface, and process the acquired first audio signal individually. This application embodiment enables the individual acquisition of multiple audio signals, allowing the application to flexibly process each acquired audio signal, supporting the needs of different conference scenarios, and solving the technical problem in the prior art where conference equipment cannot individually acquire each audio signal when simultaneously acquiring and processing multiple audio signals.

[0011] The first API interface is an AudioRecord interface, an OpenSL ES interface, or an AAudio interface.

[0012] This application provides a variety of first API interfaces, which users can choose according to their needs such as low latency, high performance or low cost development. While empowering different Android terminal platforms, it also provides strong support for the audio and video conferencing application ecosystem and supports the adaptation of conferencing applications from different manufacturers.

[0013] The first audio acquisition link corresponding to the first API interface has an independent device node, and each device node is independent of each other in the device driver layer and the hardware abstraction layer.

[0014] This application embodiment ensures system performance, stability, and compatibility by setting each audio acquisition link to have an independent device node, and by making each device node independent of each other in the device driver layer and hardware abstraction layer.

[0015] The step of acquiring the first audio signal from the first audio acquisition link through the first API interface includes:

[0016] The first audio signal is acquired from the audio framework layer of the first audio acquisition link through the first API interface.

[0017] As described above, the embodiments of this application acquire the first audio signal from the audio framework layer, which is suitable for most application scenarios. Since the audio framework layer shields the underlying hardware details, developers can focus more on the implementation of application logic without worrying about the optimization and management of underlying resources, thus reducing the development difficulty.

[0018] The step of acquiring the first audio signal from the first audio acquisition link through the first API interface includes:

[0019] The first audio signal is acquired from the device driver layer of the first audio acquisition link through the first API interface.

[0020] As described above, the embodiments of this application acquire the first audio signal from the device driver layer, thereby enabling the acquired first audio signal to have lower latency and meet the usage requirements of low latency or high real-time performance.

[0021] The method further includes:

[0022] Based on the link information corresponding to each of the audio acquisition links, candidate audio acquisition links that are not set up based on the audio acquisition framework of the operating system standard are identified in the audio acquisition links.

[0023] Based on the permission information of the application corresponding to each of the candidate audio acquisition links, a second audio acquisition link is determined from the candidate audio acquisition links;

[0024] The second audio signal is acquired from the second audio acquisition link through the second API interface. Each second audio acquisition link corresponds to one second API interface. The second API interface is the interface for acquiring the audio signal and is applicable when the operating system is configured with the corresponding custom data access permissions.

[0025] Each of the second audio signals is processed separately.

[0026] As described above, in this embodiment of the application, candidate audio acquisition links are set for audio acquisition frameworks that are not based on the operating system standard. After determining the second audio acquisition link according to the permission information corresponding to the application and the candidate audio acquisition link, the application can access the device node corresponding to the device driver layer in the second audio acquisition link through the second API interface to acquire the audio signal input by the audio input device. This further improves the flexibility of acquiring audio signals and expands the applicable scenarios for multi-channel audio signal acquisition and processing.

[0027] The permission information is used to identify whether the application is allowed to access data in the corresponding device driver layer. The step of determining the second audio acquisition link from the candidate audio acquisition links based on the permission information corresponding to the application and each candidate audio acquisition link includes:

[0028] Based on the permission information corresponding to each candidate audio acquisition link of the application, a second audio acquisition link that allows the application to access data in the corresponding device driver layer is determined among the candidate audio acquisition links;

[0029] The acquisition of the second audio signal from the second audio acquisition link via the second API interface includes:

[0030] The second audio signal is acquired from the device driver layer of the second audio acquisition link through the second API interface.

[0031] As described above, once the operating system grants applications permission to access data from the device driver layer of the audio acquisition link, the applications can access the device node corresponding to the audio input device in the device driver layer through the second API interface to acquire the audio signal input by the audio input device, further improving the flexibility of audio signal acquisition.

[0032] Secondly, embodiments of this application provide a multi-channel audio signal acquisition and processing device. The device is applicable to an application program in the application layer of a conferencing device. The conferencing device is used to connect to at least two audio input devices, and the connection links between the conferencing device and each audio input device are independent. An independent audio acquisition link is established between the application program and each audio input device. The device includes:

[0033] The information acquisition module is used to acquire the link information corresponding to each audio acquisition link of the application. The link information is used to identify whether the corresponding audio acquisition link is based on the audio acquisition framework settings of the operating system standard.

[0034] The first link determination module is used to determine, based on the link information corresponding to each of the audio acquisition links, the first audio acquisition link set by the audio acquisition framework based on the operating system standard in the audio acquisition links.

[0035] The first audio acquisition module is used to acquire a first audio signal from the first audio acquisition link through a first API interface. Each first audio acquisition link corresponds to a first API interface. The first API interface is an interface for implementing audio signal acquisition and is suitable for use when the operating system is configured with the default data access permissions.

[0036] The first audio processing module is used to process each of the first audio signals separately.

[0037] Thirdly, embodiments of this application provide a conference device, which includes a processor and a memory;

[0038] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0039] The processor is used to execute a multi-channel audio signal acquisition and processing method as described in the first aspect, according to instructions in the computer program.

[0040] Fourthly, embodiments of this application provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform a multi-channel audio signal acquisition and processing method as described in the first aspect. Attached Figure Description

[0041] Figure 1 is a schematic diagram of multi-channel audio signal acquisition and processing in the prior art according to an embodiment of this application.

[0042] Figure 2 is a flowchart illustrating a multi-channel audio signal acquisition and processing method provided in an embodiment of this application.

[0043] Figure 3 is a schematic diagram of an audio signal acquisition via a first audio acquisition link provided in an embodiment of this application.

[0044] Figure 4 is a schematic diagram of another method for acquiring audio signals through a first audio acquisition link, provided in an embodiment of this application.

[0045] Figure 5 is a flowchart illustrating another method for acquiring and processing multi-channel audio signals provided in an embodiment of this application.

[0046] Figure 6 is a schematic diagram of an embodiment of this application that provides an audio signal acquisition via a second audio acquisition link.

[0047] Figure 7 is a schematic diagram of a meeting scenario applicable to a meeting device provided in an embodiment of this application.

[0048] Figure 8 is a schematic diagram of the acquisition of shared audio auxiliary stream and microphone audio main stream using two AudioRecord interfaces according to an embodiment of this application.

[0049] Figure 9 is a schematic diagram of acquiring shared audio auxiliary stream and microphone audio main stream using the AudioRecord interface and OpenSL ES interface respectively, according to an embodiment of this application.

[0050] Figure 10 is a schematic diagram of using the AAudio interface to acquire shared audio auxiliary streams and microphone audio main streams according to an embodiment of this application.

[0051] Figure 11 is a schematic diagram of acquiring the main microphone audio stream and shared secondary audio stream using the AudioRecord / OpenSL ES / AAudio and ALSA-lib interfaces respectively, according to an embodiment of this application.

[0052] Figure 12 is a schematic diagram of a multi-channel audio signal acquisition and processing device provided in an embodiment of this application.

[0053] Figure 13 is a schematic diagram of the structure of a conference device provided in an embodiment of this application. Detailed Implementation

[0054] The following description and accompanying drawings fully illustrate specific embodiments of this application to enable those skilled in the art to practice them. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of embodiments of this application includes the entire scope of the claims and all available equivalents of the claims. In this document, each embodiment may be referred to individually or collectively by the term "invention," which is merely for convenience and is not intended to automatically limit the scope of the application to any single invention or inventive concept if more than one invention is disclosed. Relational terms such as "first" and "second" are used herein only to distinguish one entity or operation from another, without requiring or implying any actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed. The various embodiments in this document are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the structures, products, etc., disclosed in the embodiments, since they correspond to the disclosed parts, the descriptions are relatively simple; relevant details can be found in the method section.

[0055] Currently, the Android system only supports the acquisition of one HAL (Hardware Abstraction Layer) node. In scenarios requiring multi-channel audio acquisition, the audio signals from multiple device nodes need to be mixed before being acquired by the Android application layer. For example, in an audio / video conferencing scenario involving multiple audio input devices, such as microphones and HDMI (High Definition Multimedia Interface) audio input devices, the conferencing equipment needs to first mix the audio signals input from the microphone and the HDMI audio input device at the Audio HAL (Hardware Abstraction Layer), and then transmit the mixed audio to the audio framework layer for use by the application layer through a single HAL node. For example, as shown in Figure 1, Figure 1 is a schematic diagram of multi-channel audio signal acquisition and processing in the prior art provided by an embodiment of this application. In Figure 1, the audio acquisition framework of the conference device includes an application layer, a Java Framework layer, a Native Framework layer, an Audio HAL layer, a DSP (Digital Signal Processing) / Driver layer, a Hardware layer, and an IN / OUT layer.

[0056] The application layer encompasses numerous apps and related services and frameworks. For example, apps in the application layer could be conferencing applications or recording applications. The application layer includes AudioRecord and AudioTrack; AudioRecord is used to capture audio signals, and AudioTrack is used to play audio signals.

[0057] The Java Framework layer is a crucial component of the Android system architecture. It provides Android application developers with a rich set of Java interfaces and class libraries, allowing them to easily utilize various Android system functions without needing to concern themselves with the underlying implementation details. The Java Framework layer encapsulates many system services and functions, such as window management, content providers, notification management, and audio management, significantly reducing development difficulty and complexity. AudioRecord (audio recording) and AudioTrack (audio playback) are the core classes in the Java Framework layer for audio capture. AudioRecord is responsible for hardware resource management, permission verification, and the transmission of raw data streams for audio input, while AudioTrack handles device routing, multi-channel mixing, focus management, and low-latency optimization for audio output.

[0058] The Native Framework layer provides underlying support for the Java Framework layer and also provides a high-performance runtime environment for performance-critical applications or services (such as games and multimedia processing). The Native Framework layer contains many important modules, such as AudioServer. AudioRecord in the Native Framework layer is the core component for audio acquisition, capturing audio signals by calling the underlying audio driver. AudioTrack in the Native Framework layer is the core component for audio playback, providing a way to transmit audio signals to audio devices for playback. Applications can fill the AudioTrack buffer with decoded audio data (such as PCM format), and AudioTrack will automatically send this data to audio output devices (such as speakers or headphones) for playback. AudioManager is a class in the Android system's Java Framework layer used to manage audio input devices and routing strategies, ensuring that audio signals are captured from the correct audio input device (such as a microphone). AudioFlinger is an important service in the Native Framework layer, mainly responsible for managing audio devices, handling the mixing, routing, and scheduling of output audio streams. Audio Policy is responsible for managing and coordinating the use of audio resources to ensure that audio signals can be played in the best possible condition in different scenarios and on different devices.

[0059] The Audio HAL (Hardware Abstraction Layer) encapsulates the implementation details of the underlying audio hardware, providing a unified interface for upper-layer software and thus decoupling software from hardware. In the Audio HAL, the PCM (Pulse Code Modulation) Read / Write functions are used during audio acquisition. PCM Read reads the raw PCM data from the audio input device, passes it to upper-layer audio services or applications for further processing, and transfers the acquired PCM data to system memory for subsequent processing and storage. During audio output, PCM Write writes the PCM data provided by upper-layer applications or services to the audio output device, enabling the device to convert the PCM data into analog signals for audio playback. Additionally, the Mixer in the Audio HAL layer primarily mixes different input PCM data to generate a mixed audio stream.

[0060] The DSP / Driver layer comprises the ALSA Driver (Advanced Linux Sound Architecture driver) and the Device node. The DSP / Driver layer is the core module in the Linux kernel that directly manages hardware devices, responsible for abstracting hardware operations into a unified software interface. The DSP / Driver layer is primarily used for directly operating audio devices, managing hardware resources, supporting different hardware interface protocols, adapting to various audio codecs and bus standards, and responding to hardware interrupts, triggering audio signal transfer or status updates.

[0061] The ALSA Driver is the concrete implementation of the Linux audio driver layer, part of the DSP / Driver (device driver) layer, providing a standardized audio operation framework. The ALSA Driver primarily manages PCM, controls audio parameters (volume, channels, sample rate), and maintains synchronization between the audio clock and the system clock to prevent data stream drift. Device Nodes serve as the entry point for user space access to the kernel audio driver, exposing audio functionality through standard file operations and controlling access permissions for different users / processes to audio devices through file permissions. In summary, the DSP / Driver layer executes hardware operations, directly controlling physical devices; the ALSA Driver is the core implementer of audio functionality, providing standardized audio streams and control interfaces; and Device Nodes are the user space access point, abstracting away underlying complexity through file abstraction.

[0062] The Hardware layer is a physical component composed of various electronic components and devices. It provides the physical platform for upper-layer software and systems to run, and is responsible for performing actual operations and processing tasks. The Hardware layer includes ADCs (Analog-to-digital converters) and DACs (Digital-to-Analog converters). ADCs / DACs are used to convert analog audio signals to digital audio signals, and vice versa.

[0063] The IN / OUT layer is primarily responsible for managing data transmission and interaction between the system and external devices. The IN / OUT layer is used to connect audio devices, including audio input devices and audio output devices. Audio input devices include MIC (Microphone) and External audio devices, while audio output devices include Speakers. The DSP / Driver layer contains device nodes corresponding to the MIC, External audio device, and Speaker.

[0064] Specifically, as shown in Figure 1, during the audio signal acquisition process, the audio signals input from each audio input device undergo analog-to-digital conversion in the HardWare layer (external audio input devices can directly input digital audio signals without analog-to-digital conversion). After converting the audio signals into digital audio signals (PCM data), the PCM data is written to the kernel buffer managed by the ALSA Driver. The device node serves as the entry point for the Audio HAL layer to access this buffer. The Audio HAL layer initiates a PCM read operation, reading multiple channels of PCM data through the device node. Then, the Mixer mixes the multiple channels of PCM data to obtain the mixed audio data. The AudioFlinger in the Native Framework layer initiates a PCM Read operation to obtain the mixed audio data from the Audio HAL layer. Subsequently, the Native Framework layer converts the mixed audio data into target mixed audio data suitable for processing by the upper-layer APP, and writes the target mixed audio data into the APP's shared memory. The Java Framework layer accesses the shared memory to obtain the target mixed audio data. Finally, the APP in the application layer obtains the target mixed audio data from the Java Framework layer by calling AudioRecord, completing the audio signal capture.

[0065] However, existing methods for acquiring multiple audio signals have the following shortcomings:

[0066] (1) Weak practicality: The existing multi-channel audio signal acquisition method cannot fully support the use scenario of audio and video conferencing. Since the multi-channel audio signals are mixed at the bottom layer and then transmitted to the application in the application layer, the application cannot play a single audio signal locally. For example, it can play the audio signal transmitted by the external input audio device and not play the audio signal input by the microphone to prevent near-end feedback.

[0067] (2) Poor flexibility: The application cannot process each audio signal individually, such as whether to mute a certain audio signal or send a certain audio signal to a remote device, and to perform digital signal processing on a selected audio signal individually.

[0068] (3) Poor user experience: Since the application cannot process each audio signal individually, the sound quality of the mixed audio signal is poor. For example, the audio signal input by the HDMI audio input device is generally a high-fidelity signal, but after being mixed with the noisy external signal picked up by the microphone and processed by the application algorithm, it cannot be avoided that it will be suppressed and cause distortion.

[0069] In summary, to address the technical problem in existing conferencing equipment that cannot individually acquire each audio signal when simultaneously acquiring multiple audio signals, this application provides a method for acquiring and processing multiple audio signals. This application provides an independent connection link and audio acquisition link for each audio input device, enabling the application to acquire a first audio signal from each first audio acquisition link through the operating system's first API (Application Programming Interface) and process the acquired first audio signal individually. This application enables the individual acquisition of multiple audio signals, allowing the application to flexibly process each acquired audio signal, supporting the needs of different conferencing scenarios, and solving the technical problem in existing conferencing equipment that cannot individually acquire each audio signal when simultaneously acquiring multiple audio signals.

[0070] Figure 2 shows a flowchart of a multi-channel audio signal acquisition and processing method provided in this embodiment of the application. The multi-channel audio signal acquisition and processing method provided in this embodiment is applicable to applications in the application layer of a conferencing device. The conferencing device is a device specifically designed for audio and video conferencing control, connecting participants from multiple locations to achieve real-time two-way communication of voice and video. The application installed on the conferencing device can be a program for meeting management, remote collaboration, and control of audio and video devices. For example, it can be responsible for managing and controlling various audio and video devices in the meeting, as well as coordinating and adjusting sound and image parameters. In this embodiment, the conferencing device is used to connect to at least two audio input devices, where each audio input device is a device for inputting audio signals. For example, an audio input device can be a microphone or a device with a sound acquisition card (such as a computer), and the connection links between the conferencing device and each audio input device are independent. It should be noted that the independent connection links between the conferencing device and each audio input device mean that the hardware links between the conferencing device and each audio input device are independent; that is, each audio input device can be connected to the corresponding interface of the conferencing device independently without mutual interference. Independent connection links ensure that each audio input device is physically isolated from other devices, thus avoiding signal interference and crosstalk. Furthermore, the application establishes an independent audio acquisition link with each audio input device, which is used to acquire audio signals. After the conferencing equipment establishes a connection link with the audio input devices, the application scans and identifies all audio input devices connected to the conferencing equipment upon startup. The application assigns an independent audio acquisition channel to each identified audio input device to establish an independent audio acquisition link. These independent audio acquisition links ensure that each audio input device can be individually identified, controlled, and processed.

[0071] The multi-channel audio signal acquisition and processing method provided in this application includes:

[0072] Step 101: Obtain the link information corresponding to each audio acquisition link of the application. The link information is used to identify whether the corresponding audio acquisition link is based on the standard audio acquisition framework settings of the operating system.

[0073] In this embodiment, the conference equipment uses the Android operating system. The application first needs to obtain the link information corresponding to each audio acquisition link from the Android system. This link information is pre-written into the Android system and carries information about whether the audio acquisition link is set up based on the operating system's standard audio acquisition framework. A standard audio acquisition framework refers to an audio acquisition framework where the Audio HAL layer and below fully comply with the Android audio framework standard, and the audio input device is legally registered in the Android audio policy. Specifically, the Audio HAL layer implements standard interfaces, the device driver is based on ALSA, the data stream format (PCM), device node path, and permission control conform to the Android system specifications, and the audio input device is declared in the system configuration file and is not proprietary or exclusively occupied by the vendor. When the audio acquisition link is set up based on the operating system's standard audio acquisition framework, the audio acquisition link can interact well with various audio input devices and applications under a unified standard, achieving seamless integration and ensuring smooth audio signal acquisition and transmission, avoiding problems caused by incompatibility between audio input devices or applications.

[0074] In the audio acquisition chain, the DSP / Driver layer is the software layer responsible for direct interaction with hardware devices. It contains drivers for specific hardware devices, which are responsible for initializing, configuring, and controlling them. The Audio HAL layer, on the other hand, is the interface layer between the operating system kernel and the hardware circuitry. It hides the platform-specific hardware interface details, providing a unified hardware interface for the operating system and upper-layer applications. Regardless of whether the audio acquisition chain is based on the operating system's standard audio acquisition framework, it can access the corresponding audio input devices through the hardware interface provided by the Audio HAL layer, enabling normal audio signal acquisition.

[0075] Step 102: Based on the link information corresponding to each audio acquisition link, determine the first audio acquisition link in the audio acquisition link set up according to the operating system standard audio acquisition framework.

[0076] After obtaining the link information corresponding to each audio acquisition link, it is necessary to further determine the first audio acquisition link based on the audio acquisition framework set up according to the operating system standard among all audio acquisition links, based on the link information corresponding to each audio acquisition link.

[0077] Step 103: Acquire the first audio signal from the first audio acquisition link through the first API interface. Each first audio acquisition link corresponds to a first API interface. The first API interface is the interface for implementing audio signal acquisition and is suitable for use when the operating system is configured with the default data access permissions.

[0078] For the determined first audio acquisition link, the first audio signal is acquired from the first audio acquisition link through the first API interface. The first API interface is the interface for implementing audio signal acquisition, suitable for use when the operating system is configured with default data access permissions. That is, it is not limited by operating system permissions or native operating system interfaces, and audio signal acquisition can be achieved without changing any underlying application environment of the operating system. In this embodiment, each first audio acquisition link corresponds to one first API interface. Since the first audio acquisition link is based on the standard audio acquisition framework of the operating system, all links in the first audio acquisition link are connected through standard interfaces. Simultaneously, the operating system can automatically manage the access permissions of device nodes, without requiring applications to request special permissions. Therefore, the operating system can acquire the first audio signal from the first audio acquisition link through the first API interface. Furthermore, since each first audio acquisition link in this embodiment is independent, it can be ensured that the acquired first audio signal accurately reflects the original sound.

[0079] Step 104: Process each first audio signal separately.

[0080] After acquiring the first audio signal from each audio acquisition link, the application can process each first audio signal individually, such as performing echo cancellation, noise reduction, and gain control on the first audio signal separately.

[0081] The above-described embodiments of this application provide a method for acquiring and processing multi-channel audio signals. In this embodiment, the connection links between the conference equipment and each audio input device are independent, and an independent audio acquisition link is established between the application and each audio input device. For the first audio acquisition link set up based on the audio acquisition framework of the operating system standard, the application can acquire the first audio signal of the first audio acquisition link through the first API interface in the operating system and process each first audio signal individually. By providing independent connection links and audio acquisition links for each audio input device, this embodiment enables the application to acquire the first audio signal from each first audio acquisition link through the first API interface of the operating system and process the acquired first audio signal individually. This embodiment enables the individual acquisition of multiple audio signals, allowing the application to flexibly process each acquired audio signal, supporting the needs of different conference scenarios, and solving the technical problem in the prior art that conference equipment cannot individually acquire each audio signal when acquiring multiple audio signals simultaneously.

[0082] Based on the above embodiments, the first API interface is an AudioRecord interface, an OpenSL ES interface, or an AAudio interface. The first audio acquisition link corresponding to the first API interface has an independent device node, and each device node is independent of each other in the device driver layer and the hardware abstraction layer.

[0083] In this context, a device node is an abstract representation of a hardware device in the operating system. Each audio device has its corresponding device node. The device node for the first audio acquisition link corresponding to the first API interface is independent; for example, a microphone and an external audio input device each correspond to different device nodes. This independence allows the operating system to clearly distinguish between different audio devices, and applications can select which audio device to use by specifying a particular device node. Each device node is independent within the device driver layer, meaning that the driver code, resource management, and error handling for the audio device corresponding to each device node are separate within the device driver layer. Furthermore, each device node is independent within the hardware abstraction layer, meaning that the interface, device configuration, and state management for the audio device corresponding to each device node are independent within the hardware abstraction layer.

[0084] The following describes each of the first API interfaces:

[0085] (1) AudioRecord interface

[0086] AudioRecord is a high-level API provided by the Android SDK and is a Java class for audio resource management. The AudioRecord interface provides direct access to audio signals, is suitable for real-time audio processing, and is applicable to all Android versions. The AudioRecord interface is relatively simple and easy to use, suitable for most application scenarios. Due to its low cost, many manufacturers use this API to implement single-channel audio acquisition. When using the AudioRecord interface, the Android system platform needs to have two prerequisites: (1) There are at least two physical device nodes, and they are independent of each other in terms of hardware link, device driver layer, and hardware abstraction layer; (2) In addition, each physical device node can be mapped one-to-one with the physical device in the MediaRecorder.AudioSource class of the Android system. For example, the microphone can be specified as the audio source through MediaRecorder.AudioSource.MIC (microphone recording audio source); and the remote microphone can be specified as the audio source through MediaRecorder.AudioSource.REMOTE_SUBMIX (remote recording audio source).

[0087] (2) OpenSL ES (Open Sound Library for Embedded Systems, Embedded Audio Acceleration Standard) interface

[0088] The OpenSL ES interface is a cross-platform audio API available for Android and other platforms. It provides low-level access to audio devices, suitable for audio applications requiring high performance and low latency. While OpenSL ES is available on multiple platforms, not just Android, it has a relatively high learning curve and is best suited for advanced users. When implementing multi-channel audio acquisition, OpenSL ES requires each audio acquisition link to have an independent device node, an independent device driver layer, and an independent hardware abstraction layer to ensure system performance, stability, and compatibility.

[0089] (3) AAudio (Android Audio) interface

[0090] AAudio provides low-latency and high-performance audio processing, primarily designed for applications requiring extremely low latency, such as real-time audio processing and game audio. The AAudio interface is compatible with newer Android versions and features a more modern API design, such as support for switching between multiple devices, making it more suitable for current multi-channel audio streaming scenarios in audio and video conferencing. However, the AAudio interface relies on underlying system audio strategies, such as those at the hardware abstraction layer, making it relatively difficult to use. Using the AAudio interface requires that the hardware, system, and application software all have a framework for multi-channel audio signal acquisition and processing. For example, the device nodes for audio input devices such as microphones and USB devices must be independent (e.g., independent hardware IIS lines), and the audio input devices must be independent of each other in the device driver layer and the hardware abstraction layer.

[0091] This application provides a variety of first API interfaces, which users can choose according to their needs such as low latency, high performance or low cost development. While empowering different Android terminal platforms, it also provides strong support for the audio and video conferencing application ecosystem and supports the adaptation of conferencing applications from different manufacturers.

[0092] In one embodiment, step 103, acquiring the first audio signal from the first audio acquisition link via the first API interface, includes:

[0093] Step 1031: Acquire the first audio signal from the audio framework layer of the first audio acquisition link through the first API interface.

[0094] In this embodiment, as shown in Figure 3, which is a schematic diagram of acquiring audio signals through a first audio acquisition link according to an embodiment of this application, the audio acquisition framework of the first audio acquisition link includes an application layer, a Java Framework layer, a Native Framework layer, an Audio HAL layer, a DSP (Digital Signal Processing) / Driver layer, a Hardware layer, and an IN / OUT layer. The functions of each layer are similar to those in Figure 1, and will not be described again in this embodiment. The audio framework layer is the Java Framework layer. Specifically, in Figure 3, the IN / OUT layer is used to connect audio devices, which include audio input devices and audio output devices. Audio input devices include MIC (Microphone) and External audio devices, which can be HDMI audio input devices, USB audio input devices, wireless screen mirroring devices, or screen projection devices, etc. The audio output devices of the IN / OUT layer include speakers. The DSP / Driver layer contains device nodes corresponding to the MIC, external audio device, and speaker, respectively.

[0095] Specifically, as shown in Figure 3, during the audio signal acquisition process, the audio signal input from each audio input device undergoes analog-to-digital conversion in the HardWare layer (external audio input devices can directly input digital audio signals without analog-to-digital conversion). After converting the audio signal into a digital audio signal (PCM data), each channel of PCM data is written to the kernel buffer managed by the ALSA Driver. The Device node serves as the entry point for the Audio HAL layer to access this kernel buffer. The Audio HAL layer initiates a PCM read operation, reading each channel of PCM data from the ALSA Driver kernel buffer through the Device node. The AudioFlinger in the Native Framework layer initiates a PCM Read operation through the AIDL (Android Interface Definition Language) interface to obtain each channel of PCM data from the Audio HAL layer. Subsequently, the Native Framework layer converts each channel of PCM data into target PCM data suitable for processing by upper-layer software and writes each channel of target PCM data into the application's shared memory. The Java Framework layer accesses the shared memory to obtain each channel of target PCM data. Finally, the application layer's APP can obtain the target PCM data for each channel from the Java Framework layer through the AudioRecord interface, OpenSL ES interface, or AAudio interface, thereby completing the acquisition of multiple audio signals. In Figure 3, the APP's MIC Capture captures the audio signal corresponding to the MIC, and External audio Capture captures the audio signal corresponding to the External audio device.

[0096] In Figure 3, the Device information in the Audio HAL layer provides basic information about the audio devices, including their type, model, and manufacturer. This allows the system and upper-layer apps to clearly identify the connected audio devices for targeted operations and display. The basic information also includes device capabilities, such as the audio device's sampling rate range and supported audio formats, and the camera device's resolution and frame rate. Before acquiring audio signals, the ALSA drive interacts with the Device information in the Audio HAL layer to obtain basic information about the connected audio devices, such as device type, device ID, and default configuration parameters. This allows for accurate identification of the existing audio devices, enabling the initialization of device parameters and determination of the range and characteristics of various supported audio parameters. Similarly, the AudioManager, through interaction with Device information, also obtains basic information about the connected audio devices for subsequent targeted operations, such as determining output routes and allocating system resources.

[0097] The Device plug notify function in the Audio HAL layer monitors the plugging and unplugging status of audio devices in real time. When an audio device is plugged in or unplugged, it promptly sends a notification to the operating system and relevant components, enabling the operating system to respond accordingly. Device plug notify also triggers automatic system configuration; for example, it can trigger a series of automatic configuration operations from the operating system. When a new audio device is plugged in, the operating system can automatically load the corresponding driver based on the device information, perform device initialization and parameter configuration, allowing the audio device to be quickly put into use. During audio signal acquisition, the ALSA driver interacts with the Device plug notify function to obtain real-time audio device plugging and unplugging information, thereby dynamically loading and unloading driver resources.

[0098] Furthermore, during audio signal acquisition, AudioPolicy determines which audio input device to use, such as a microphone or external audio input device. AudioFlinger then acquires audio signals from the designated audio input device according to this policy, ensuring that the acquired audio signals meet the requirements of the app and the operating system. Additionally, AudioPolicy can allocate appropriate resources to different audio streams based on the app's needs and the operating system's resource availability, such as determining parameters like the sampling rate, number of channels, and audio format for each audio stream. AudioFlinger then creates and manages the audio streams based on the allocated resources, ensuring that each audio stream receives suitable resources for proper audio processing.

[0099] Additionally, the playback in Figure 3 refers to the playback of recorded audio. The APP can also play the previously recorded audio signal through the Speaker. Before the Speaker plays the audio signal, the echo signal Loopback needs to be obtained in the HardWare layer. By pre-registering the Loopback Device node corresponding to the Loopback in the DSP / Driver layer, the Loopback can be sent back to the APP to build the delay for echo cancellation and thus perform algorithm alignment.

[0100] As described above, the embodiments of this application acquire the first audio signal from the audio framework layer, which is suitable for most application scenarios. Since the audio framework layer shields the underlying hardware details, developers can focus more on the implementation of application logic without worrying about the optimization and management of underlying resources, thus reducing the development difficulty.

[0101] Based on the above embodiments, step 103, which involves acquiring the first audio signal from the first audio acquisition link via the first API interface, includes:

[0102] Step 1032: Acquire the first audio signal from the device driver layer of the first audio acquisition link through the first API interface.

[0103] In another embodiment, the application can also acquire the first audio signal from the device driver layer of the first audio acquisition link through the first API interface, where the device driver layer is the DSP / Drive layer. For example, as shown in Figure 4, which is a schematic diagram of another method for acquiring audio signals through the first audio acquisition link according to an embodiment of this application, when the APP calls the first API interface, the first API interface triggers interaction with the ALSA driver. After receiving the instruction from the first API interface, the ALSA driver begins to acquire audio signals. During the acquisition of audio signals, the audio signals input by each audio input device undergo analog-to-digital conversion in the HardWare layer (external audio input devices can directly input digital audio signals without analog-to-digital conversion). After converting the audio signals into digital audio signals (PCM data), each channel of PCM data is written to the kernel buffer managed by the ALSA Driver. The APP reads the PCM data from the kernel buffer managed by the ALSA Driver through the first API interface and processes each channel of PCM data separately.

[0104] As described above, the embodiments of this application acquire the first audio signal from the device driver layer, thereby enabling the acquired first audio signal to have lower latency and meet the usage requirements of low latency or high real-time performance.

[0105] This application also provides another method for acquiring and processing multi-channel audio signals, as shown in Figure 5. Figure 5 is a flowchart illustrating another method for acquiring and processing multi-channel audio signals provided in this application. The method shown in Figure 5 is a specific embodiment of the above-mentioned method for acquiring and processing multi-channel audio signals. The method for acquiring and processing multi-channel audio signals provided in this application includes:

[0106] Step 201: Obtain the link information corresponding to each audio acquisition link of the application. The link information is used to identify whether the corresponding audio acquisition link is based on the standard audio acquisition framework settings of the operating system.

[0107] Step 202: Based on the link information corresponding to each audio acquisition link, determine the first audio acquisition link in the audio acquisition link set up according to the operating system standard audio acquisition framework.

[0108] Step 203: Acquire the first audio signal from the first audio acquisition link through the first API interface. Each first audio acquisition link corresponds to a first API interface. The first API interface is the interface for implementing audio signal acquisition and is suitable for use when the operating system is configured with the default data access permissions.

[0109] Step 204: Process each first audio signal separately.

[0110] Step 205: Based on the link information corresponding to each audio acquisition link, identify candidate audio acquisition links that are not set up based on the operating system standard audio acquisition framework.

[0111] In this embodiment, it is also necessary to determine candidate audio acquisition links that are not based on the operating system's standard audio acquisition framework based on the link information. The reason why an audio acquisition link is not based on the operating system's standard audio acquisition framework may be that the manufacturer has not managed the device nodes of some devices separately at the Hardware Abstraction Layer (HAL) and above in the Android system, resulting in the device lacking a corresponding interface for the application to call, preventing the application from communicating with the device driver layer through the HAL, or the device node's device driver layer using a proprietary definition, etc.

[0112] Step 206: Based on the permission information corresponding to each candidate audio acquisition link of the application, determine the second audio acquisition link among the candidate audio acquisition links.

[0113] After identifying the candidate audio acquisition links, it is necessary to further obtain the permission information corresponding to each candidate audio acquisition link for the application. This permission information includes the application's data access permissions for each layer within the candidate audio acquisition links, such as data acquisition permissions or data write permissions. This permission information can be predefined and declared in the operating system. After obtaining the permission information corresponding to each candidate audio acquisition link, it is necessary to further determine the second audio acquisition link from among the candidate audio acquisition links based on the permission information. In one embodiment, the permission information is used to identify whether the application is allowed to access data in the corresponding device driver layer, that is, whether the application supports accessing the corresponding audio signal in the device node of the device driver layer, so that the application can acquire audio signals.

[0114] Step 206 involves determining the second audio acquisition link from among the candidate audio acquisition links based on the permission information corresponding to each candidate audio acquisition link in the application, including:

[0115] Based on the permission information corresponding to each candidate audio acquisition link, a second audio acquisition link is determined from the candidate audio acquisition links that allows the application to access data in the corresponding device driver layer.

[0116] In this embodiment, based on the permission information corresponding to each candidate audio acquisition link, a second audio acquisition link is determined among the candidate audio acquisition links, allowing the application to access data from the device driver layer of the corresponding candidate audio acquisition link. That is, the application can acquire audio signals by accessing the device node in the device driver layer of the second audio acquisition link. It should be noted that this permission needs to be granted to the application by the user in the Android system. In one embodiment, the permission information can also be specifically restricted to whether the application is allowed to access the device node of the audio input device pointed to by the second audio acquisition link in the device driver layer, thereby restricting the application from obtaining audio signals from the device nodes of the audio input devices corresponding to different second audio acquisition links.

[0117] Step 207: Acquire the second audio signal from the second audio acquisition link through the second API interface. Each second audio acquisition link corresponds to a second API interface. The second API interface is the interface for implementing audio signal acquisition and is suitable for when the operating system is configured with the corresponding custom data access permissions.

[0118] After determining the second audio acquisition link, the second audio signal can be acquired from the second audio acquisition link through the second API interface. In this embodiment, each second audio acquisition link corresponds to a second API interface. The second API interface is the interface for implementing audio signal acquisition and is suitable when the operating system is configured with corresponding custom data access permissions, where custom data access permissions are custom data access permissions. In this embodiment, the custom data access permissions need to define permissions that allow the application to access data in the corresponding device driver layer, or define permissions that allow the application to access the device node of the audio input device pointed to by the second audio acquisition link in the device driver layer. In this embodiment, the second API interface can be the ALSA-lib (Advanced Linux Sound Architecture Library) interface, where the ALSA-lib interface is the interface provided in the ALSA user-space library for interacting with the ALSA driver. As part of the Linux audio framework, the ALSA-lib interface provides direct access to audio input devices, but requires the operating system to grant permissions, and is suitable for application scenarios that require direct control of audio hardware. In addition, it can be understood that in this embodiment, the second audio signal is acquired from the device driver layer of the second audio acquisition link through the second API interface. In one embodiment, as shown in Figure 6, which is a schematic diagram of acquiring audio signals through a second audio acquisition link according to an embodiment of this application, when the audio acquisition link of the external audio device is not based on the standard audio acquisition framework of the operating system, after granting the application permission to access the device node corresponding to the external audio device in the device driver layer in the Android system, the application can acquire the second audio signal input from the external audio device through the ALSA-lib interface.

[0119] Step 208: Process each second audio signal separately.

[0120] After acquiring the second audio signal, the application can process each second audio signal individually.

[0121] As described above, in the embodiments of this application, for audio acquisition links that are not based on the standard audio acquisition framework of the operating system, when the operating system grants the application permission to access data from the device driver layer of the audio acquisition link, the application can access the device node corresponding to the audio input device in the device driver layer through the second API interface to acquire the audio signal input by the audio input device, which further improves the flexibility of acquiring audio signals and expands the applicable scenarios for multi-channel audio signal acquisition and processing.

[0122] In one embodiment, the meeting scenario applicable to the meeting equipment is shown in Figure 7. Figure 7 is a schematic diagram of a meeting scenario provided by an embodiment of this application. In Figure 7, the near-end meeting room includes acoustic environment preparation to support remote audio and video conferencing scenarios, such as meeting equipment used in meetings, microphones, speakers, and external audio input devices (such as HDMI audio input devices, USB audio input devices, wireless screen sharing devices, or screen projection devices, etc., with a computer as an example of an external audio input device in Figure 7). The meeting equipment is equipped with a meeting application. Two audio input paths need to be set up in the near-end meeting room: an auxiliary stream shared audio input path and a mainstream microphone call audio input path. The auxiliary stream shared audio input path is used to collect the shared audio auxiliary stream, and the mainstream microphone call audio input path is used to collect the mainstream microphone audio.

[0123] The setup of the auxiliary stream shared audio input path and the audio signal acquisition process are as follows:

[0124] First, connect the computer and the conferencing equipment via HDMI cable. During the meeting, the computer plays audio signals, such as audio content from the shared PowerPoint presentation or other audio / video content. Once the Android system on the conferencing equipment recognizes the connected computer, an independent audio acquisition link is established between the conferencing application and the computer for subsequent audio signal acquisition during the meeting. During the audio / video conference, the conferencing application acquires the computer's audio signal through the Android system's audio signal acquisition interfaces (AudioRecord interface / OpenSL ES interface / AAudio interface / ALSA-lib interface), obtaining a shared audio auxiliary stream. The conferencing application performs necessary signal processing on the shared audio auxiliary stream, such as muting, gain adjustment, and equalization. Simultaneously, the encoding module in the conferencing application adaptively selects different encoders based on the music scene and network bandwidth to encode the shared audio auxiliary stream before sending it over the network to remote users. Additionally, the conferencing application transmits the processed shared audio auxiliary stream to speakers for near-end playback, ensuring that users in the nearby meeting room can hear the locally presented shared audio content.

[0125] The setup of the mainstream microphone call audio input path and the audio signal acquisition process are as follows:

[0126] First, the conferencing equipment connects to a microphone or a built-in microphone audio input device to capture audio signals from nearby users during audio and video conferencing. Once the Android system of the conferencing equipment recognizes the microphone audio input device, the conferencing application establishes an independent audio acquisition link with the microphone to capture the microphone's audio signal during the meeting. During the audio and video conference, the conferencing application acquires the microphone's input audio signal by accessing the audio signal acquisition interfaces in the Android system (AudioRecord interface / OpenSL ES interface / AAudio interface / ALSA-lib interface), obtaining the microphone's main audio stream. The conferencing application performs necessary signal processing on the microphone's main audio stream, such as echo cancellation, mute, gain adjustment, and equalization. Simultaneously, the encoding module in the conferencing application adaptively selects different encoders based on the music scene and network bandwidth to encode the microphone's main audio stream before sending it to remote users over the network.

[0127] In addition, during the process of transmitting the microphone audio main stream and shared audio auxiliary stream to remote users over the network, the microphone audio main stream and shared audio auxiliary stream can be further processed according to the meeting scenario, such as whether to perform audio mixing.

[0128] When acquiring shared audio auxiliary streams and microphone audio main streams, if both the audio acquisition links of the shared audio auxiliary stream and the microphone audio main stream support acquisition using the first API interface, then the audio signal acquisition interface of the shared audio auxiliary stream and the audio signal acquisition interface of the microphone audio main stream can be any combination of the AudioRecord interface, the OpenSL ES interface, and the AAudio interface.

[0129] For example, two AudioRecord interfaces can be used to acquire the shared audio auxiliary stream and the microphone audio main stream respectively, as shown in Figure 8. Figure 8 is a schematic diagram of using two AudioRecord interfaces to acquire the shared audio auxiliary stream and the microphone audio main stream respectively, according to an embodiment of this application. In Figure 8, AudioSource.MIC and AudioSource.REMOTE_SUBMIX are specified to acquire the microphone audio main stream and the shared audio auxiliary stream respectively.

[0130] Alternatively, a combination of an AudioRecord interface and an OpenSL ES interface can be used to separately acquire the shared audio auxiliary stream and the microphone audio main stream. As shown in Figure 9, which is a schematic diagram of acquiring the shared audio auxiliary stream and the microphone audio main stream using the AudioRecord interface and the OpenSL ES interface respectively, both require custom physical devices (AudioSource) when creating acquisition object instances. Furthermore, it should be noted that since the AudioRecord interface can support scenarios involving the acquisition and processing of multiple audio signals, why is a combination of the AudioRecord interface and the OpenSL ES interface still needed? This embodiment mainly considers the differences and applicability between the two audio signal acquisition interfaces. In the process of acquiring multiple audio signals, for audio signals that require lower latency and higher performance, the OpenSL ES interface is more advantageous than the AudioRecord interface. Moreover, compared with the AAudio interface, the OpenSL ES interface is more versatile and less dependent on the underlying layers (such as the Audio HAL layer and the audio framework layer). In the case that many manufacturers' Android systems still rely on the native system and have not adapted the audio HAL layer and the underlying hardware drivers below, the OpenSL ES interface is more versatile.

[0131] Alternatively, multiple AAudio interfaces can be used to capture shared audio auxiliary streams and microphone audio mainstreams. For example, as shown in Figure 10, which is a schematic diagram of capturing shared audio auxiliary streams and microphone audio mainstreams using AAudio interfaces according to an embodiment of this application, in Figure 10, while capturing the shared audio auxiliary streams and microphone audio mainstreams, it is also necessary to further capture the echo signal Loopback. The echo signal Loopback is used for echo cancellation of the microphone audio mainstream. By obtaining the echo signal Loopback in the HardWare layer, all downstream audio playback in the entire Android system can be obtained, and the echo cancellation delay can be constructed for algorithm alignment.

[0132] It is understood that shared audio auxiliary streams and microphone audio main streams can also be acquired through combinations of AudioRecord and AAudio interfaces, or combinations of OpenSL ES and AAudio interfaces, etc., which will not be listed one by one in this embodiment.

[0133] Additionally, when the audio acquisition link does not support acquisition using the first AIP interface, the second API interface must be used. For example, assuming the audio acquisition link of the external audio input device is not based on the operating system's standard audio acquisition framework, after granting the application the appropriate permissions, the audio signal from the external audio input device can be acquired through the ALSA-lib interface. For example, as shown in Figure 11, Figure 11 is a schematic diagram of acquiring the microphone's main audio stream and shared secondary audio stream using the AudioRecord / OpenSL ES / AAudio and ALSA-lib interfaces respectively, according to an embodiment of this application. After granting permissions in the Android system, the conferencing application can bypass the hardware abstraction layer and the audio framework layer, directly accessing the audio signal corresponding to the device node of the external audio input device in the device driver layer through the ALSA-lib interface.

[0134] This embodiment provides a general solution covering multiple audio channels for simultaneous acquisition across all scenarios, and offers four typical scenarios. Users can choose different API interfaces as needed based on low latency, high performance, or low cost development, enabling different Android terminal platforms.

[0135] This application embodiment also provides a multi-channel audio signal acquisition and processing device, as shown in Figure 12. Figure 12 is a structural schematic diagram of a multi-channel audio signal acquisition and processing device provided in this application embodiment. The multi-channel audio signal acquisition and processing device provided in this application embodiment is applicable to an application program in the application layer of a conference device. The conference device is used to connect to at least two audio input devices, and the connection links between the conference device and each audio input device are independent of each other. An independent audio acquisition link is established between the application program and each audio input device. The multi-channel audio signal acquisition and processing device includes:

[0136] The information acquisition module 301 is used to acquire the link information corresponding to each audio acquisition link of the application. The link information is used to identify whether the corresponding audio acquisition link is based on the operating system's standard audio acquisition framework settings.

[0137] The first link determination module 302 is used to determine the first audio acquisition link based on the audio acquisition framework set by the operating system standard in the audio acquisition links according to the link information corresponding to each audio acquisition link.

[0138] The first audio acquisition module 303 is used to acquire a first audio signal from the first audio acquisition link through the first API interface. Each first audio acquisition link corresponds to a first API interface. The first API interface is an interface for implementing audio signal acquisition and is suitable for use when the operating system is configured with the default data access permissions.

[0139] The first audio processing module 304 is used to process each first audio signal separately.

[0140] The first API interface is either the AudioRecord interface, the OpenSL ES interface, or the AAudio interface.

[0141] Among them, the first audio acquisition link corresponding to the first API interface has an independent device node, and each device node is independent of each other in the device driver layer and the hardware abstraction layer.

[0142] Specifically, the first audio acquisition module 303 is used to acquire the first audio signal from the audio framework layer of the first audio acquisition link through the first API interface.

[0143] Specifically, the first audio acquisition module 303 is used to acquire the first audio signal from the device driver layer of the first audio acquisition link through the first API interface.

[0144] The multi-channel audio signal acquisition and processing device also includes:

[0145] The candidate link determination module is used to determine candidate audio acquisition links that are not set up based on the audio acquisition framework of the operating system standard, based on the link information corresponding to each audio acquisition link.

[0146] The second link determination module is used to determine the second audio acquisition link from the candidate audio acquisition links based on the permission information of the application corresponding to each candidate audio acquisition link.

[0147] The second audio acquisition module is used to acquire the second audio signal from the second audio acquisition link through the second API interface. Each second audio acquisition link corresponds to a second API interface. The second API interface is the interface for implementing audio signal acquisition and is suitable for the case where the operating system is configured with the corresponding custom data access permissions.

[0148] The second audio processing module is used to process each second audio signal separately.

[0149] The permission information is used to identify whether the application is allowed to access data in the corresponding device driver layer. The second link determination module is specifically used to determine the second audio acquisition link that allows the application to access data in the corresponding device driver layer from the candidate audio acquisition links based on the permission information corresponding to each candidate audio acquisition link.

[0150] The second audio acquisition module is specifically used to acquire the second audio signal from the device driver layer of the second audio acquisition link through the second API interface.

[0151] The multi-channel audio signal acquisition and processing device provided in this application embodiment is included in the conference equipment and can be used to execute the multi-channel audio signal acquisition and processing method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0152] It is worth noting that in the embodiments of the above-mentioned multi-channel audio signal acquisition and processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0153] This embodiment also provides a conference device, as shown in FIG13. FIG13 is a structural schematic diagram of a conference device provided in this application embodiment. The conference device 40 provided in this application embodiment includes a processor 400 and a memory 401.

[0154] Memory 401 is used to store computer program 402 and transfer computer program 402 to processor 400;

[0155] The processor 400 is used to execute the steps in the above embodiment of a multi-channel audio signal acquisition and processing method according to the instructions in the computer program 402.

[0156] For example, computer program 402 may be divided into one or more modules / units, one or more of which are stored in memory 401 and executed by processor 400 to complete this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 402 in conference device 40.

[0157] The conference device 40 may include, but is not limited to, a processor 400 and a memory 401. Those skilled in the art will understand that FIG13 is merely an example of the conference device 40 and does not constitute a limitation on the conference device 40. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the conference device 40 may also include input / output devices, network access devices, buses, etc.

[0158] The processor 400 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0159] The memory 401 can be an internal storage unit of the conferencing equipment 40, such as a hard drive or RAM of the conferencing equipment 40. The memory 401 can also be an external storage device of the conferencing equipment 40, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the conferencing equipment 40. Furthermore, the memory 401 can include both internal and external storage units of the conferencing equipment 40. The memory 401 is used to store computer programs and other programs and data required by the conferencing equipment 40. The memory 401 can also be used to temporarily store data that has been output or will be output.

[0160] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0161] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] This application also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a multi-channel audio signal acquisition and processing method, the method comprising the following steps:

[0166] Obtain the link information corresponding to each audio acquisition link of the application. The link information is used to identify whether the corresponding audio acquisition link is based on the operating system's standard audio acquisition framework settings.

[0167] Based on the link information corresponding to each audio acquisition link, the first audio acquisition link is determined in the audio acquisition link based on the audio acquisition framework set by the operating system standard;

[0168] The first audio signal is acquired from the first audio acquisition link through the first API interface. Each first audio acquisition link corresponds to a first API interface. The first API interface is the interface for acquiring audio signals and is suitable for use when the operating system is configured with the default data access permissions.

[0169] Each first audio signal is processed separately.

[0170] Note that the above are merely preferred embodiments and the technical principles applied in this application. Those skilled in the art will understand that the embodiments of this application are not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the embodiments of this application. Therefore, although the embodiments of this application have been described in detail through the above embodiments, the embodiments of this application are not limited to the above embodiments. More other equivalent embodiments may be included without departing from the concept of the embodiments of this application, and the scope of the embodiments of this application is determined by the scope of the appended claims.

Claims

1. A method for acquiring and processing multi-channel audio signals, wherein, The method is applicable to an application program in the application layer of a conferencing device, wherein the conferencing device is used to connect to at least two audio input devices, and the connection links between the conferencing device and each audio input device are independent of each other. An independent audio acquisition link is established between the application program and each audio input device. The method includes: Obtain the link information corresponding to each audio acquisition link of the application, wherein the link information is used to identify whether the corresponding audio acquisition link is based on the operating system's standard audio acquisition framework settings; Based on the link information corresponding to each of the audio acquisition links, a first audio acquisition link is determined in the audio acquisition links based on the audio acquisition framework set according to the operating system standard; The first audio signal is acquired from the first audio acquisition link through the first API interface. Each first audio acquisition link corresponds to a first API interface. The first API interface is the interface for acquiring audio signals and is suitable for the case where the operating system is configured with the default data access permissions. Each of the first audio signals is processed separately.

2. The multi-channel audio signal acquisition and processing method according to claim 1, wherein, The first API interface is the AudioRecord interface, the OpenSLES interface, or the AAudio interface.

3. The method for acquiring and processing multi-channel audio signals according to claim 2, wherein, The first audio acquisition link corresponding to the first API interface has independent device nodes, and each device node is independent of the other in the device driver layer and the hardware abstraction layer.

4. The method for acquiring and processing multi-channel audio signals according to claim 3, wherein, The acquisition of the first audio signal from the first audio acquisition link via the first API interface includes: The first audio signal is acquired from the audio framework layer of the first audio acquisition link through the first API interface.

5. The method for acquiring and processing multi-channel audio signals according to claim 3, wherein, The acquisition of the first audio signal from the first audio acquisition link via the first API interface includes: The first audio signal is acquired from the device driver layer of the first audio acquisition link through the first API interface.

6. The method for acquiring and processing multi-channel audio signals according to any one of claims 1 to 5, wherein, The method further includes: Based on the link information corresponding to each of the audio acquisition links, candidate audio acquisition links that are not set up based on the audio acquisition framework of the operating system standard are identified in the audio acquisition links. Based on the permission information of the application corresponding to each of the candidate audio acquisition links, a second audio acquisition link is determined from the candidate audio acquisition links; The second audio signal is acquired from the second audio acquisition link through the second API interface. Each second audio acquisition link corresponds to one second API interface. The second API interface is the interface for acquiring the audio signal and is applicable when the operating system is configured with the corresponding custom data access permissions. Each of the second audio signals is processed separately.

7. The method for acquiring and processing multi-channel audio signals according to claim 6, wherein, The permission information is used to identify whether the application is allowed to access data in the corresponding device driver layer. The step of determining the second audio acquisition link from the candidate audio acquisition links based on the permission information corresponding to the application and each candidate audio acquisition link includes: Based on the permission information corresponding to each candidate audio acquisition link of the application, a second audio acquisition link that allows the application to access data in the corresponding device driver layer is determined among the candidate audio acquisition links; The acquisition of the second audio signal from the second audio acquisition link via the second API interface includes: The second audio signal is acquired from the device driver layer of the second audio acquisition link through the second API interface.

8. A multi-channel audio signal acquisition and processing device, wherein, The device is applicable to an application program in the application layer of a conferencing device. The conferencing device is used to connect to at least two audio input devices, and the connection links between the conferencing device and each audio input device are independent. An independent audio acquisition link is established between the application program and each audio input device. The device includes: The information acquisition module is used to acquire the link information corresponding to each audio acquisition link of the application. The link information is used to identify whether the corresponding audio acquisition link is based on the audio acquisition framework settings of the operating system standard. The first link determination module is used to determine, based on the link information corresponding to each of the audio acquisition links, the first audio acquisition link set up according to the audio acquisition framework of the operating system standard in the audio acquisition links; The first audio acquisition module is used to acquire a first audio signal from the first audio acquisition link through a first API interface. Each first audio acquisition link corresponds to a first API interface. The first API interface is an interface for implementing audio signal acquisition and is suitable for use when the operating system is configured with the default data access permissions. The first audio processing module is used to process each of the first audio signals separately.

9. A conference device, wherein, The conference equipment includes a processor and a memory; The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is used to execute a multi-channel audio signal acquisition and processing method as described in any one of claims 1 to 7 according to the instructions in the computer program.

10. A storage medium for storing computer-executable instructions, wherein, The computer-executable instructions, when executed by a computer processor, are used to perform a multi-channel audio signal acquisition and processing method as described in any one of claims 1 to 7.

Images