Audio encoding method and apparatus, audio decoding method and apparatus, device and storage medium

Abstract

The present application provides an audio encoding method and apparatus, an audio decoding method and apparatus, a device and a storage medium. The audio encoding method comprises: performing first feature extraction on an audio signal by means of a backbone network, so as to obtain a first audio feature of the audio signal; performing second feature extraction on the audio signal by means of a secondary network, so as to obtain a second audio feature of the audio signal, the number of convolutional layers comprised in the backbone network being greater than the number of convolutional layers comprised in the secondary network, and feature dimensions of the first audio feature being the same as feature dimensions of the second audio feature; fusing the first audio feature and the second audio feature to obtain a fused audio feature; and performing signal encoding processing on the fused audio feature, so as to obtain an audio code stream of the audio signal.

Description

Audio encoding methods, audio decoding methods, devices, equipment and storage media

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 2025101253655, filed on January 24, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to artificial intelligence technology, and more particularly to an audio encoding method, an audio decoding method, an apparatus, a device, and a storage medium. Background Technology

[0004] Audio encoding and decoding technology is a crucial application in the field of artificial intelligence, and a core technology in communication services, including remote audio and video calls. Simply put, speech coding technology aims to transmit as much speech information as possible with minimal network bandwidth. From the perspective of Shannon's information theory, speech coding is a type of source coding. The goal of source coding is to compress the amount of data we want to transmit as much as possible at the encoding end, removing redundancy, while simultaneously achieving lossless (or near-lossless) reconstruction at the decoding end.

[0005] In related technologies, the audio bitstream encoded at the encoding end is difficult to retain the most authentic information of the input signal, resulting in poor audio encoding quality. Summary of the Invention

[0006] This application provides an audio encoding method, an audio decoding method, an apparatus, a device, and a storage medium that can improve the encoding quality of audio.

[0007] The technical solution of this application embodiment is implemented as follows:

[0008] This application provides an audio encoding method applied to an electronic device, the method comprising:

[0009] The first audio feature of the audio signal is obtained by extracting the first feature of the audio signal through the backbone network;

[0010] The second feature of the audio signal is obtained by extracting the second feature of the audio signal through a secondary network;

[0011] The backbone network has a greater number of convolutional layers than the secondary network, and the feature dimension of the first audio feature is the same as that of the second audio feature.

[0012] The first audio feature and the second audio feature are fused to obtain the fused audio feature;

[0013] The fused audio features are subjected to signal encoding processing to obtain the audio bitstream of the audio signal.

[0014] This application provides an audio decoding method applied to an electronic device, the method comprising:

[0015] Obtain an audio bitstream, wherein the audio bitstream is generated according to the audio encoding method provided in the embodiments of this application;

[0016] The audio bitstream is subjected to signal decoding processing to obtain the audio feature estimation value corresponding to the audio bitstream;

[0017] By reconstructing the network, the estimated audio features are reconstructed to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0018] This application provides an audio encoding device, including:

[0019] The first feature extraction module is configured to extract the first feature from the audio signal through the backbone network to obtain the first audio feature of the audio signal.

[0020] The second feature extraction module is configured to extract a second feature from the audio signal through a secondary network to obtain a second audio feature of the audio signal; wherein the number of convolutional layers in the backbone network is greater than the number of convolutional layers in the secondary network, and the feature dimension of the first audio feature is the same as the feature dimension of the second audio feature.

[0021] The feature fusion module is configured to fuse the first audio feature and the second audio feature to obtain a fused audio feature;

[0022] The signal encoding module is configured to perform signal encoding processing on the fused audio features to obtain the audio bitstream of the audio signal.

[0023] This application provides an audio decoding device, including:

[0024] The acquisition module is configured to acquire an audio bitstream, wherein the audio bitstream is generated according to the audio encoding method provided in the embodiments of this application;

[0025] The decoding module is configured to perform signal decoding processing on the audio bitstream to obtain the audio feature estimation value corresponding to the audio bitstream;

[0026] The reconstruction module is configured to reconstruct the estimated audio features through a reconstruction network to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0027] This application provides a method for processing audio bitstreams, wherein the audio bitstreams are generated according to the audio encoding method provided in this application, or decoded based on the audio decoding method provided in this application.

[0028] This application provides a method for storing a bitstream, comprising generating a bitstream by executing the audio encoding method provided in this application; and storing the bitstream.

[0029] This application provides a method for transmitting a bit stream, characterized in that it involves generating a bit stream by performing an audio encoding method provided in this application; and transmitting the bit stream.

[0030] This application provides an electronic device, the electronic device comprising:

[0031] Memory is used to store executable instructions or computer programs.

[0032] When a processor executes computer-executable instructions or computer programs stored in the memory, it implements the audio encoding method or the audio decoding method provided in the embodiments of this application.

[0033] This application provides a computer-readable storage medium storing a bitstream formed by computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, they implement the audio encoding method or the audio decoding method provided in this application.

[0034] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, they implement the audio encoding method or the audio decoding method provided in this application.

[0035] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the audio encoding method or the audio decoding method provided in this application.

[0036] The embodiments of this application have the following beneficial effects:

[0037] Because the backbone network contains more convolutional layers, it can extract deeper first audio features. Conversely, because the secondary network has relatively fewer convolutional layers compared to the backbone network, it can extract shallower second audio features, making it easier to capture shallow details in the audio signal. By fusing first and second audio features with the same feature dimension, both deep first and shallow second audio features can be utilized simultaneously. This results in fused audio features containing richer information, enhancing the representational power of the fused audio features, and thus providing higher-quality feature input for subsequent signal coding processing, thereby improving the audio coding quality. Attached Figure Description

[0038] Figure 1 is a schematic diagram of spectrum comparison at different code rates provided in the embodiments of this application;

[0039] Figure 2 is a schematic diagram of the architecture of the audio codec system 100 provided in an embodiment of this application;

[0040] Figure 3A is a schematic diagram of the structure of the electronic device 500 provided in an embodiment of this application;

[0041] Figure 3B is a schematic diagram of the structure of the electronic device 600 provided in an embodiment of this application;

[0042] Figure 4A is a schematic diagram of the first flow of the audio encoding method provided in the embodiment of this application;

[0043] Figure 4B is a second flowchart of the audio encoding method provided in the embodiment of this application;

[0044] Figure 5 is a flowchart illustrating the audio decoding method provided in an embodiment of this application;

[0045] Figure 6 is a schematic diagram of a voice communication link provided in an embodiment of this application;

[0046] Figure 7 is a flowchart illustrating the speech coding method provided in an embodiment of this application;

[0047] Figure 8A is a schematic diagram of a common convolutional network provided in an embodiment of this application;

[0048] Figure 8B is a schematic diagram of a dilated convolutional network provided in an embodiment of this application;

[0049] Figure 9 is a schematic diagram of the first analysis network provided in an embodiment of this application;

[0050] Figure 10A is a schematic diagram of the residual block structure used in the coding block provided in the embodiment of this application;

[0051] Figure 10B is a schematic diagram of the residual unit structure provided in an embodiment of this application;

[0052] Figure 11A is a first schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in an embodiment of this application;

[0053] Figure 11B is a second schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application;

[0054] Figure 11C is a third schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application;

[0055] Figure 11D is a fourth schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application;

[0056] Figure 11E is a fifth schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application;

[0057] Figure 11F is a sixth schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application;

[0058] Figure 11G is a seventh schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application;

[0059] Figure 12A is a first schematic diagram of the first reconstructed network provided in an embodiment of this application;

[0060] Figure 12B is a second schematic diagram of the first reconstruction network provided in an embodiment of this application.

[0061] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0064] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0065] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0066] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0067] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0068] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0069] 1) Neural Network (NN): A mathematical model that mimics the behavioral characteristics of animal neural networks to perform distributed parallel information processing. This type of network relies on the complexity of the system to adjust the relationships between a large number of interconnected nodes, thereby achieving the purpose of information processing.

[0070] 2) Deep Learning (DL): This is a new research direction in the field of Machine Learning (ML). Deep learning learns the inherent patterns and representational layers of sample data. The information gained during this learning process greatly helps in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to have analytical and learning capabilities like humans, and to be able to recognize data such as text, images, and sound.

[0071] 3) Quantization: This refers to the process of approximating a signal's continuous values ​​(or a large number of discrete values) into a finite number (or a small number) of discrete values. Quantization includes vector quantization (VQ) and scalar quantization.

[0072] Vector quantization is an effective lossy compression technique based on Shannon's rate distortion theory. The basic principle of vector quantization is to use the index (also called the quantization value) of the codeword in the code table that best matches the input vector for transmission and storage, while decoding only requires a simple table lookup operation. For example, a vector space can be formed by combining several scalar data points, and this vector space can be divided into several small regions. During quantization, the vector falling into a small region is replaced with its corresponding index as the input vector.

[0073] Scalar quantization is the quantization of a scalar, that is, one-dimensional vector quantization. It divides the dynamic range into several small intervals, each with a representative value (i.e., index). When the input signal falls into a certain interval, the input signal is quantized into that representative value.

[0074] 4) Entropy coding: A lossless coding method that does not lose any information during the encoding process according to the entropy principle. It is also a key module in lossy coding and is located at the end of the encoder. Entropy coding includes Shannon coding, Huffman coding, Exponential Golomb coding, and Arithmetic coding.

[0075] Speech coding technology aims to transmit as much speech information as possible with minimal network bandwidth. Speech codecs can achieve compression ratios of over 10 times, meaning that 10MB of speech data can be compressed to only 1MB for transmission, significantly reducing the bandwidth required for information transmission. For example, for a wideband speech signal with a sampling rate of 16000Hz, using a 16-bit sampling depth (the fineness of speech intensity recording during sampling), the uncompressed version has a bitrate (data transmission rate per unit time) of 256kbps. However, with speech coding technology, even lossy coding, the reconstructed speech signal quality can approach that of the uncompressed version within a bitrate range of 10-20kbps, and may even be perceived as indistinguishable. For services requiring higher sampling rates, such as 32000Hz ultrawideband speech, the bitrate range must be at least 30kbps.

[0076] In communication systems, to ensure smooth communication, the industry deploys standard voice codec protocols, such as standards from international and domestic standards organizations like ITU-T, 3GPP, IETF, AVS, and CCSA, including standards like G.711, G.722, AMR series, EVS, and OPUS. Referring to Figure 1, which is a schematic diagram comparing the spectrum at different bit rates provided in this application embodiment, it illustrates the relationship between compression bit rate and quality. Curve 101 is the spectrum curve of the original speech, i.e., the uncompressed signal; curve 102 is the spectrum curve of the OPUS encoder at a bit rate of 20kbps; and curve 103 is the spectrum curve of the OPUS encoding at a bit rate of 6kbps. As shown in Figure 1, as the encoding bit rate increases, the compressed signal becomes closer to the original signal.

[0077] The principle of speech coding is roughly as follows: speech coding can directly encode speech waveform samples one by one; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.

[0078] The above-mentioned encoding principles are all derived from speech signal modeling, i.e., compression methods based on signal processing, which cannot guarantee the encoding quality of audio. To improve encoding efficiency while ensuring speech quality, embodiments of this application provide an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as a terminal device, a server, or a combination of both. The following description uses the example of an electronic device implemented as a terminal device.

[0079] For example, see Figure 2, which is a schematic diagram of the architecture of the audio codec system 100 provided in the embodiment of this application. The audio codec system 100 includes: server 200, network 300, terminal device 400 (i.e., encoding end) and terminal device 500 (i.e., decoding end). The network 300 can be a local area network, a wide area network, or a combination of the two.

[0080] In some embodiments, a client 410 runs on the terminal device 400. The client 410 can be various types of clients, such as instant messaging clients, web conferencing clients, live streaming clients, browsers, etc. In response to an audio acquisition command triggered by the sender (e.g., the initiator of a web conference, the host, the initiator of a voice call, etc.), the client 410 calls the microphone built into the terminal device 400 to acquire audio signals, and performs audio encoding processing on the acquired audio signals to obtain an audio bitstream.

[0081] For example, client 410 calls the audio encoding method provided in the embodiments of this application to encode the acquired audio signal, that is, to extract the first feature of the audio signal through the backbone network to obtain the first audio feature of the audio signal; to extract the second feature of the audio signal through the secondary network to obtain the second audio feature of the audio signal; wherein, the number of convolutional layers included in the backbone network is greater than the number of convolutional layers included in the secondary network, and the feature dimension of the first audio feature is the same as the feature dimension of the second audio feature; to fuse the first audio feature and the second audio feature to obtain the fused audio feature; and to perform signal encoding processing on the fused audio feature to obtain the audio bitstream of the audio signal.

[0082] Client 410 can send the audio stream to server 200 via network 300. After receiving the audio stream, server 200 sends the audio stream to the terminal device 500 associated with the recipient (such as participants in a web conference, audience members, or recipients of a voice call).

[0083] After receiving the audio stream sent by the server 200, the client 510 (such as an instant messaging client, web conferencing client, live streaming client, browser, etc.) can perform audio decoding processing on the audio stream to obtain a synthesized audio signal, thereby realizing audio communication.

[0084] For example, client 510 calls the audio decoding method provided in the embodiments of this application to decode the received audio bitstream, wherein the audio bitstream is generated according to the audio encoding method provided in the embodiments of this application; the audio bitstream is subjected to signal decoding processing to obtain the audio feature estimation value corresponding to the audio bitstream; and the audio feature estimation value is subjected to reconstruction processing through a reconstruction network to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0085] For example, the server 200 shown in Figure 2 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal devices 400 and 500 shown in Figure 2 can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, in-vehicle terminals, etc., but are not limited to these. The terminal devices (e.g., terminal devices 400 and 500) and server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0086] In some embodiments, the terminal device or server 200 can also implement the audio encoding or decoding method provided in this application embodiment by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as a live streaming APP, a web conferencing APP, or an instant messaging APP; it can also be a mini-program, that is, a program that only needs to be downloaded into a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0087] Referring to Figure 3A, which is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of this application, and taking a terminal device as an example, the electronic device 500 shown in Figure 3A includes: at least one processor 520, a memory 550, at least one network interface 530, and a user interface 540. The various components in the electronic device 500 are coupled together through a bus system 550. It is understood that the bus system 550 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 550 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 550 in Figure 3A.

[0088] The processor 520 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0089] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 520.

[0090] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0091] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0092] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0093] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 530, exemplary network interfaces 530 including Bluetooth, WiFi, and Universal Serial Bus (USB).

[0094] In some embodiments, the audio encoding device provided in this application can be implemented in software. FIG3A shows an audio encoding device 555 stored in memory 550, which can be software in the form of programs and plug-ins, including the following software modules: a first feature extraction module 5551, a second feature extraction module 5552, a feature fusion module 5553, and a signal encoding module 5554. The first feature extraction module 5551, the second feature extraction module 5552, the feature fusion module 5553, and the signal encoding module 5554 are used to implement audio encoding functions. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented.

[0095] Referring to Figure 3B, which is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of this application, taking the electronic device 600 as a terminal device as an example, the electronic device 600 shown in Figure 3B includes: at least one processor 620, a memory 650, at least one network interface 630, and a user interface 640. The various components in the electronic device 600 are coupled together through a bus system 650. The memory 650 includes an operating system 651 and a network communication module 652. It should be noted that the function of the structure in Figure 3B is similar to that of the structure in Figure 3A. The audio decoding device provided in this embodiment of the application can be implemented in software. Figure 3B shows an audio decoding device 655 stored in the memory 650, which can be software in the form of programs and plug-ins, including the following software modules: an acquisition module 6551, a decoding module 6552, and a reconstruction module 6553. The acquisition module 6551, the decoding module 6552, and the reconstruction module 6553 are used to implement the audio decoding function. These modules are logically related and can therefore be arbitrarily combined or further split according to the implemented functions.

[0096] As mentioned above, the audio encoding method provided in this application embodiment can be implemented by various types of electronic devices. Referring to Figure 4A, which is a first flowchart of the audio encoding method provided in this application embodiment, the audio encoding function is realized through the audio encoding method. The following description is based on steps 101 to 104 shown in Figure 4A.

[0097] In step 101, the first feature of the audio signal is extracted through the backbone network to obtain the first audio feature of the audio signal.

[0098] It should be noted that, in the embodiments of this application, the backbone network refers to a deep neural network structure with a relatively large number of convolutional layers (compared to the secondary network), used to perform multi-level abstract transformations on audio signals to extract their deep, global acoustic features. Conversely, the secondary network is a shallow neural network structure with fewer convolutional layers (compared to the backbone network), used to extract shallow, detailed features of the audio signal or as a supplementary path to the backbone network; the structural complexity of the secondary network is lower than that of the backbone network.

[0099] For example, audio signals can be obtained by recording in real time using a microphone device, or an audio file can be loaded locally as the audio signal. This application embodiment does not limit the specific method of obtaining the audio signal.

[0100] In some embodiments, the backbone network includes multiple cascaded first neural network layers. Feature extraction is performed on the input of the nth first neural network layer through the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer. Here, n is a positive integer that increases sequentially, 0 < n ≤ N, N is the number of the first neural network layers in the backbone network, and the input of the first first neural network layer is an audio signal. The feature extraction result of the Nth first neural network layer is determined as the first audio feature.

[0101] Here, the first neural network layer refers to the basic functional unit in the backbone network used for layer-by-layer feature extraction, which includes different types such as causal convolutional layers, preprocessing layers (including convolution, pooling, and activation operations), and coding blocks.

[0102] By extracting the first feature from the audio signal through a backbone network constructed from multiple cascaded first neural network layers, the original audio signal can be abstracted and transformed at multiple levels (which can be understood as data compression), so that the vector dimension of the first audio feature is smaller than that of the audio signal, thereby reducing the amount of data and facilitating subsequent encoding and transmission.

[0103] Continuing with the above example, feature extraction is performed on the input of the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer, which can be achieved by one of the following processes executed by the nth first neural network layer of the backbone network: performing convolutional processing on the input of the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer; performing preprocessing on the input of the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer; performing encoding processing on the input of the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer.

[0104] For example, referring to the schematic diagram of the first analysis network (i.e., the backbone network, which includes multiple cascaded first neural network layers) shown in FIG. 9, the multiple cascaded first neural network layers correspond to the causal convolutional layer, preprocessing layer, and encoding block shown in FIG. 9. As shown in FIG. 9, when n = 1, convolutional processing is performed on the input of the first first neural network layer to obtain the feature extraction result of the first first neural network layer. That is, by invoking a causal convolutional layer with 16 channels, the input tensor (i.e., the audio signal x(n)) can be expanded into a causal convolutional feature of 16×320 (i.e., the feature extraction result of the first first neural network layer). When n = 2, preprocessing is performed on the input of the second first neural network layer to obtain the feature extraction result of the second first neural network layer. That is, after expanding the input tensor (i.e., the audio signal) into a causal convolutional feature of 16×320 (i.e., the feature extraction result of the first first neural network layer) by invoking a causal convolutional layer with 16 channels, a convolutional operation is performed on the causal convolutional feature of 16×320, along with pooling processing with a factor of 2, and the activation function can be a parametric rectified linear unit (PReLU) to generate a tensor of 16×160 (i.e., the feature extraction result of the second first neural network layer). When 2 < n < N, encoding processing (implemented through an encoding block) is performed on the input of the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer. When n = N, convolutional processing is performed on the input of the Nth first neural network layer to obtain the first audio feature (i.e., F1(n) in FIG. 9).

[0105] Continuing with the above example, referring to FIG. 4B, encoding processing is performed on the input of the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer, which can be achieved through the following steps 201 to 203. The following is a specific description.

[0106] In step 201, feature residual processing is performed on the input of the nth first neural network layer through the residual block included in the nth first neural network layer to obtain residual features.

[0107] In some embodiments, the residual block includes multiple cascaded residual units; the input of the nth first neural network layer is processed by the residual block included in the nth first neural network layer to obtain residual features, which can be achieved in the following way: the input of the nth first neural network layer is processed by the first residual unit of the multiple cascaded residual units; the residual result output by the first residual unit is output to the subsequent cascaded residual units, and residual processing and residual result output are continued by the subsequent cascaded residual units; the residual result output by the last residual unit is used as the residual feature.

[0108] Here, a residual block is a structural unit in the first neural network layer used to perform feature residual processing. It consists of multiple cascaded residual units and can receive the input signal of the nth first neural network layer. The residuals are processed sequentially by each residual unit and the processing results are transmitted. Finally, the residual result output by the last residual unit is used as the residual feature, providing hierarchically optimized basic data for subsequent feature extraction.

[0109] For example, the backbone network includes four coding blocks, each coding block includes one residual block, and one residual block includes four or five cascaded residual units. As shown in Figure 10A, when the number of multiple cascaded residual units used for feature residual processing is five, the first residual unit performs residual processing on the input of the nth first neural network layer and outputs the residual result of the first residual unit to the second residual unit; the second residual unit performs residual processing on the residual result of the first residual unit and outputs the residual result of the second residual unit to the third residual unit; the third residual unit performs residual processing on the residual result of the second residual unit and outputs the residual result of the third residual unit to the fourth residual unit; the fourth residual unit performs residual processing on the residual result of the third residual unit and outputs the residual result of the fourth residual unit to the fifth residual unit; the fifth residual unit performs residual processing on the residual result of the fourth residual unit to obtain the residual features.

[0110] The embodiments of this application are not limited to the number of coding blocks and residual blocks in the backbone network, and can be any positive integer such as 2, 3, 4, 5, etc. The embodiments of this application are also not limited to the number of residual units in the residual blocks, and can be any positive integer such as 2, 3, 4, 5, 6, etc. The number of residual units in multiple residual blocks can be the same or different. For example, one residual block contains 4 residual units and another residual block contains 5 residual units.

[0111] Continuing from the previous example, the first residual unit of a series of cascaded residual units can be used to perform residual processing on the input of the nth first neural network layer. This can be achieved as follows: The first residual unit of the series of cascaded residual units performs the following processing: it performs convolution processing on the input of the nth first neural network layer to obtain the convolution result of the first residual unit; it adds the convolution result of the first residual unit to the residual features to obtain the residual result output by the first residual unit.

[0112] In deep networks, gradients can gradually diminish during backpropagation, making it difficult to update parameters near the input layer and causing network training to stagnate. Residual units, by introducing skip connections (corresponding to the addition of the convolution result of the first residual unit to the residual features mentioned above), allow gradients to propagate directly through the residual path, avoiding the vanishing gradient problem during multi-layer propagation. This makes deep networks easier to optimize, and their performance does not decrease with increasing layer count.

[0113] Continuing from the previous example, further residual processing and output of residual results through subsequent cascaded residual units can be achieved as follows: The j-th residual unit performs the following processing through multiple cascaded residual units: convolves the residual result output by the (j-1)-th residual unit to obtain the convolution result of the j-th residual unit; adds the convolution result of the j-th residual unit to the residual result output by the (j-1)-th residual unit to obtain the residual result output by the j-th residual unit; outputs the residual result output by the j-th residual unit to the (j+1)-th residual unit; where j is a sequentially increasing positive integer, 1 < j < J, and J is the number of residual units.

[0114] Continuing the example above, each residual unit includes a dilated convolution operator. The input of the nth first neural network layer is convolved through the first residual unit of a cascaded network, which can be achieved as follows: the input of the nth first neural network layer is dilated using the dilated convolution operator included in the first residual unit. That is, the j-th residual unit performs the following processing: the residual result output by the (j-1)-th residual unit is dilated using the dilated convolution operator included in the j-th residual unit, resulting in the dilated convolution result of the j-th residual unit, where j is a sequentially increasing positive integer, 1 < j ≤ J, and J is the number of residual units. It should be noted that each residual unit contains a dilated convolution operator with a specified dilation rate. Using a dilated convolution operator with an asymptotic dilation rate is equivalent to using different receptive fields to extract features of the input at different resolutions, allowing for a more comprehensive analysis of the data. After each residual unit is convolved by the dilated convolution operator with the dilation rate, it is added to the shallow features (i.e., the input of each residual unit) from the jump connection, thereby directly utilizing the shallow feature information, so that the network can make full use of the shallow feature information during the learning process.

[0115] In some embodiments, each residual unit further includes at least one causal convolution operator. After performing dilated convolution processing on the input of the nth first neural network layer using the dilated convolution operator included in the first residual unit, the following processing can be performed: causal convolution processing is applied to the obtained dilated convolution result using the at least one causal convolution operator included in the first residual unit, and the obtained causal convolution result is used as the convolution result of the first residual unit. That is, after performing dilated convolution processing on the residual result output of the (j-1)th residual unit using the dilated convolution operator included in the jth residual unit to obtain the dilated convolution result of the jth residual unit, causal convolution processing is applied to the dilated convolution result of the jth residual unit using the at least one causal convolution operator included in the jth residual unit, and the causal convolution result of the jth residual unit is used as the convolution result of the jth residual unit. It should be noted that each residual unit further includes at least one causal convolution operator, through which local information of the features input to the causal convolution operator is further extracted.

[0116] In some embodiments, grouped convolution can be applied to the convolution operators (including dilated convolution operators and causal convolution operators) of the residual units. Grouped convolution divides the input channels into multiple groups for convolution operations, with the input and output channels only associated within each group. It should be noted that when the input channels are divided into multiple groups, the corresponding output channels are also divided into multiple groups; that is, the number of input channel groups is the same as the number of output channel groups, thus ensuring that after convolution within a group, the input and output channels are associated only within that group. Here, it is assumed that the input channels and output channels of a feature input to a certain convolution operator are 4. If the number of groups is 1, each input channel is associated with all 4 output channels. If the number of groups is 2, the 4 input channels are first divided into two groups, 0-1 and 2-3. Within each group, the input channel is associated with the output channel within that group. For example, input channel 0-1 in the first group is associated with output channel 0-1, and input channel 2-3 in the second group is associated with output channel 2-3. By introducing grouped convolution, we can avoid any input channel from being associated with all output channels, reduce the number of connections, and thus reduce complexity.

[0117] For example, when applying grouped convolution to the dilated convolution operator included in the residual unit, dilated convolution processing of the input of the nth first neural network layer can be achieved as follows: the input channels of the input of the nth first neural network layer are grouped to obtain multiple groups, wherein each group includes the first element corresponding to at least two channels in the input of the nth first neural network layer; dilated convolution processing is performed on the first element within each group. Similarly, when applying grouped convolution to the causal convolution operator included in the residual unit, causal convolution processing of the resulting dilated convolution result can be achieved as follows: the input channels of the dilated convolution result are grouped to obtain multiple groups, wherein each group includes the second element corresponding to at least two channels in the dilated convolution result; causal convolution processing is performed on the second element within each group.

[0118] In step 202, the residual features are processed by convolution through the convolutional layer included in the nth first neural network layer to obtain convolutional features.

[0119] For example, convolutional features are obtained by sliding a convolutional kernel (filter) on the feature map of the residual features and calculating a weighted sum of local regions. Convolutional processing can capture local patterns and structural information in audio signals, thereby extracting higher-level features.

[0120] In step 203, the convolutional features are downsampled through the downsampling layer included in the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer.

[0121] For example, downsampling can be performed using pooling methods such as max pooling and average pooling, or it can be performed using methods such as bilinear interpolation and nearest neighbor interpolation. The embodiments of this application do not limit the specific downsampling method.

[0122] In other embodiments, feature extraction is performed on the input of the nth first neural network layer through the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer. This can be achieved in the following way: the audio signal is convolved through the first first neural network layer to obtain the feature extraction result of the first first neural network layer; the feature extraction result of the first first neural network layer is preprocessed through the second first neural network layer to obtain the feature extraction result of the second first neural network layer; the feature extraction result of the (m-1)th first neural network layer is encoded through the mth first neural network layer to obtain the feature extraction result of the mth first neural network layer, where m is a positive integer that increases sequentially, 2 < m < N; the feature extraction result of the (N-1)th first neural network layer is convolved through the Nth first neural network layer to obtain the feature extraction result of the Nth first neural network layer.

[0123] For example, as shown in Figure 9, after calling the backbone network based on the audio signal, the audio signal is convolved through the first neural network layer in the backbone network to obtain the feature extraction result of the first neural network layer (a 16×320 tensor obtained after causal convolution in Figure 9). The feature extraction result of the first neural network layer is preprocessed through the second neural network layer to obtain the feature extraction result of the second neural network layer (a 16×160 tensor obtained after preprocessing in Figure 9). The feature extraction result of the (m-1)th neural network layer is encoded through the m-th neural network layer to obtain the feature extraction result of the m-th neural network layer (a tensor obtained after encoding the encoded block in Figure 9). The feature extraction result of the (N-1)th neural network layer is convolved through the N-th neural network layer to obtain the feature extraction result of the N-th neural network layer (a 56×1 tensor F1(n) obtained after causal convolution in Figure 9).

[0124] As shown in Figure 9, the backbone network consists of four cascaded encoding blocks with different downsampling factors. Each encoding block contains a residual block (including at least one residual unit), a convolutional layer, and a downsampling layer (i.e., a pooling layer). Each residual block includes five dilated convolution-based residual units (the input and output feature dimensions of the residual units remain unchanged). The convolutional layer doubles the number of input channels, and the activation function can be PReLU to ensure sufficient data volume and avoid data loss. The pooling layer is a pooling operation with a down factor to perform downsampling and achieve data compression. Here, the down factors of the four encoding blocks are set to 2, 4, 4, and 5, respectively. Therefore, the number of output channels of the four encoding blocks are set to 32, 64, 128, and 256, respectively. After processing by the four encoding blocks, the input 16×160 tensor is transformed into tensors of 32×80, 64×20, 128×5, and 256×1, respectively. For example, the residual block in the first encoding block performs residual processing on the preprocessed features (i.e., the 16×160 tensor in Figure 9, corresponding to the feature extraction result of the second first neural network layer mentioned above), and outputs the residual result from the residual block in the first encoding block to the feature encoding block in the first encoding block. After processing by the feature encoding block in the first encoding block (including a convolutional layer and a pooling layer), the encoding result of the feature encoding block in the first encoding block (i.e., a 32×80 tensor) is obtained, and the encoding result of the feature encoding block in the first encoding block (i.e., a 32×80 tensor) is output to the second encoding block; through the first... The residual blocks in the two coding blocks perform residual processing on the encoding result of the feature coding block in the first coding block (i.e., a 32×80 tensor). The residual result of the residual block in the second coding block is then output to the feature coding block in the second coding block. After processing by the feature coding block in the second coding block (including a convolutional layer and a pooling layer), the encoding result of the feature coding block in the second coding block (i.e., a 64×20 tensor) is obtained. The encoding result of the feature coding block in the second coding block is then output to the third coding block. This process is repeated sequentially, and the output of the last coding block is causally convolved and used as the first audio feature. The embodiments of this application are not limited to the number of coding blocks; it can be any positive integer such as 2, 3, 4, or 5.

[0125] The backbone network is constructed using multiple cascaded first neural network layers. Layered feature extraction is achieved by combining differentiated causal convolutional layers, preprocessing layers, and coding blocks. This approach enables preliminary abstraction and dimensionality regularization of audio signals through initial convolution and preprocessing, while leveraging the modular design of coding blocks (including residual blocks, convolutional layers, and downsampling layers) to achieve deep feature transformation. The residual blocks effectively circumvent the gradient vanishing problem in deep networks through skip connections, ensuring optimizability of deep parameters. Combined with dilated convolutions (with progressive dilation), the receptive field is expanded to capture multi-resolution information. Furthermore, causal convolutions enhance local feature extraction. Furthermore, the introduction of grouped convolution can reduce the number of channel associations, thereby lowering computational complexity. Simultaneously, configuring differentiated downsampling factors and channel numbers in the coding blocks can progressively compress the audio data volume (reducing subsequent encoding and transmission costs) while simultaneously increasing the number of channels to retain key feature information and prevent data loss. In addition, the number of layers (such as coding blocks, residual blocks, and residual units) and processing parameters in the backbone network can be flexibly configured to adapt to different audio feature extraction needs in various scenarios, ultimately achieving efficient extraction of highly representative, low-dimensional first-order audio features, balancing feature extraction accuracy, network optimization efficiency, and engineering application flexibility.

[0126] It should be noted that step 101 is similar to step 11 below.

[0127] Referring again to Figure 4A, in step 102, the audio signal is subjected to second feature extraction through a secondary network to obtain the second audio feature of the audio signal.

[0128] It should be noted that the secondary network has fewer convolutional layers than the backbone network. This design facilitates the capture of shallow details in the audio signal, reduces computational resource consumption, and improves processing efficiency.

[0129] In some embodiments, the secondary network includes multiple cascaded second neural network layers; the feature extraction result of the g-th first neural network layer in the backbone network is obtained, wherein the feature extraction result of the g-th first neural network layer is obtained based on the audio signal, 0 < g < N, and g is a positive integer; through the k-th second neural network layer of the secondary network, feature extraction is performed on the input of the k-th second neural network layer to obtain the feature extraction result of the k-th second neural network layer, wherein k is a positive integer that increases sequentially, 0 < k ≤ K, K is the number of the second neural network layers in the secondary network, and the input of the first second neural network layer is the feature extraction result of the g-th first neural network layer; the feature extraction result of the k-th second neural network layer is determined as the second audio feature.

[0130] Continuing from the previous example, to extract features from the input of the kth second neural network layer of the secondary network, the feature extraction result of the kth second neural network layer can be obtained in the following ways: Perform one of the following processes on the kth second neural network layer of the secondary network: perform convolution processing on the input of the kth second neural network layer to obtain the feature extraction result of the kth second neural network layer; or perform encoding processing on the input of the kth second neural network layer to obtain the feature extraction result of the kth second neural network layer.

[0131] Continuing from the previous example, encoding the input of the k-th second neural network layer to obtain the feature extraction result of the k-th second neural network layer can be achieved in the following way: by performing at least one convolutional process on the input of the k-th second neural network layer through at least one convolutional layer, the convolutional features are obtained; by performing pooling processing on the convolutional features through the pooling layer included in the k-th second neural network layer, the feature extraction result of the k-th second neural network layer is obtained.

[0132] In other embodiments, feature extraction is performed on the input of the kth second neural network layer through the kth second neural network to obtain the feature extraction result of the kth second neural network layer. This can be achieved as follows: the feature extraction result of the gth first neural network layer is encoded through the first second neural network layer to obtain the feature extraction result of the first second neural network layer; the feature extraction result of the (x-1)th second neural network layer is encoded through the xth second neural network layer to obtain the feature extraction result of the xth second neural network layer, where x is a positive integer that increases sequentially, 2 < x < K; the feature extraction result of the (k-1)th second neural network layer is convolved through the Kth second neural network layer to obtain the feature extraction result of the Kth second neural network layer.

[0133] Exemplarily, referring to FIG. 11C, FIG. 11C is a third schematic diagram of an optional structure of the first analysis network and the first long-term connection network (i.e., the secondary network, and the secondary network includes multiple cascaded second neural network layers) provided by an embodiment of the present application. When k = 1, through the first second neural network layer, convolution pooling processing is performed on the feature extraction result of the first first neural network layer to obtain the feature extraction result of the first second neural network layer. That is, through the first second neural network layer, a convolution operation is performed on the feature extraction result of the first first neural network layer (the 16×320 tensor obtained through causal convolution in FIG. 11C), and pooling (Pooling) processing with a factor of 2 is performed, and the activation function can be a parametric rectified linear unit (PReLU) to generate a 16×160 tensor (the 16×160 tensor obtained through convolution pooling in FIG. 11C, that is, the feature extraction result of the first second neural network layer). When 2 < k < K, convolution pooling processing is performed on the input of the k-th second neural network layer (the feature extraction result of the (k - 1)-th second neural network layer) to obtain the feature extraction result of the k-th second neural network layer. When k = K, convolution processing is performed on the input of the K-th first neural network layer (the feature extraction result of the (K - 1)-th second neural network layer) to obtain the second audio feature (the 56×1 tensor F LoRa (n) in FIG. 11C).

[0134] In some other embodiments, the secondary network includes multiple cascaded third neural network layers; second feature extraction of the audio signal is performed through the secondary network to obtain the second audio feature of the audio signal, which can be implemented in the following manner: through the q-th third neural network layer of the secondary network, feature extraction is performed on the input of the q-th third neural network layer to obtain the feature extraction result of the q-th third neural network layer, where q is a positive integer that increases sequentially, 0 < q ≤ Q, and Q is the number of third neural network layers in the secondary network, and the input of the first third neural network layer is the audio signal; the feature extraction result of the Q-th third neural network layer is determined as the second audio feature.

[0135] For example, referring to Figure 11A, Figure 11A is a first schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application. As shown in Figure 11A, when the secondary network (i.e., the first long-term connection network, which includes multiple cascaded third neural network layers) is called based on the audio signal, the audio signal is convolved through the first third neural network layer in the secondary network to obtain the feature extraction result of the first third neural network layer (a 16×320 tensor obtained after causal convolution in Figure 11A). The feature extraction result of the first third neural network layer is encoded (convolutional pooling) through the second third neural network layer to obtain the feature extraction result of the second third neural network layer (a 16×160 tensor obtained after convolutional pooling in Figure 11A), and so on. Finally, the feature extraction result of the Q-1th third neural network layer is convolved through the Q-th third neural network layer to obtain the feature extraction result of the Q-th third neural network layer (a 56×1 tensor F obtained after causal convolution in Figure 11A). LoRa (n)).

[0136] The secondary network, employing fewer convolutional layers than the backbone network, significantly reduces computational resource consumption and improves audio feature processing efficiency. It achieves precise adaptation through flexible input modes (directly receiving audio signals or shallow features from the backbone network). This allows it to efficiently capture shallow audio details from intermediate features transmitted by the backbone network, and also directly extract features from the original audio to meet diverse scenario needs. Through cascaded second / third neural network layers performing layered processing such as convolution and encoding (convolutional pooling, etc.), combined with PReLU activation functions and pooling techniques, it ensures the representation quality of the second audio features while optimizing feature dimensionality. The resulting low-dimensional, highly effective features not only complement the first audio features, but the standardized network structure and processing steps also facilitate integration with subsequent processes, balancing the practicality, flexibility, and engineering implementation efficiency of feature extraction.

[0137] It should be noted that step 102 is similar to step 12 below.

[0138] Referring again to Figure 4A, in step 103, the first audio feature and the second audio feature are fused to obtain the fused audio feature.

[0139] It should be noted that the first and second audio features have the same feature dimensions. The fused audio features contain information at different levels, which can provide higher quality feature input for subsequent signal encoding processing, thereby improving the encoding quality of the audio.

[0140] In some embodiments, the backbone network includes a first subnetwork and a second subnetwork. The first subnetwork is used to perform a first feature extraction, and the second subnetwork is used to perform a third feature extraction. The first audio feature and the second audio feature are fused to obtain a fused audio feature, which can be achieved by adding the first audio feature and the second audio feature to obtain a third audio feature; and then using the second subnetwork to extract the third feature to obtain the fused audio feature.

[0141] For example, referring to Figure 11D, the first audio feature (the 128×5 tensor after encoding the coded block in Figure 11D) and the second audio feature (tensor F in Figure 11D) are... LoRa The sum of (n) is used as the third audio feature (the 128×5 tensor after feature fusion in Figure 11D). The third audio feature is extracted through the second sub-network (corresponding to the coding block with Down_factor=5 in Figure 11D and the cascaded causal convolution) to obtain the fused audio feature (tensor F(n) in Figure 11D).

[0142] The specific implementation of the third feature extraction can be found in the explanation of the causal convolution of the coding block above, and will not be repeated here.

[0143] In other embodiments, fusing the first audio feature and the second audio feature to obtain the fused audio feature can also be achieved by: adding the first audio feature and the second audio feature to obtain the third audio feature; and performing a nonlinear mapping on the third audio feature using an activation function to obtain the fused audio feature.

[0144] For example, referring to Figure 11A, the first audio feature (tensor F1(n) in Figure 11A) and the second audio feature (tensor F in Figure 11A) are... LoRa The three audio features are added together to obtain the third audio feature. Then, the third audio feature is nonlinearly mapped by a pre-set activation function (such as tanh) to obtain the fused audio feature (tensor F(n) in Figure 11A).

[0145] In other embodiments, fusing the first audio feature and the second audio feature to obtain the fused audio feature can also be achieved by: concatenating the first audio feature and the second audio feature to obtain the fourth audio feature; integrating the fourth audio feature by channel dimension through a preset dimension compression layer to obtain the fifth audio feature; and performing attention encoding on the fifth audio feature to obtain the fused audio feature.

[0146] For example, since the first and second audio features have the same dimension (e.g., both are 128×5 tensors), the concatenation process can be performed along the channel dimension (e.g., the direction of the number of feature channels), expanding the dimension of the fourth audio feature to 256×5 (i.e., doubling the number of channels while keeping the time step unchanged). The dimension compression layer can be implemented using a 1×1 convolution kernel, compressing the channel dimension of the fourth audio feature from 256 back to 128 through convolution operations, while simultaneously completing the information interaction and integration between different feature channels, resulting in a preliminary fused feature (i.e., the fifth audio feature) with the same dimension as the original feature. Subsequently, an attention mechanism (e.g., a channel attention module) is introduced to calculate the weight coefficient for each channel feature of the fifth audio feature (obtaining global channel information through global average pooling, and then outputting the weight in the 0-1 interval through a fully connected layer and a sigmoid activation function). This weight coefficient is then multiplied element-wise by the fifth audio feature by channel, thereby enhancing key audio information (e.g., the pitch component in the speech signal and the energy peak in the audio emotional features) and suppressing redundant information, ultimately outputting the fused audio feature.

[0147] It should be noted that step 103 is similar to step 13 below.

[0148] Referring again to Figure 4A, in step 104, the fused audio features are processed by signal encoding to obtain the audio bitstream of the audio signal.

[0149] Here, signal encoding processing of fused audio features refers to converting the fused audio features into a format suitable for storage, transmission or further processing. In this process, the fused audio features are encoded into a specific data stream, namely an audio bitstream, which is more suitable for transmission in network environments with limited bandwidth.

[0150] In some embodiments, before performing signal encoding processing on the fused audio features to obtain the audio bitstream of the audio signal, the following processing may also be performed: obtaining a target bitrate mode for the audio signal from multiple bitrate modes; correspondingly, step 104 can be implemented in the following way: performing signal encoding processing on the fused audio features through the target bitrate mode to obtain the audio bitstream of the audio signal.

[0151] Following the example above, the target bitrate mode is used to indicate the use of a target code table for signal encoding processing. The audio bitstream of the audio signal is obtained by encoding the fused audio features using the target bitrate mode, which can be achieved as follows: Quantize the fused audio features using the target code table to obtain quantized values; then entropy encode the quantized values ​​using the target bitrate corresponding to the target code table to obtain the audio bitstream of the audio signal.

[0152] For example, the fused audio features F(n) of an audio signal can be obtained using scalar quantization (each component is quantized individually) and entropy coding. Furthermore, the embodiments of this application do not limit the combination of vector quantization (multiple adjacent components are combined into a vector for joint quantization) and entropy coding techniques. In addition to entropy coding, direct coding can also be used; for example, if a codebook has 8 elements, 3 bits are used for encoding; if a codebook has 16 elements, 4 bits are used for encoding, and so on.

[0153] It should be noted that step 104 is similar to step 14 below.

[0154] Through steps 101 to 104, the backbone network, with multiple cascaded first neural network layers at its core, combines techniques such as residual blocks (including skip connections), dilated convolutions, and grouped convolutions to extract highly representative, low-dimensional first audio features while avoiding the gradient vanishing problem in deep networks and reducing computational complexity. The secondary network, with its streamlined convolutional layer architecture, efficiently captures shallow audio details and outputs second audio features complementary to the first audio features. The feature fusion step integrates the first and second audio features through flexible paths, generating high-quality fused features that combine both deep and shallow information. The encoding step, combined with the target bitrate mode, uses diverse techniques such as quantization and entropy coding to transform the fused features into an audio bitstream suitable for bandwidth-limited environments. The entire process is tightly integrated, with flexible configuration of network parameters and encoding modes, ensuring both the accuracy and efficiency of feature extraction while improving the adaptability and audio quality of the encoded output, thus balancing technological advancement with engineering practicality.

[0155] In other embodiments, after performing signal encoding processing on the fused audio features to obtain the audio bitstream of the audio signal, the following processing can also be performed: determining the frame header based on the target bitrate mode; and generating an audio bitstream encapsulation of the audio signal based on the audio bitstream and the frame header.

[0156] For example, the frame header (such as a 1-bit frame header) can be placed at the beginning of the audio stream encapsulation, and the audio stream can be placed after the frame header. The target bitrate mode can also be placed in other positions within the audio stream encapsulation.

[0157] As mentioned above, the audio decoding method provided in this application embodiment can be implemented by various types of electronic devices. Referring to Figure 5, which is a first flowchart of the audio decoding method provided in this application embodiment, the audio decoding function is realized through the audio decoding method. The following description is based on steps 301 to 303 shown in Figure 5.

[0158] In step 301, an audio bitstream is obtained, wherein the audio bitstream is generated according to the audio encoding method provided in the embodiments of this application.

[0159] For example, an audio stream file can be read from a local file system or network storage, or a real-time audio stream (such as an audio stream in a live streaming broadcast or video conference) can be received over a network. This application does not limit the specific method of obtaining the audio stream.

[0160] In some embodiments, in response to a decoding request for an audio stream encapsulation, an audio stream encapsulation is obtained, and a target bitrate mode is obtained from the frame header included in the audio stream encapsulation. The audio stream included in the audio stream encapsulation is obtained by audio encoding an audio signal using the target bitrate mode and the audio encoding method provided in the embodiments of this application. The target bitrate mode is obtained from multiple bitrate modes.

[0161] Here, the decoding request is used to instruct the audio stream to undergo audio decoding processing.

[0162] In step 302, the audio bitstream is subjected to signal decoding processing to obtain the audio feature estimation value corresponding to the audio bitstream.

[0163] It's important to note that signal decoding is the inverse process of signal encoding. Since the decoding process at the decoding end is the reverse of the encoding process at the encoding end, the values ​​generated during decoding are estimates relative to the values ​​generated during encoding. For example, the data values ​​representing audio features generated during decoding are estimates relative to the audio features in the encoding process. That is, the former is not necessarily equivalent to the original audio feature data values. For instance, there may be subtle differences due to the encoding and decoding processes, and there may be situations where decoding cannot completely reconstruct the original audio feature data values. Therefore, the former is also called an "audio feature estimate."

[0164] In some embodiments, a target bitrate pattern is obtained, wherein the target bitrate pattern is obtained from multiple bitrate patterns; the audio bitstream is subjected to signal decoding processing using the target bitrate pattern to obtain an estimated audio feature value corresponding to the audio bitstream.

[0165] The target bitrate mode is used to indicate the use of a target code table for signal decoding of the audio bitstream. The audio feature estimation value corresponding to the audio bitstream is obtained by performing signal decoding on the audio bitstream using the target bitrate mode. This can be achieved as follows: Entropy decoding is performed on the audio bitstream using the target bitrate corresponding to the target code table to obtain the quantized value of the audio bitstream; then, inverse quantization is performed on the quantized value of the audio bitstream using the target code table to obtain the audio feature estimation value corresponding to the audio bitstream.

[0166] For example, for an audio bitstream, the audio bitstream is first entropy decoded using the bitrate corresponding to the target code table to obtain the quantization value corresponding to the audio bitstream. Then, the audio feature estimation value F′(n) corresponding to the audio bitstream is obtained by looking up the target code table.

[0167] It should be noted that step 302 is similar to step 21 below.

[0168] In step 303, the audio feature estimates are reconstructed using a reconstruction network to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0169] Here, the audio feature estimates are reconstructed by a reconstruction network, with the aim of recovering an audio signal that is similar to the original audio signal from the encoded audio bitstream (i.e., reconstructing the audio signal).

[0170] In some embodiments, the reconstruction network includes multiple cascaded fourth neural network layers. The reconstruction network reconstructs the audio feature estimates to obtain the reconstructed audio signal corresponding to the audio bitstream. This can be achieved as follows: The t-th fourth neural network layer of the reconstruction network is used to reconstruct the features of its input, resulting in the feature reconstruction result of the t-th fourth neural network layer. Here, t is a sequentially increasing positive integer, 0 < t ≤ T, and T is the number of the fourth neural network layers in the reconstruction network. The input of the first fourth neural network layer is the audio feature estimate. The feature reconstruction result of the T-th fourth neural network layer is then determined as the reconstructed audio signal.

[0171] Continuing from the previous example, to reconstruct the features of the t-th fourth neural network layer by reconstructing the input of the t-th fourth neural network layer, the feature reconstruction result of the t-th fourth neural network layer can be obtained in the following ways: Perform one of the following processes on the t-th fourth neural network layer of the reconstructed network: perform convolution processing on the input of the t-th fourth neural network layer to obtain the feature reconstruction result of the t-th fourth neural network layer; perform decoding processing on the input of the t-th fourth neural network layer to obtain the feature reconstruction result of the t-th fourth neural network layer; or perform post-processing on the input of the t-th fourth neural network layer to obtain the feature reconstruction result of the t-th fourth neural network layer.

[0172] Exemplarily, refer to FIG. 12A. FIG. 12A is a first schematic diagram of the first reconstruction network provided by an embodiment of the present application. As shown in FIG. 12A, multiple cascaded fourth neural network layers correspond to the causal convolution, decoding block, and post-processing shown in FIG. 12A. When t = 1, the audio feature estimate is subjected to convolution processing to obtain the feature reconstruction result of the first fourth neural network layer. That is, by invoking the causal convolution layer, the input tensor (i.e., the audio feature estimate) can be expanded into a 256×1 causal convolution feature. When 1 < t < T - 1, the input of the t-th fourth neural network layer (the feature reconstruction result of the previous fourth neural network layer) is decoded to obtain the feature reconstruction result of the t-th fourth neural network layer. When t = T - 1, the input of the (T - 1)-th fourth neural network layer is post-processed to obtain the feature reconstruction result of the (T - 1)-th fourth neural network layer (the 16×320 tensor obtained after post-processing in FIG. 12A). When t = T, the input of the T-th fourth neural network layer (the feature reconstruction result of the (T - 1)-th fourth neural network layer) is subjected to convolution processing to obtain the reconstructed audio signal (the 1×320 tensor x′(n) obtained after causal convolution in FIG. 12A).

[0173] Continuing with the above example, the decoding process of the input of the t-th fourth neural network layer to obtain the feature reconstruction result of the t-th fourth neural network layer can be achieved as follows: The input of the t-th fourth neural network layer is subjected to convolution processing through the convolution layer included in the t-th fourth neural network layer to obtain convolution features; the convolution features are upsampled through the upsampling layer included in the t-th fourth neural network layer to obtain upsampled features; and the upsampled features are subjected to feature residual processing through the residual block included in the t-th fourth neural network layer to obtain the feature reconstruction result of the t-th fourth neural network layer.

[0174] Continuing with the above example, the residual block includes multiple cascaded residual units. The feature residual processing of the upsampled features through the residual block included in the t-th fourth neural network layer to obtain the feature reconstruction result of the t-th fourth neural network layer can be achieved as follows: The input of the t-th fourth neural network layer is subjected to residual processing through the first residual unit of the multiple cascaded residual units; the residual result output by the first residual unit is output to the subsequent cascaded residual units, and the subsequent cascaded residual units continue to perform residual processing and output of the residual result; and the residual result output by the last residual unit is used as the feature reconstruction result of the t-th fourth neural network layer.

[0175] Here, the specific implementation manner of the feature residual processing can refer to the description of step 201 above and will not be elaborated here.

[0176] As shown in Figure 12A, the reconstruction network comprises four cascaded decoding blocks with different upsampling factors. Each decoding block contains a residual block (including at least one residual unit), a convolutional layer, and an upsampling layer. Each residual block includes five dilated convolution-based residual units (the input and output feature dimensions of the residual units remain unchanged). The convolutional layer halves the number of input channels, and the activation function can be PReLU to ensure sufficient data volume and avoid data loss. The upsampling layer is an upsampling operation with an up factor (e.g., using a repeat operation to perform upsampling through repeated padding) to restore the data. Here, the up factors of the four decoding blocks are set to 5, 4, 4, and 2, respectively. Therefore, the number of output channels of the four decoding blocks are set to 128, 64, 32, and 16, respectively. After processing through four decoding blocks, the input 256×1 tensor is converted into tensors of 128×5, 64×20, 32×80, and 16×160, respectively. The embodiments of this application are not limited to the number of decoding blocks; it can be any positive integer such as 2, 3, 4, or 5.

[0177] Through steps 301 to 303, efficient decoding of the reverse process of the adaptive encoding flow is achieved. It supports flexible acquisition of audio bitstreams from multiple channels such as local and network sources, and can accurately call the corresponding code table based on the target bitrate mode to complete entropy decoding and inverse quantization, ensuring the adaptability of the decoding process and the encoding end to improve feature estimation accuracy. The reconstruction network is based on multiple cascaded fourth neural network layers, combined with convolution, decoding (including upsampling), post-processing and residual block techniques. Through differentiated upsampling factors and channel number configuration, features are gradually restored. The application of residual blocks further ensures the optimization effect of deep networks. Finally, it can efficiently reconstruct an audio signal similar to the original signal from the audio bitstream, and the number of decoding blocks and other parameters can be flexibly adjusted, taking into account decoding quality, adaptability and engineering practicality.

[0178] It should be noted that step 303 is similar to step 22 below, and the feature reconstruction at the decoding end is the reverse process of the feature extraction process at the encoding end.

[0179] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0180] The embodiments of this application can be applied to various audio scenarios, such as voice calls and instant messaging. The following explanation uses voice calls as an example:

[0181] In related technologies, the principle of speech coding is roughly as follows: speech coding can directly encode speech waveform samples one by one; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.

[0182] The above encoding principles all originate from speech signal modeling, i.e., compression methods based on signal processing. To improve encoding quality while maintaining speech encoding efficiency compared to signal processing-based compression methods, this application provides a speech encoding method (i.e., an audio encoding method). For an input speech signal, a first feature vector is obtained based on a first analysis network (i.e., a backbone network); in parallel, a first long-time connection network (i.e., a secondary network) is used to obtain a first long-time connection feature vector. The first feature vector and the first long-time connection feature vector have the same dimension and format. Finally, the first feature vector and the first long-time connection feature vector are added together, and an activation function (such as tanh) is used to obtain the final feature vector. The final feature vector is then compressed and encoded. Specifically, code tables with different quantization precipitates can be used for quantization and encoding of the feature vector to achieve multi-rate encoding and decoding effects. The neural network employs a "block-like" operation, which reduces algorithm complexity and improves encoding performance.

[0183] This application embodiment can be applied to the voice communication link shown in Figure 6. Taking a Voice over Internet Protocol (VoIP) conferencing system as an example, the voice encoding and decoding technologies involved in this application embodiment are deployed in the encoding and decoding parts to solve the basic function of voice compression. The encoder is deployed on the uplink client 601, and the decoder is deployed on the downlink client 602. Voice is acquired through the uplink client and pre-processed, enhanced, encoded, etc. The encoded bitstream is transmitted to the downlink client 602 through the network. The downlink client 602 performs decoding, enhancement, etc., so that the decoded voice can be played back on the downlink client 602.

[0184] To ensure forward compatibility (i.e., compatibility between the new encoder and existing encoders), a transcoder needs to be deployed in the system's backend (i.e., the server) to address the interoperability issue between the new and existing encoders. For example, if the sending end (uplink client) is a new NN encoder, and the receiving end (downlink client) is a Public Switched Telephone Network (PSTN) (G.722), the NN decoder needs to be executed in the backend to generate the voice signal. Then, the G.722 encoder needs to be called to generate a specific bitstream to achieve the transcoding function, allowing the receiving end to correctly decode based on this specific bitstream.

[0185] Before detailing the speech coding method provided in the embodiments of this application, we will first introduce dilated convolutional networks.

[0186] Referring to Figures 8A and 8B, Figure 8A is a schematic diagram of a conventional convolutional network (e.g., causal convolution) provided in an embodiment of this application, and Figure 8B is a schematic diagram of a dilated convolutional network provided in an embodiment of this application. Compared to conventional convolutional networks, dilated convolution can increase the receptive field while maintaining the same feature map size, and can also avoid errors caused by upsampling and downsampling. Although the kernel size shown in Figures 8A and 8B is 3×3, the receptive field 801 of the conventional convolution shown in Figure 8A is only 3, while the receptive field 802 of the dilated convolution shown in Figure 8B reaches 5. That is, for a convolutional kernel of size 3×3, the receptive field of the conventional convolution shown in Figure 8A is 3 and the dilation rate (the number of intervals between points in the convolutional kernel) is 1; while the receptive field of the dilated convolution shown in Figure 8B is 5 and the dilation rate is 2.

[0187] The convolutional kernel can also move on a plane similar to Figure 8A or Figure 8B, which involves the concept of stride rate. For example, if the convolutional kernel moves 1 unit at a time, the corresponding stride rate is 1.

[0188] In addition, there is the concept of convolution channel count, which refers to the number of parameters corresponding to the convolution kernels used in convolution analysis. Theoretically, the more channels there are, the more comprehensive the signal analysis and the higher the accuracy; however, the more channels there are, the higher the complexity. For example, a 1×320 tensor can be used with a 24-channel convolution operation, and the output will be a 24×320 tensor.

[0189] It should be noted that the size of the dilated convolution kernel (for example, for speech signals, the kernel size can be set to 1×3), dilation rate, shift rate, and number of channels can be defined according to actual application needs. This application does not impose specific limitations on these aspects.

[0190] Referring to Figure 7, which is a flowchart illustrating the speech coding method provided in an embodiment of this application, the speech coding method provided in an embodiment of this application will be described below with reference to Figure 7:

[0191] The following processing is performed on the encoding end:

[0192] For the input signal x(n) of the nth frame (i.e., the audio signal mentioned above), the first analysis network (i.e., the backbone network mentioned above) is invoked to obtain the first feature vector F1(n) (i.e., the first audio feature mentioned above). Specifically, the dimension of the first feature vector is smaller than the dimension of the input signal (to reduce the amount of data). A specific implementation is that for each frame x(n), a dilated convolutional neural network (Dilated CNN) (encoder part) is invoked to generate a lower-dimensional feature vector F1(n). This application does not limit the neural network structure of the first analysis network, such as an autoencoder, a fully connected (FC) network, a long short-term memory (LSTM) network, a convolutional neural network (CNN) + LSTM, etc. Internally, the neural network employs a "block-based" operation to reduce algorithm complexity.

[0193] For x(n), the first long-term connection network (i.e., the secondary network mentioned above) is called in parallel to obtain the first long-term connection feature vector F. LoRa (n) (i.e., the second audio feature mentioned above), in particular, the feature dimension of the first long-time connection feature vector is consistent with the feature dimension of the first feature vector.

[0194] Using the feature merging module, the first feature vector and the first long-time connection feature vector are fused to obtain the first fused feature vector (i.e., the fused audio feature mentioned above). Specifically, F1(n) and F... LoRa After adding the corresponding dimensions (n), the final first fused feature vector F(n) is output through the activation function.

[0195] The first fused feature vector is quantized using either vector or scalar methods to obtain an index value. This index value is then directly encoded or entropy encoded to obtain the bitstream (i.e., the audio bitstream mentioned above), which is then transmitted to the decoder. Specifically, for F(n), code tables with different quantization precisions are used for quantization and encoding to achieve multi-rate encoding and decoding effects.

[0196] The following processing is performed on the decoding end:

[0197] The bitstream encoded by the decoder obtains the quantized value F′(n) of the first fused feature vector (i.e., the audio feature estimation value mentioned above).

[0198] The first reconstruction network (i.e., the reconstruction network mentioned above) is invoked to generate the reconstructed signal estimate x′(n) (i.e., the reconstructed speech signal, also known as the reconstructed audio signal). Specifically, the structure of the first reconstruction network can be referenced from the structure of the first analysis network, similar to its "inverse" process. In particular, depending on the actual codeword length in the bitstream, code tables with different quantization precisions can be used to generate F′(n) with different precisions to achieve multi-rate encoding and decoding effects.

[0199] The speech coding method provided in the embodiments of this application will be described in detail below.

[0200] In some embodiments, taking a speech signal with a sampling rate of Fs = 16000Hz as an example (it should be noted that the method provided in this application embodiment is also applicable to scenarios with other sampling rates, including but not limited to: 8000Hz, 32000Hz, 48000Hz). Similarly, this application embodiment does not limit the combination of frame lengths under other sampling rates. For example, the frame length is set to 40ms; for Fs = 16000Hz, this is equivalent to each frame still containing 320 sample points.

[0201] The following describes the encoding and decoding ends in detail with reference to the flowchart shown in Figure 7.

[0202] The encoding process is as follows:

[0203] Step 11: Call the first analysis network based on x(n).

[0204] As mentioned earlier, the 320 sample points in the nth frame are denoted as x(n). The purpose of the first analysis network is to generate a lower-dimensional feature vector F1(n) based on the signal x(n) by calling a deep network. In this embodiment, the dimension of x(n) is 320, and the dimension of F1(n) is 56. From the perspective of data volume, it is clear that the first analysis network plays a role in "dimensionality reduction," which is essentially a form of data compression.

[0205] Referring to the network structure diagram of the first analysis network shown in Figure 9, the process of data compression (i.e., feature extraction) by the first analysis network is explained in detail below:

[0206] First, calling a 16-channel causal convolution can generate a 16×320 tensor from the input tensor (i.e., the feature vector, corresponding to the 1×320 feature vector x(n) in the figure).

[0207] Then, the 16×320 tensor is preprocessed. For example, after performing a convolution operation on the 16×320 tensor, a pooling operation with a factor of 2 is performed, and the activation function can be PReLU, to generate a 16×160 tensor.

[0208] Next, four encoding blocks with different downsampling factors are concatenated. Each encoding block contains a residual block, a convolutional layer, and a pooling layer. Each residual block includes five dilated convolution-based residual units (the input and output feature dimensions of the residual units remain unchanged). The convolutional layer doubles the number of input channels, and the activation function can be PReLU to ensure sufficient data volume and avoid data loss. The pooling layer is a pooling operation with a down factor to perform downsampling and achieve data compression. Here, the down factors of the four encoding blocks are set to 2, 4, 4, and 5, respectively. Therefore, the number of output channels of the four encoding blocks are set to 32, 64, 128, and 256, respectively. After processing by the four encoding blocks, the input 16×160 tensor is transformed into tensors of 32×80, 64×20, 128×5, and 256×1, respectively. In this application embodiment, the number of coding blocks is not limited, and can be any positive integer such as 2, 3, 4, 5, etc. Furthermore, this application embodiment does not limit the number of residual units in a coding block, and can be any positive integer such as 2, 3, 4, 5, 6, etc. The number of residual units in multiple coding blocks can be the same or different; for example, one coding block contains 4 residual units, and another coding block contains 5 residual units.

[0209] This section further introduces the Residual Unit. A Residual Unit is a module in a deep neural network that introduces cross-layer connections, making the neural network easier to optimize during training and avoiding problems such as vanishing or exploding gradients. Its core idea is to perform residual learning on the input within the module, that is, to bypass a portion of the layers and directly pass the input information to the output through a direct path, allowing the network to better utilize shallow feature information during the learning process. Figure 10A is a schematic diagram of the residual block structure used in the encoding block of the first analysis network. This residual block includes five dilated convolution-based residual units, each containing a dilated convolution block with a specified dilation rate (e.g., dilation rate = 3). In this embodiment, using five dilated convolution blocks with progressively increasing dilation rates is equivalent to using different receptive fields to extract features of the input at different resolutions, allowing for a more comprehensive analysis of the data. After residual processing through 5 dilated convolutional blocks with specified dilation rates, the residual is added to the input from the jump connection to obtain the output of the residual block, and then the output is output to the convolutional layer connected to the residual block.

[0210] Here, we further describe any residual unit in Figure 10A. As shown in Figure 10B, for any residual unit, it contains at least one dilated convolution with a specified dilation rate (used to expand the receptive field), which can be activated using PReLU. Furthermore, one or more causal convolutions (used to extract local information) can be cascaded, which can also be activated using PReLU. The kernel size of the dilated convolution with the specified dilation rate can be 3, 5, 7, 9, etc., and the kernel size of the causal convolution can be 1, 3, etc. This application and embodiment do not limit the kernel size of the dilated convolution with the specified dilation rate or the causal convolution. Additionally, the causal convolution or dilated convolution in this embodiment can also be implemented by other convolution units with similar or equivalent functions.

[0211] Furthermore, for residual units, a grouped convolution algorithm is introduced to reduce algorithm complexity. Grouped convolution divides the input channels into multiple groups for convolution operations, associating input and output channels only within each group. Here, we assume there are 16 input channels and 32 output channels. If the number of groups is 1, each input channel is associated with all 32 output channels. If the number of groups is 2, the 16 input channels are first divided into two groups: 0-7 and 8-15. Within each group, the input channel is associated with the output channel within that group. For example, input channels 0-7 in the first group are associated with output channels 0-15, and input channels 8-15 in the second group are associated with output channels 16-31. For instance, the 0th output channel is associated only with input channels 0-7 and not with input channels 8-15, and the 25th output channel is associated only with input channels 8-15 and not with input channels 0-7. This comparison shows that introducing grouped convolution can avoid any correlation between input channels and all output channels, reducing the number of connections and thus lowering complexity. However, since a larger number of groups reduces the correlation between input and output channels, it also affects the encoding performance; therefore, a larger number of groups is not always better. In this embodiment, the dilated convolutions contained in the five residual blocks corresponding to the four coding blocks can use different group number configurations, as shown in Table 1.

[0212] Table 1. Configuration of the number of blocks used by residual units in different coding blocks

[0213] Finally, the 256×1 tensor is further processed by causal convolution and Tahn activation to output a 56-dimensional first feature vector. Specifically, according to the calculations of the first analysis network, each value in the 56-dimensional first feature vector is between [-1, 1]. Furthermore, this embodiment does not limit the use of two or more causal convolutions. For example, the 256×1 tensor can be processed by one causal convolution to output a 128×1 tensor; then, the 128×1 tensor can be processed by another causal convolution to output a 56-dimensional feature vector (i.e., F1(n)). This increases computational complexity only slightly while achieving better feature vector extraction.

[0214] Step 12: Call the first long-term connection network based on x(n).

[0215] The purpose of the first long-term connection network is to generate another lower-dimensional feature vector F based on the signal x(n) by calling a deep network. LoRa (n). Where F LoRa The dimensions of F1(n) are the same as those of F1(n).

[0216] The role of the first long-term connection network is to "transmit" the characteristics of the input signal through a first long-term feature vector via another path, and then fuse it with the first feature vector output by the first analysis network through the feature fusion module. Referring to Figures 10A and 10B, the residual units are mostly "single-layer" connections. However, compared to the first analysis network, the first long-term connection network spans at least one neural network layer in the first analysis network, exhibiting "cross-layer" or even multi-layer (i.e., multiple neural network layers in the first analysis network) connections. In other words, compared to the first analysis network, the first long-term connection network connects at least one neural network layer in the first analysis network. From an information transmission perspective, establishing long-term connections between the input signal and the feature vector achieves better feature extraction results.

[0217] In implementation, the topology of the first long-term connection network is similar to that of the first analysis network. However, to control computational complexity, the internal structure of the first long-term connection network is simplified, containing only simple operations such as causal convolution and pooling. Furthermore, the input and output dimensions of each neural network layer in the first long-term connection network are referenced from the corresponding neural network layer in the first analysis network, and are consistent with the input and output dimensions of the corresponding neural network layer in the first analysis network. Therefore, in terms of results, the properties of the first long-term connection feature vector output by the first long-term connection network are consistent with the properties of the first feature vector output by the first analysis network, and the feature dimensions of the first long-term connection feature vector output by the first long-term connection network are consistent with the feature dimensions of the first feature vector output by the first analysis network, making it suitable for subsequent feature fusion operations.

[0218] Referring to Figure 11A, which is a first schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application, the following describes the feature extraction process of the first long-term connection network, referring to the network structure diagram of the first long-term connection network shown in Figure 11A (the first long-term connection network includes multiple cascaded third neural network layers):

[0219] 1) First, a 16-channel causal convolution (i.e., the first third neural network layer) is called, which can generate a 16×320 tensor from the input tensor (i.e. the feature vector, corresponding to the 1×320 feature vector x(n) in the figure).

[0220] 2) Next, the 16×320 tensor is preprocessed through the first convolutional-pooling layer (i.e., the first convolutional-pooling layer in the first long-term connected network shown in Figure 11A, which is also the second third neural network layer). For example, after one convolution operation through the convolutional-pooling layer, a pooling operation with a down factor of 2 is performed, and the activation function can be PReLU, to generate a 16×160 tensor.

[0221] 3) Similarly, the second to fifth convolutional-pooling layers (i.e., the third to sixth third neural network layers) correspond to the four different downsampling factors of the encoding blocks in the first analysis network. For example, the second convolutional-pooling layer corresponds to the first encoding block in the first analysis network, and the third convolutional-pooling layer corresponds to the second encoding block in the first analysis network. The down_factors of the above four convolutional-pooling layers are set to 2, 4, 4, and 5, respectively, and the number of output channels are set to 32, 64, 128, and 256, respectively. Specifically, the second to fifth convolutional-pooling layers use operations similar to those of the first convolutional-pooling layer. Obviously, the calculation of the convolutional-pooling layers is greatly simplified compared to the encoding blocks in the first analysis network.

[0222] 4) Finally, the 256×1 tensor is passed through a causal convolution (i.e., the last third neural network layer) and activated by Tahn, which outputs a 56-dimensional first long-term connection feature vector. Therefore, the first long-term connection feature vector has completely identical attributes and feature dimensions to the aforementioned first feature vector. In this embodiment, the number of causal convolutions used is not limited; two or more causal convolutions can be used. For example, the 256×1 tensor can be passed through a causal convolution to output a 128×1 tensor; then, the 128×1 tensor can be passed through another causal convolution to output a 56-dimensional feature vector (i.e., F). LoRa (n)). In this way, with only a small increase in computational complexity, feature vector extraction can be completed more effectively.

[0223] Step 13: Feature fusion.

[0224] Obtain the first feature vector F1(n) and the first long-term connection feature vector F LoRa After (n), a feature fusion operation is performed. For example, the first feature vector output from the first analysis network is added to the first long-term connection feature vector output from the first long-term connection network along the corresponding dimension, and then tanh activation is performed. In this way, the final feature vector F(n) is output, and each value in the final feature vector F(n) is between [-1, 1].

[0225] Step 14: Quantization encoding.

[0226] For the feature vector F(n), scalar quantization (quantizing each component individually) and entropy coding methods can be used. Furthermore, the embodiments of this application do not limit the combination of vector quantization (combining multiple adjacent components into a vector for joint quantization) and entropy coding techniques. In addition to entropy coding, direct coding can also be used; for example, if a codebook has 8 elements, 3 bits are used for encoding; if a codebook has 16 elements, 4 bits are used for encoding, and so on.

[0227] This section focuses on the implementation description of the feature vector F(n). As described above, the 320-point input x(n) of each frame, after passing through the first analysis network, the first long-term connection network, and feature fusion processing, yields a 56-dimensional feature vector. This application provides a method based on scalar quantization and entropy coding, specifically including: for each dimension, dividing the interval [-1, 1] into 11 equal parts to form a codebook containing 11 elements, and quantizing each dimension value into one of the 11 labels. According to Shannon's entropy theorem, for a codebook containing 11 labels and uniformly distributed, the entropy (average bits) is... Therefore, the average bit rate per frame is 193.76 bits. With a 20ms framing method, there are 50 frames per second; therefore, the average bit rate is 9.69kbps. According to entropy coding theory, probability distribution statistics can be performed on each of the above dimensions to generate 56 code tables. Generally, each dimension is non-uniformly distributed; therefore, the actual bit rate is around 9.69kbps, or less than 9.69kbps.

[0228] In addition, to achieve the goal of multi-rate encoding and decoding of feature vector F(n), the following implementation can be adopted:

[0229] 1) The implementation method described above, which is a code table containing 11 elements in each dimension, is named code table-1, with a reference code rate of 9.69kbps.

[0230] 2) Similarly, for each dimension, the interval [-1, 1] is divided into 9 equal parts, with an entropy of 3.17; the average code rate is 3.17 × 56 × 50 = 8.88 kbps. This is named Code Table-2, with a reference code rate of 8.88 kbps. In this way, there are at least two code rate options, with different code rates corresponding to different qualities; multi-rate coding is achieved.

[0231] 3) Similarly, for each dimension, the interval [-1, 1] is divided into 7 equal parts with an average code rate of 7.86kbps, named code table-3; the interval [-1, 1] is divided into 5 equal parts with an average code rate of 6.50kbps, named code table-4.

[0232] In this way, by configuring the quantization precision, at least four code rate feature coding modes can be achieved for the original 56-dimensional feature vector obtained from the calculation of each frame of data. This application does not limit other multi-code rate construction methods or the number of code tables.

[0233] In some embodiments, for each frame of the bitstream, the bitrate used for encoding (corresponding to the code table used for encoding) is first represented using 2 bits, and then the corresponding index values ​​obtained from 56-dimensional scalar quantization and entropy encoding are written into the bitstream. According to experiments, high-quality compression can be achieved for 16kHz wideband signals in the range of 5-10kbps.

[0234] The decoding process is as follows:

[0235] Step 21, Decode.

[0236] Decoding is the reverse process of encoding. The received bitstream is parsed. The first 2 bits are parsed to obtain the encoding bitrate, and the corresponding code table (a total of 4 code tables) is selected. Then, based on the bitstream, 56 index values ​​are parsed out, and by looking up the quantization table, the estimated value of the feature vector F′(n) can be obtained.

[0237] The following is a detailed description using a configuration with an encoding rate of 9.69 kbps as an example:

[0238] 1) Parse the frame header. The encoding mode bit is 0, which indicates that the encoding rate bit is 9.69kbps, indicating that code table-1 is used.

[0239] 2) The bitstream is parsed in 56 steps using entropy encoding and decoding technology. The quantization value F′(n) of the 56-dimensional feature vector can be obtained through code table-1.

[0240] Step 22: Invoke the first reconstruction network.

[0241] The purpose of the first reconstruction network is to use the quantized value F′(n) of the feature vector to generate the estimated value x′(n) of the signal by calling the deep neural network.

[0242] Referring to the network structure diagram of the first reconstruction network shown in Figure 12A, the network structure of the first reconstruction network is highly similar to that of the first analysis network, for example, causal convolution. The post-processing structure in the first reconstruction network is similar to the pre-processing in the first analysis network. The decoding block structure is symmetrical to the encoding block on the encoding side. The encoding block on the encoding side first undergoes dilated convolution followed by pooling to complete downsampling, while the decoding block on the decoding side first undergoes pooling to complete upsampling, followed by dilated convolution. The specific process is as follows:

[0243] 1) Calling a causal convolution can expand the input tensor from 56×1 to 256×1.

[0244] 2) Next, four decoding blocks with different upsampling factors (Up_factor) are concatenated. Each decoding block contains a convolutional layer, an upsampling layer, and a residual block. The convolutional layer halves the number of input channels; the upsampling layer contains a specific Up_factor to perform upsampling; and the residual block contains five dilated convolution-based residual units. The Up_factors of the four decoding blocks are set to 5, 4, 4, and 2, respectively. Therefore, the number of output channels for the four decoding blocks are set to 128, 64, 32, and 16, respectively. After processing by the four decoding blocks, the 256×1 tensor is converted into tensors of 128×5, 64×20, 32×80, and 16×160, respectively. Note that the number of decoding blocks in this embodiment is not limited and can be any positive integer such as 2, 3, 4, or 5.

[0245] Here, for upsampling modules that contain a specific Up_factor, a repeat operation can be used to complete the upsampling operation by repeatedly filling in the data. This saves complexity.

[0246] Here, the configuration of the five dilated convolution-based residual units at the decoding end is similar to that at the encoding end, including but not limited to the internal structure of the residual units, convolution kernel size, and dilation rate. The number of groups used in the dilated convolution in the decoding block is configured as shown in Table 2. Here, a higher group size of 2 is used in the decoding block to associate more input and output channels, thereby improving the quality of speech reconstruction.

[0247] Table 2. Configuration of the number of groups used by residual units in different decoding blocks

[0248] 3) Post-process the 16×160 tensor output by the concatenated decoder block. For example, perform a factor-2 Repeat operation on the 16×160 tensor output by the concatenated decoder block to complete upsampling, then perform convolution operation and use the PReLU activation function to generate a 16×320 tensor.

[0249] 4) Calling a causal convolution can convert the input 16×320 tensor into a 1×320 tensor, thus completing the signal generation.

[0250] Other details of the first reconstruction network are the reverse process of the first analysis network mentioned above, and will not be repeated here.

[0251] This application embodiment can obtain optimal parameters by jointly training the relevant networks of the encoder and decoder through data collection. Users only need to prepare the data and set the corresponding network structure. After training is completed in the background, the trained model can be put into use.

[0252] Therefore, through the process described in the above embodiments, a low bit-rate speech encoding and decoding scheme based on signal processing and deep learning networks can be completed.

[0253] In other embodiments, see Figure 11B, which is a second schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application. The following is a detailed description of the network structure shown in Figure 11B.

[0254] In the preceding embodiment (corresponding to step 12 above), the principle and implementation of the first long-time connection network were introduced; the characteristics of the so-called long-time connection network were explained by comparing it with a general residual unit. Here, another feasible implementation method is given.

[0255] As shown in Figure 11B, the dimensions of the input and output tensors of the long-term connection network in this embodiment are completely consistent with those of the first analysis network. However, the internal structure of the first long-term connection network has been further simplified (the simplification method can be seen in the differences between Figure 11A and Figure 11B). For example, by modifying the number of convolutional-pooling layers in the first long-term connection network and matching the corresponding down_factor, it is ensured that a 1×320 tensor is mapped to a 56×1 tensor. The structure of the first long-term connection network is more streamlined and has lower computational complexity than that of the first analysis network.

[0256] Therefore, the embodiments of this application do not force various configuration parameters (such as the number of neural network layers, the structure of the neural network layers, the Down_factor parameter, etc.) to be consistent with the first analysis network.

[0257] For details on the specific implementation of the decoding end, please refer to the descriptions of steps 21 to 22 above, which will not be repeated here.

[0258] In other embodiments, see Figure 11C, which is a third schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application. The following is a detailed description of the network structure shown in Figure 11C.

[0259] As shown in Figure 11C, the input to the first long-term connected network can be the output of the first causal convolution in the first analysis network, corresponding to a dimension of 16×320. In this configuration, the first long-term connected network can correspond one-to-one with the structures in the first analysis network except for the first causal convolution. Apart from this, the other implementation methods and details are consistent with step 12 above, and will not be repeated here.

[0260] In principle, the first causal convolution of the first analysis network maps the original input to a 16-channel representation, which is equivalent to describing the original signal using multiple neural network channels. Therefore, it is feasible to obtain another form of the first long-term feature vector by using the 16×320 representation as a starting point for the first long-term connection network analysis. This application does not limit other methods of obtaining the input to the first long-term connection network. For example, the preprocessed 16×160 tensor of the first analysis network in Figure 11C can be used as the input to the first long-term connection network; correspondingly, the first long-term connection network reduces one convolution-pooling operation. The principle is similar and will not be described in detail further.

[0261] Therefore, this application embodiment does not mandate that the input to the first long-term connection network be the original signal input, i.e., the input to the first analysis network. Specific implementation details for the decoding end can be found in steps 21 to 22 above, and will not be repeated here.

[0262] In other embodiments, see Figure 11D, which is a fourth schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application. The following is a detailed description of the network structure shown in Figure 11D.

[0263] As shown in Figure 11D, the input of the first long-term connection network is exactly the same as that of the first analysis network. However, in addition to further simplification of the internal structure of the first long-term connection network, the data structure of the output variables has also changed (see the differences between Figure 11A and Figure 11D for these changes). For example, after one causal convolution and three convolution-pooling layers, the 1×320 tensor is transformed into a 128×5 tensor, which serves as the output of the first long-term connection network. Furthermore, it is fused with the output of the intermediate node (not the final output) of the first analysis network, i.e., the output of the second encoding block (Donw_factor=4), to generate a new 128×5 tensor. Then, the remaining operations of the original first analysis network are performed, ultimately generating a 56×1 tensor.

[0264] Therefore, this embodiment of the application implements feature fusion between the output of the first long-term connection network and the intermediate nodes of the first analysis network. Specific implementation details for the decoding end can be found in steps 21 to 22 above, and will not be repeated here.

[0265] In other embodiments, see Figure 11F, which is a sixth schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application. The following is a detailed description of the network structure shown in Figure 11F.

[0266] As shown in Figure 11F, the input to the first long-term connection network can be the output of the first causal convolution in the first analysis network, corresponding to a dimension of 16×320. The connection method between the first long-term connection network and the first analysis network has been changed (see the difference between Figure 11A and Figure 11F for this change). For example, after the first long-term connection network performs two convolution-pooling layer operations, it transforms the 16×320 tensor into a 64×20 tensor and reuses the intermediate node of the first analysis network (not the final output), namely the third encoding block (Donw_factor=4), to generate a 128×5 tensor. The first long-term connection network uses the 128×5 tensor as the input to the third convolution-pooling layer, transforming the 128×5 tensor into a 256×1 tensor. Subsequently, through the causal convolution layer, the 256×1 tensor is transformed into a 56×1 tensor F. LoRa (n).

[0267] Therefore, this embodiment of the application implements inputting the output of the intermediate node of the first long-term connection network (the 64×20 tensor output of the convolution-pooling layer with Donw_factor=2 in Figure 11F) into the intermediate node of the first analysis network (the third coding block in Figure 11F), and inputting the output of the intermediate node of the first analysis network (the 128×5 tensor output of the third coding block in Figure 11F) into the intermediate node of the first long-term connection network (the convolution-pooling layer with Donw_factor=5 in Figure 11F), thereby obtaining F1(n) and F LoRa F(n) and the fused F(n). The specific implementation of the decoding end can be referred to the description of steps 21 to 22 above, and will not be repeated here.

[0268] In other embodiments, see Figure 11G, which is a seventh schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application. The following is a detailed description of the network structure shown in Figure 11G.

[0269] As shown in Figure 11G, the input to the first long-term connection network can be the output of the first causal convolution in the first analysis network, which corresponds to a dimension of 16×320. The connection method between the first long-term connection network and the first analysis network has been changed (the change here can be seen in the difference between Figure 11A and Figure 11G). For example, after the first long-term connection network performs two convolutional-pooling layer operations, it transforms the 16×320 tensor into a 64×20 tensor. This 64×20 tensor is then fused with the output of the second encoding block (Donw_factor=4), an intermediate node of the first analysis network (not the final output), to generate a new 64×20 tensor. Next, the third encoding block of the original first analysis network is processed, generating a 128×5 tensor. After this, the first analysis network performs the remaining operations based on the fourth encoding block and a causal convolutional layer, resulting in a 56×1 tensor F1(n). The first long-term connection network uses this 128×5 tensor as input to the third convolutional-pooling layer, transforming it into a 256×1 tensor. Finally, a causal convolutional layer transforms this 256×1 tensor into a 56×1 tensor F1(n). LoRa (n), finally, based on F1(n) and F LoRa (n) Perform a second feature fusion to obtain F(n).

[0270] Therefore, this embodiment of the application implements feature fusion of the output of the intermediate node of the first long-term connection network and the intermediate node of the first analysis network, and then performs the remaining operations of the first analysis network and the first long-term connection network based on the fused features to obtain F1(n) and F2(n). LoRa F(n) and the fused F(n). The specific implementation of the decoding end can be referred to the description of steps 21 to 22 above, and will not be repeated here.

[0271] As can be seen from the above, the connection between the first analysis network (backbone network) and the first long-term connection network (secondary network) can exhibit high flexibility and diversity. Through interaction and fusion at different levels, the feature extraction capability of the model is significantly improved. The connection method can be optimized according to specific tasks and data, thereby increasing the flexibility of the model.

[0272] The connection methods between the backbone network and the secondary network include, but are not limited to, the following:

[0273] 1) Independent Extraction and Fusion: The backbone network and secondary networks extract features independently, and then in the final stage of the network, these features (i.e., F1(n) and F1(n)) are combined. LoRa (n)) are integrated to achieve more comprehensive information utilization (as shown in the connection methods in Figure 11A, Figure 11B and Figure 11E).

[0274] 2) Intermediate Node Connections: The outputs of intermediate nodes in the backbone network are used as inputs to some neural network layers in the secondary network, and the outputs of intermediate nodes in the secondary network can also be used as inputs to some neural network layers in the backbone network (as shown in Figure 11F). There is no limit to the number of times the intermediate nodes of the backbone and secondary networks can be used for input and output, meaning that the secondary network can reuse some neural network layers of the backbone network. After the backbone and secondary networks extract features, the extracted features are fused to obtain fused features. This feature fusion can be performed at the final stage of the network (as shown in Figure 11C), or it can be achieved by fusing the outputs of the secondary network with the outputs of the intermediate nodes of the backbone network (as shown in Figure 11D) to enhance the expressive power of the features.

[0275] 3) Layered fusion: The output of the neural network layer of the secondary network is fused with the output of the intermediate node of the backbone network at least once to achieve in-depth feature mining and integration. For example, the output of the neural network layer of the secondary network is fused with the output of the intermediate node of the backbone network once, and a second feature fusion is performed at the last output of the backbone network (as shown in the connection method in Figure 11G); the output of the neural network layer of the secondary network is fused with the output of the intermediate node of the backbone network twice.

[0276] The following describes an implementation example in a narrowband ultra-low bit rate communication scenario.

[0277] In terms of application scenarios, there is a demand for narrowband ultra-low bitrate encoding. For example, walkie-talkies used in emergency scenarios and voice communication in satellite communication scenarios generally use narrowband encoding, i.e., Fs = 8000Hz. Due to channel capacity limitations, even with narrowband encoding, the bit rate is 1kbps or even lower. At this bit rate, traditional signal processing techniques are insufficient in terms of absolute sound quality, resulting in unclear sound. The embodiments of this application, through the end-to-end neural network encoding method described in the foregoing embodiments, can achieve relatively high-quality narrowband voice communication at bit rates of 1kbps or even lower.

[0278] Referring to the foregoing embodiments, an embodiment for a narrowband ultra-low bit rate scenario is given here.

[0279] The encoding side is implemented as follows:

[0280] For Fs = 8000Hz and a frame length of 40ms, the number of sample points per frame remains 320.

[0281] Referring to Figure 11E, which is a fifth schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application, Figure 11E shows the network structure of the first analysis network and the first long-term connection network corresponding to the narrowband ultra-low bit rate scenario. The dimensions of the input quantity, the input and output quantities of the intermediate network structure, and the internal parameters of the network structure are consistent with the aforementioned embodiments (corresponding to steps 11 to 13 above). The difference is that the dimensions of the output first feature vector and the first long-term connection feature vector are reduced to 12×1. Furthermore, through feature fusion, the final feature vector is obtained, with a dimension of 12×1, and the value of each dimension is between [-1, 1].

[0282] For the feature vector F(n), scalar quantization (quantizing each component individually) and entropy coding methods can be used. Furthermore, the embodiments of this application do not limit the combination of vector quantization (combining multiple adjacent components into a vector for joint quantization) and entropy coding techniques. In addition to entropy coding, direct coding can also be used; for example, if a codebook has 4 elements, 2 bits are used for encoding; if a codebook has 8 elements, 3 bits are used for encoding, and so on.

[0283] For narrowband ultra-low bitrate scenarios, the target bitrate is 1kbps or even lower. For each dimension, based on the actual bitrate, one of the following two code tables can be selected for quantization. For example, code table-1 is [-1.0, -0.66, -0.33, 0, 0.25, 0.5, 0.75, 1.0], corresponding to 3 bits; code table-2 is [-1.0, 0, 0.5, 1.0], corresponding to 2 bits. Taking the 3-bit code table-1 as an example, the required bitrate for each frame is 36 bits; with 25 frames per second, the bitrate is 25 × 36 = 0.9kbps. Similarly, taking the 2-bit code table-2 as an example, the bitrate is 0.6kbps. Therefore, for the selected actual bitrate, quantization and direct encoding of the feature vector can be completed directly using either 2-bit or 3-bit encoding. The embodiments of this application do not limit the implementation methods such as entropy encoding and vector quantization, and will not be elaborated here.

[0284] It should be noted that the first analysis network and the first long-term connection network in the narrowband ultra-low bit rate communication scenario can also adopt the different connection methods described in the foregoing embodiments (as shown in Figures 11C, 11D, 11F and 11G), which will not be elaborated here.

[0285] The decoding end is implemented as follows:

[0286] The decoding end for narrowband ultra-low bitrate scenarios is similar to the aforementioned embodiments. First, through decoding, a 12-dimensional feature vector estimate F′(n) can be obtained. As shown in Figure 12B, based on F′(n), the estimated signal x′(n) is generated by calling the first reconstruction network. Except for the input dimension being 12×1, the other methods are similar to the aforementioned embodiments and will not be described again here.

[0287] In summary, the embodiments of this application further improve the encoding technology of end-to-end neural networks by introducing long-term connection analysis networks, and can improve the encoding quality of audio while keeping the complexity of the neural network model acceptable.

[0288] The following continues to describe an exemplary structure of the audio encoding device 555 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG3A, the software module stored in the audio encoding device 555 in the memory 550 may include:

[0289] The first feature extraction module 5551 is configured to extract the first feature of the audio signal through the backbone network to obtain the first audio feature of the audio signal.

[0290] The second feature extraction module 5552 is configured to extract a second feature from the audio signal through a secondary network to obtain a second audio feature of the audio signal; wherein the number of convolutional layers in the backbone network is greater than the number of convolutional layers in the secondary network, and the feature dimension of the first audio feature is the same as the feature dimension of the second audio feature.

[0291] The feature fusion module 5553 is configured to fuse the first audio feature and the second audio feature to obtain a fused audio feature.

[0292] The signal encoding module 5554 is configured to perform signal encoding processing on the fused audio features to obtain the audio bitstream of the audio signal.

[0293] In some embodiments, the backbone network includes multiple cascaded first neural network layers; the first feature extraction module 5551 is further configured to extract features from the input of the nth first neural network layer through the nth first neural network layer of the backbone network to obtain the feature extraction result of the nth first neural network layer, where n is a positive integer that increases sequentially, 0 < n ≤ N, N is the number of layers in the first neural network in the backbone network, and the input of the first first neural network layer is the audio signal; the feature extraction result of the Nth first neural network layer is determined as the first audio feature.

[0294] In some embodiments, the backbone network includes multiple cascaded first neural network layers; the first feature extraction module 5551 is further configured to perform convolution processing on the audio signal through the first first neural network layer to obtain the feature extraction result of the first first neural network layer; perform preprocessing on the feature extraction result of the first first neural network layer through the second first neural network layer to obtain the feature extraction result of the second first neural network layer; perform encoding processing on the feature extraction result of the (m-1)th first neural network layer through the m-th first neural network layer to obtain the feature extraction result of the m-th first neural network layer, where m is a positive integer that increases sequentially, 2 < m < N; and perform convolution processing on the feature extraction result of the (N-1)th first neural network layer through the N-th first neural network layer to obtain the feature extraction result of the N-th first neural network layer.

[0295] In some embodiments, the first feature extraction module 5551 is further configured to perform one of the following processes through the nth first neural network layer of the backbone network: performing convolution processing on the input of the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer; performing preprocessing processing on the input of the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer; or performing encoding processing on the input of the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer.

[0296] In some embodiments, the first feature extraction module 5551 is further configured to perform feature residual processing on the input of the nth first neural network layer through the residual block included in the nth first neural network layer to obtain residual features; perform convolution processing on the residual features through the convolutional layer included in the nth first neural network layer to obtain convolutional features; and perform downsampling on the convolutional features through the downsampling layer included in the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer.

[0297] In some embodiments, the residual block includes a plurality of cascaded residual units; the first feature extraction module 5551 is further configured to perform residual processing on the input of the nth first neural network layer through the first residual unit of the plurality of cascaded residual units; output the residual result output by the first residual unit to the subsequent cascaded residual units, and continue to perform residual processing and output residual results through the subsequent cascaded residual units; and use the residual result output by the last residual unit as the residual feature.

[0298] In some embodiments, the first feature extraction module 5551 is further configured to perform the following processing through the first residual unit of the plurality of cascaded residual units: performing convolution processing on the input of the nth first neural network layer to obtain the convolution result of the first residual unit; adding the convolution result of the first residual unit to the residual feature to obtain the residual result output by the first residual unit.

[0299] In some embodiments, the first feature extraction module 5551 is further configured to perform the following processing through the j-th residual unit of the plurality of cascaded residual units: perform convolution processing on the residual result output by the (j-1)-th residual unit to obtain the convolution result of the j-th residual unit; add the convolution result of the j-th residual unit to the residual result output by the (j-1)-th residual unit to obtain the residual result output by the j-th residual unit; and output the residual result output by the j-th residual unit to the (j+1)-th residual unit; where j is a positive integer that increases sequentially, 1 < j < J, and J is the number of residual units.

[0300] In some embodiments, each residual unit includes a dilated convolution operator; the first feature extraction module 5551 is further configured to perform dilated convolution processing on the input of the nth first neural network layer using the dilated convolution operator included in the first residual unit.

[0301] In some embodiments, each residual unit further includes at least one causal convolution operator; the first feature extraction module 5551 is further configured to perform causal convolution processing on the obtained dilated convolution result through at least one causal convolution operator included in the first residual unit, and use the obtained causal convolution result as the convolution result of the first residual unit.

[0302] In some embodiments, the secondary network includes multiple cascaded second neural network layers; the second feature extraction module 5552 is further configured to obtain the feature extraction result of the g-th first neural network layer in the backbone network, wherein the feature extraction result of the g-th first neural network layer is obtained based on the audio signal, 0 < g < N, and g is a positive integer; through the k-th second neural network layer of the secondary network, feature extraction is performed on the input of the k-th second neural network layer to obtain the feature extraction result of the k-th second neural network layer, wherein k is a positive integer that increases sequentially, 0 < k ≤ K, K is the number of layers of the second neural network in the secondary network, and the input of the first second neural network layer is the feature extraction result of the g-th first neural network layer; and the feature extraction result of the k-th second neural network layer is determined as the second audio feature.

[0303] In some embodiments, the secondary network includes multiple cascaded second neural network layers; the second feature extraction module 5552 is further configured to obtain the feature extraction result of the g-th first neural network layer in the backbone network, wherein the feature extraction result of the g-th first neural network layer is obtained based on the audio signal, 0 < g < N, and g is a positive integer; through the k-th second neural network layer of the secondary network, feature extraction is performed on the input of the k-th second neural network layer to obtain the feature extraction result of the k-th second neural network layer, wherein k is a positive integer that increases sequentially, 0 < k ≤ K, K is the number of layers of the second neural network in the secondary network, and the input of the first second neural network layer is the feature extraction result of the g-th first neural network layer; and the feature extraction result of the k-th second neural network layer is determined as the second audio feature.

[0304] In some embodiments, the second feature extraction module 5552 is further configured to perform one of the following processes through the kth second neural network layer of the secondary network: performing convolution processing on the input of the kth second neural network layer to obtain the feature extraction result of the kth second neural network layer; or performing encoding processing on the input of the kth second neural network layer to obtain the feature extraction result of the kth second neural network layer.

[0305] In some embodiments, the second feature extraction module 5552 is further configured to perform at least one convolutional process on the input of the kth second neural network layer through at least one convolutional layer included in the kth second neural network layer to obtain convolutional features; and to perform pooling processing on the convolutional features through a pooling layer included in the kth second neural network layer to obtain the feature extraction result of the kth second neural network layer.

[0306] In some embodiments, the secondary network includes multiple cascaded third neural network layers. The second feature extraction module 5552 is further configured to extract features from the input of the q-th third neural network layer through the q-th third neural network layer of the secondary network to obtain the feature extraction result of the q-th third neural network layer, where q is a positive integer that increments sequentially, 0 < q ≤ Q, Q is the number of the third neural network layers in the secondary network, and the input of the first third neural network layer is the audio signal; the feature extraction result of the q-th third neural network layer is determined as the second audio feature.

[0307] In some embodiments, the second feature extraction module 5552 is further configured to: encode the feature extraction result of the g-th first neural network layer through the first second neural network layer to obtain the feature extraction result of the first second neural network layer; encode the feature extraction result of the (x-1)-th second neural network layer through the x-th second neural network layer to obtain the feature extraction result of the x-th second neural network layer, where x is a positive integer that increases sequentially, 2 < x < K; and perform convolution processing on the feature extraction result of the (K-1)-th second neural network layer through the K-th second neural network layer to obtain the feature extraction result of the K-th second neural network layer.

[0308] In some embodiments, the backbone network includes a first sub-network and a second sub-network. The first sub-network is used to perform the first feature extraction. The feature fusion module 5553 is further configured to add the first audio feature and the second audio feature to obtain a third audio feature; and to perform third feature extraction on the third audio feature through the second sub-network to obtain the fused audio feature.

[0309] In some embodiments, the signal encoding module 5554 is further configured to obtain a target bitrate mode for the audio signal from multiple bitrate modes; perform signal encoding processing on the fused audio features using the target bitrate mode to obtain the audio bitstream of the audio signal; determine a frame header based on the target bitrate mode; and generate an audio bitstream encapsulation for the audio signal based on the audio bitstream and the frame header.

[0310] In some embodiments, the target bitrate mode is used to indicate signal encoding processing using a target code table; the signal encoding module 5554 is further configured to quantize the fused audio features using the target code table to obtain quantized values ​​of the fused audio features; and to perform entropy encoding processing on the quantized values ​​using the target bitrate corresponding to the target code table to obtain the audio bitstream of the audio signal.

[0311] The following continues to describe an exemplary structure of the audio decoding device 655 provided in this application embodiment as a software module. In some embodiments, as shown in FIG3B, the software module stored in the audio decoding device 655 in the memory 650 may include:

[0312] The acquisition module 6551 is configured to acquire an audio bitstream, wherein the audio bitstream is generated according to the audio encoding method provided in the embodiments of this application.

[0313] The decoding module 6552 is configured to perform signal decoding processing on the audio bitstream to obtain the audio feature estimation value corresponding to the audio bitstream.

[0314] The reconstruction module 6553 is configured to reconstruct the estimated audio features through a reconstruction network to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0315] In some embodiments, the reconstruction network includes multiple cascaded fourth neural network layers; the reconstruction module 6553 is further configured to perform feature reconstruction on the input of the t-th fourth neural network layer through the t-th fourth neural network layer of the reconstruction network to obtain the feature reconstruction result of the t-th fourth neural network layer, where t is a positive integer that increases sequentially, 0 < t ≤ T, T is the number of the fourth neural network layers in the reconstruction network, and the input of the first fourth neural network layer is the audio feature estimate; the feature reconstruction result of the t-th fourth neural network layer is determined as the reconstructed audio signal.

[0316] In some embodiments, the reconstruction module 6553 is further configured to perform one of the following processes through the t-th fourth neural network layer of the reconstruction network: performing convolution processing on the input of the t-th fourth neural network layer to obtain the feature reconstruction result of the t-th fourth neural network layer; performing decoding processing on the input of the t-th fourth neural network layer to obtain the feature reconstruction result of the t-th fourth neural network layer; and performing post-processing processing on the input of the t-th fourth neural network layer to obtain the feature reconstruction result of the t-th fourth neural network layer.

[0317] In some embodiments, the reconstruction module 6553 is further configured to perform convolution processing on the input of the t-th fourth neural network layer through the convolutional layer included in the t-th fourth neural network layer to obtain convolutional features; perform upsampling on the convolutional features through the upsampling layer included in the t-th fourth neural network layer to obtain upsampled features; and perform feature residual processing on the upsampled features through the residual block included in the t-th fourth neural network layer to obtain the feature reconstruction result of the t-th fourth neural network layer.

[0318] In some embodiments, the reconstruction module 6553 is further configured to acquire a target bitrate mode, wherein the target bitrate mode is acquired from multiple bitrate modes; and to perform signal decoding processing on the audio bitstream using the target bitrate mode to obtain an audio feature estimate corresponding to the audio bitstream.

[0319] In some embodiments, the target bitrate mode is used to indicate the use of a target code table for signal decoding of the audio bitstream; the reconstruction module 6553 is further configured to perform entropy decoding on the audio bitstream using the target bitrate corresponding to the target code table to obtain the quantization value corresponding to the audio bitstream; and to perform inverse quantization on the quantization value corresponding to the audio bitstream using the target code table to obtain the audio feature estimation value corresponding to the audio bitstream.

[0320] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the audio encoding or audio decoding method described in this application.

[0321] This application provides a computer-readable storage medium storing a bitstream formed by computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the audio encoding method or audio decoding method provided in this application is implemented, for example, the audio encoding method shown in FIG4A or the audio decoding method shown in FIG5.

[0322] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, they implement the audio encoding method or the audio decoding method provided in this application, such as the audio encoding method shown in FIG4A or the audio decoding method shown in FIG5.

[0323] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0324] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0325] Some implementations may involve systems, methods, and / or computer-readable media at any possible level of integration technical detail. The computer-readable medium may include a computer-readable non-transitory storage medium (or multiple media) having computer-readable program instructions on it for causing a processor to perform operations, and may also include storage of a bitstream generated according to the audio encoding method described above. When executed by a processor, the computer program / instructions may implement the steps of the audio encoding method to generate the bitstream, or implement the steps of the audio decoding method to decode the bitstream.

[0326] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0327] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0328] In summary, through the embodiments of this application, since the backbone network contains more convolutional layers, it can extract deeper first audio features. Furthermore, since the secondary network has relatively fewer convolutional layers compared to the backbone network, it can extract shallower second audio features, facilitating the capture of shallow details in the audio signal. By fusing first and second audio features with the same feature dimension, both deep first and shallow second audio features can be utilized simultaneously, resulting in fused audio features containing richer information and enhancing their representational capabilities. This provides higher-quality feature input for subsequent signal encoding processing, thereby improving the audio encoding quality.

[0329] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An audio encoding method applied to an electronic device, the method comprising: The first audio feature of the audio signal is obtained by extracting the first feature of the audio signal through the backbone network; The second feature of the audio signal is obtained by extracting the second feature of the audio signal through a secondary network; The backbone network has a greater number of convolutional layers than the secondary network, and the feature dimension of the first audio feature is the same as that of the second audio feature. The first audio feature and the second audio feature are fused to obtain the fused audio feature; The fused audio features are subjected to signal encoding processing to obtain the audio bitstream of the audio signal.

2. The method according to claim 1, wherein, The backbone network includes multiple cascaded first neural network layers; The step of extracting the first feature of the audio signal through the backbone network to obtain the first audio feature of the audio signal includes: By using the nth first neural network layer of the backbone network, feature extraction is performed on the input of the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer, where n is a positive integer that increases sequentially, 0 < n ≤ N, N is the number of the first neural network layers in the backbone network, and the input of the first first neural network layer is the audio signal; The feature extraction result of the Nth first neural network layer is determined as the first audio feature.

3. The method according to any one of claims 1 to 2, wherein, The step of extracting features from the input of the nth first neural network layer through the backbone network to obtain the feature extraction result of the nth first neural network layer includes: One of the following processes is performed through the nth first neural network layer of the backbone network: The input of the nth first neural network layer is convolved to obtain the feature extraction result of the nth first neural network layer; The input of the nth first neural network layer is preprocessed to obtain the feature extraction result of the nth first neural network layer; The input of the nth first neural network layer is encoded to obtain the feature extraction result of the nth first neural network layer.

4. The method according to any one of claims 1 to 2, wherein, The step of extracting features from the input of the nth first neural network layer through the backbone network to obtain the feature extraction result of the nth first neural network layer includes: The audio signal is convolved through the first neural network layer to obtain the feature extraction result of the first neural network layer. The feature extraction results of the first neural network layer are preprocessed through the second first neural network layer to obtain the feature extraction results of the second first neural network layer. The feature extraction result of the (m-1)th first neural network layer is encoded through the m-th first neural network layer to obtain the feature extraction result of the m-th first neural network layer, where m is a positive integer that increases sequentially, and 2 < m < N; The feature extraction results of the (N-1)th first neural network layer are obtained by convolution processing through the Nth first neural network layer.

5. The method according to any one of claims 1 to 3, wherein, The process of encoding the input of the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer includes: By using the residual blocks included in the nth first neural network layer, feature residual processing is performed on the input of the nth first neural network layer to obtain residual features; The residual features are processed by convolutional layers included in the nth first neural network layer to obtain convolutional features; The convolutional features are downsampled using the downsampling layer included in the nth first neural network layer to obtain the feature extraction result of the nth first neural network layer.

6. The method according to claim 5, wherein, The residual block comprises multiple cascaded residual units; The step of performing feature residual processing on the input of the nth first neural network layer through the residual block included in the nth first neural network layer to obtain residual features includes: The input of the nth first neural network layer is subjected to residual processing through the first residual unit of the plurality of cascaded residual units; The residual result output by the first residual unit is output to the subsequent cascaded residual units, and the residual processing and residual result output are continued through the subsequent cascaded residual units; The residual result output by the last residual unit is used as the residual feature.

7. The method according to claim 6, wherein, The first residual unit, which passes through the plurality of cascaded residual units, performs residual processing on the input of the nth first neural network layer, including: The following processing is performed through the first residual unit of the plurality of cascaded residual units: The input of the nth first neural network layer is convolved to obtain the convolution result of the first residual unit; The convolution result of the first residual unit is added to the residual feature to obtain the residual result output by the first residual unit.

8. The method according to claim 7, wherein, Each of the residual units includes a dilated convolution operator; The convolution processing of the input to the nth first neural network layer includes: The input of the nth first neural network layer is subjected to dilated convolution processing using the dilated convolution operator included in the first residual unit.

9. The method according to claim 8, wherein, Each residual unit further includes at least one causal convolution operator; After performing dilated convolution processing on the input of the nth first neural network layer using the dilated convolution operator included in the first residual unit, the method further includes: The dilated convolution result is processed by causal convolution using at least one causal convolution operator included in the first residual unit, and the resulting causal convolution result is used as the convolution result of the first residual unit.

10. The method according to any one of claims 6 to 9, wherein, The process of continuing residual processing and outputting residual results through the subsequently cascaded residual units includes: The following processing is performed through the j-th residual unit of the plurality of cascaded residual units: The residual result output by the (j-1)th residual unit is convolved to obtain the convolution result of the j-th residual unit; The convolution result of the j-th residual unit is added to the residual result output by the (j-1)-th residual unit to obtain the residual result output by the j-th residual unit. The residual result output by the j-th residual unit is output to the (j+1)-th residual unit; Where j is a positive integer that increases sequentially, 1 < j < J, and J is the number of residual units.

11. The method according to any one of claims 1 to 10, wherein, The secondary network includes multiple cascaded second neural network layers; The step of extracting a second feature from the audio signal through a secondary network to obtain the second audio feature of the audio signal includes: Obtain the feature extraction result of the g-th first neural network layer in the backbone network, wherein the feature extraction result of the g-th first neural network layer is obtained based on the audio signal, 0 < g < N, and g is a positive integer; By using the kth second neural network layer of the secondary network, feature extraction is performed on the input of the kth second neural network layer to obtain the feature extraction result of the kth second neural network layer, where k is a positive integer that increases sequentially, 0 < k ≤ K, K is the number of the second neural network layers in the secondary network, and the input of the first second neural network layer is the feature extraction result of the gth first neural network layer; The feature extraction result of the Kth second neural network layer is determined as the second audio feature.

12. The method according to any one of claims 1 to 11, wherein, The step of extracting features from the input of the kth second neural network layer through the subnetwork to obtain the feature extraction result of the kth second neural network layer includes: One of the following processes is performed through the k-th second neural network layer of the secondary network: The input of the kth second neural network layer is convolved to obtain the feature extraction result of the kth second neural network layer; The input of the kth second neural network layer is encoded to obtain the feature extraction result of the kth second neural network layer.

13. The method according to any one of claims 1 to 11, wherein, The step of extracting features from the input of the kth second neural network layer through the subnetwork to obtain the feature extraction result of the kth second neural network layer includes: The feature extraction result of the g-th first neural network layer is encoded through the first second neural network layer to obtain the feature extraction result of the first second neural network layer. The feature extraction result of the (x-1)th second neural network layer is encoded through the xth second neural network layer to obtain the feature extraction result of the xth second neural network layer, where x is a positive integer that increases sequentially, and 2 < x < K; The feature extraction results of the (K-1)th second neural network layer are convolved through the Kth second neural network layer to obtain the feature extraction results of the Kth second neural network layer.

14. The method according to any one of claims 1 to 12, wherein, The process of encoding the input of the k-th second neural network layer to obtain the feature extraction result of the k-th second neural network layer includes: By using at least one convolutional layer included in the k-th second neural network layer, the input of the k-th second neural network layer is subjected to at least one convolutional process to obtain convolutional features; The convolutional features are pooled using the pooling layer included in the k-th second neural network layer to obtain the feature extraction result of the k-th second neural network layer.

15. The method according to any one of claims 1 to 10, wherein, The secondary network includes multiple cascaded third neural network layers; The step of extracting a second feature from the audio signal through a secondary network to obtain the second audio feature of the audio signal includes: By using the qth third neural network layer of the secondary network, feature extraction is performed on the input of the qth third neural network layer to obtain the feature extraction result of the qth third neural network layer, where q is a positive integer that increases sequentially, 0 < q ≤ Q, Q is the number of the third neural network layer in the secondary network, and the input of the first third neural network layer is the audio signal; The feature extraction result of the Qth third neural network layer is determined as the second audio feature.

16. The method according to any one of claims 1 to 15, wherein, The backbone network includes a first sub-network and a second sub-network, wherein the first sub-network is used to perform the first feature extraction. The process of fusing the first audio feature and the second audio feature to obtain the fused audio feature includes: The first audio feature and the second audio feature are added together to obtain the third audio feature; The third audio feature is extracted by the second sub-network to obtain the fused audio feature.

17. The method according to any one of claims 1 to 16, wherein, Before performing signal encoding processing on the fused audio features to obtain the audio bitstream of the audio signal, the method further includes: Obtain the target bitrate mode for the audio signal from multiple bitrate modes; The step of performing signal encoding processing on the fused audio features to obtain the audio bitstream of the audio signal includes: The fused audio features are processed by signal encoding using the target bitrate mode to obtain the audio bitstream of the audio signal; After performing signal encoding processing on the fused audio features to obtain the audio bitstream of the audio signal, the method further includes: Based on the target bitrate pattern, determine the frame header; Based on the audio bitstream and the frame header, an audio bitstream encapsulation of the audio signal is generated.

18. The method according to claim 17, wherein, The target code rate mode is used to indicate that a target code table is used for signal encoding processing; The step of performing signal encoding processing on the fused audio features using the target bitrate mode to obtain the audio bitstream of the audio signal includes: The fused audio features are quantized using the target code table to obtain quantized values ​​of the fused audio features. The quantization value is entropy encoded using the target bitrate corresponding to the target code table to obtain the audio bitstream of the audio signal.

19. An audio decoding method applied to an electronic device, the method comprising: Obtain an audio bitstream, wherein the audio bitstream is generated by the audio encoding method according to any one of claims 1 to 18; The audio bitstream is subjected to signal decoding processing to obtain the audio feature estimation value corresponding to the audio bitstream; By reconstructing the network, the estimated audio features are reconstructed to obtain the reconstructed audio signal corresponding to the audio bitstream.

20. The method according to claim 19, wherein, The method further includes: Obtain a target bitrate pattern, wherein the target bitrate pattern is obtained from multiple bitrate patterns; The step of performing signal decoding processing on the audio bitstream to obtain the estimated audio feature values ​​corresponding to the audio bitstream includes: The audio stream is decoded using the target bitrate mode to obtain the estimated audio features corresponding to the audio stream.

21. The method according to any one of claims 19 to 20, wherein, The target bitrate mode is used to indicate the use of a target code table for signal decoding processing of the audio bitstream; The step of performing signal decoding processing on the audio bitstream using the target bitrate mode to obtain the estimated audio feature value corresponding to the audio bitstream includes: The audio bitstream is entropy decoded using the target bitrate corresponding to the target code table to obtain the quantization value corresponding to the audio bitstream. By using the target code table, the quantization value corresponding to the audio bitstream is inversely quantized to obtain the audio feature estimation value corresponding to the audio bitstream.

22. The method according to any one of claims 19 to 21, wherein, The reconstructed network includes multiple cascaded fourth neural network layers; The step of reconstructing the estimated audio features using a reconstruction network to obtain the reconstructed audio signal corresponding to the audio bitstream includes: By using the t-th fourth neural network layer of the reconstruction network, feature reconstruction is performed on the input of the t-th fourth neural network layer to obtain the feature reconstruction result of the t-th fourth neural network layer, where t is a positive integer that increases sequentially, 0 < t ≤ T, T is the number of the fourth neural network layer in the reconstruction network, and the input of the first fourth neural network layer is the audio feature estimate. The feature reconstruction result of the Tth fourth neural network layer is determined as the reconstructed audio signal.

23. The method according to any one of claims 19 to 22, wherein, The step of performing feature reconstruction on the input of the t-th fourth neural network layer through the reconstruction network to obtain the feature reconstruction result of the t-th fourth neural network layer includes: One of the following processes is performed through the t-th fourth neural network layer of the reconstructed network: The input of the t-th fourth neural network layer is convolved to obtain the feature reconstruction result of the t-th fourth neural network layer; The input of the t-th fourth neural network layer is decoded to obtain the feature reconstruction result of the t-th fourth neural network layer; The input of the t-th fourth neural network layer is post-processed to obtain the feature reconstruction result of the t-th fourth neural network layer.

24. The method according to any one of claims 19 to 23, wherein, The process of decoding the input of the t-th fourth neural network layer to obtain the feature reconstruction result of the t-th fourth neural network layer includes: By using the convolutional layers included in the t-th fourth neural network layer, the input of the t-th fourth neural network layer is processed by convolution to obtain convolutional features; The convolutional features are upsampled through the upsampling layer included in the t-th fourth neural network layer to obtain upsampled features; By using the residual blocks included in the t-th fourth neural network layer, feature residual processing is performed on the upsampled features to obtain the feature reconstruction result of the t-th fourth neural network layer.

25. A method for processing an audio bitstream, wherein the audio bitstream is generated by the audio encoding method according to any one of claims 1 to 18, or decoded based on the audio decoding method according to any one of claims 19 to 24.

26. A method for storing a bit stream, characterized in that, The audio encoding method according to any one of claims 1 to 18 is used to generate a bitstream; and the bitstream is stored.

27. A method for transmitting a bit stream, characterized in that, The audio encoding method according to any one of claims 1 to 18 is used to generate a bitstream; and the bitstream is transmitted.

28. An audio encoding apparatus, comprising: The first feature extraction module is configured to extract the first feature from the audio signal through the backbone network to obtain the first audio feature of the audio signal. The second feature extraction module is configured to extract a second feature from the audio signal through a secondary network to obtain a second audio feature of the audio signal; wherein the number of convolutional layers in the backbone network is greater than the number of convolutional layers in the secondary network, and the feature dimension of the first audio feature is the same as the feature dimension of the second audio feature. The feature fusion module is configured to fuse the first audio feature and the second audio feature to obtain a fused audio feature; The signal encoding module is configured to perform signal encoding processing on the fused audio features to obtain the audio bitstream of the audio signal.

29. An audio decoding device, comprising: The acquisition module is configured to acquire an audio bitstream, wherein the audio bitstream is generated by the audio encoding method according to any one of claims 1 to 18; The decoding module is configured to perform signal decoding processing on the audio bitstream to obtain the audio feature estimation value corresponding to the audio bitstream; The reconstruction module is configured to reconstruct the estimated audio features through a reconstruction network to obtain the reconstructed audio signal corresponding to the audio bitstream.

30. An electronic device comprising: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the audio encoding method according to any one of claims 1 to 18, or the audio decoding method according to any one of claims 19 to 24.

31. A computer-readable storage medium storing a bitstream formed by computer-executable instructions or a computer program, wherein when the computer-executable instructions or the computer program are executed by a processor, they implement the audio encoding method according to any one of claims 1 to 18, or the audio decoding method according to any one of claims 19 to 24.

32. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein when executed by a processor, the computer-executable instructions or the computer program implement the audio encoding method according to any one of claims 1 to 18, or the audio decoding method according to any one of claims 19 to 24.

33. A computer program product comprising computer-executable instructions, which, when executed by a processor, implement the audio encoding method according to any one of claims 1 to 18, or the audio decoding method according to any one of claims 19 to 24.

Images