Audio decoding method, audio encoding method, apparatus, electronic device, computer-readable storage medium, and computer program product

Abstract

An audio decoding method, an audio encoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The audio decoding method comprises: performing signal decoding on an audio bitstream to obtain an audio feature estimation value corresponding to the audio bitstream; performing first reconstruction on the audio feature estimation value by means of a backbone network to obtain a first reconstructed signal corresponding to the audio bitstream; performing second reconstruction on the audio feature estimation value by means of a secondary network to obtain a second reconstructed signal corresponding to the audio bitstream, wherein the number of convolution layers comprised in the backbone network is greater than the number of convolution layers comprised in the secondary network, and the dimension of the first reconstructed signal is the same as the dimension of the second reconstructed signal; and fusing the first reconstructed signal and the second reconstructed signal to obtain a reconstructed audio signal corresponding to the audio bitstream.

Description

Audio decoding methods, audio encoding methods, devices, electronic devices, computer-readable storage media, and computer program products

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 2025101217199, filed on January 24, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to artificial intelligence technology, and more particularly to an audio decoding method, an audio encoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0004] Audio encoding and decoding technology is a crucial application in the field of artificial intelligence, and a core technology in communication services, including remote audio and video calls. Simply put, speech coding technology aims to transmit as much speech information as possible with minimal network bandwidth. From the perspective of Shannon's information theory, speech coding is a type of source coding. The goal of source coding is to compress the amount of data we want to transmit as much as possible at the encoding end, removing redundancy, while simultaneously achieving lossless (or near-lossless) reconstruction at the decoding end.

[0005] In related technologies, the signal reconstructed at the decoding end is difficult to recover the most authentic information of the input signal, resulting in poor audio decoding quality. Summary of the Invention

[0006] This application provides an audio decoding method, an audio encoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the decoding quality of audio.

[0007] The technical solution of this application embodiment is implemented as follows:

[0008] This application provides an audio decoding method applied to an electronic device, the method comprising:

[0009] The audio bitstream is decoded to obtain the estimated audio features corresponding to the audio bitstream;

[0010] The first reconstruction of the audio feature estimate is performed through the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream.

[0011] The estimated audio features are reconstructed using a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream.

[0012] Wherein, the number of convolutional layers in the backbone network is greater than the number of convolutional layers in the secondary network, and the dimension of the first reconstructed signal is the same as the dimension of the second reconstructed signal;

[0013] By fusing the first reconstructed signal and the second reconstructed signal, the reconstructed audio signal corresponding to the audio bitstream is obtained.

[0014] This application provides an audio encoding method applied to an electronic device, the method comprising:

[0015] Feature extraction is performed on the audio signal to obtain the audio features of the audio signal;

[0016] The audio features are encoded to obtain an audio bitstream of the audio signal, wherein the audio bitstream is used for decoding based on the audio decoding method provided in the embodiments of this application.

[0017] This application provides an audio encoding method, wherein the audio bitstream is decoded based on the audio decoding method provided in this application, or generated according to the audio encoding method provided in this application.

[0018] This application provides an audio decoding device, including:

[0019] The signal decoding module is configured to perform signal decoding on the audio bitstream to obtain the audio feature estimation value corresponding to the audio bitstream;

[0020] The first reconstruction module is configured to perform a first reconstruction on the audio feature estimate through the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream.

[0021] The second reconstruction module is configured to perform a second reconstruction on the audio feature estimate through a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream;

[0022] Wherein, the number of convolutional layers in the backbone network is greater than the number of convolutional layers in the secondary network, and the dimension of the first reconstructed signal is the same as the dimension of the second reconstructed signal;

[0023] The fusion module is configured to fuse the first reconstructed signal and the second reconstructed signal to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0024] This application provides an audio encoding device, including:

[0025] The feature extraction module is configured to extract features from the audio signal to obtain the audio features of the audio signal;

[0026] The signal encoding module is configured to encode the audio features to obtain an audio bitstream of the audio signal, wherein the audio bitstream is used for decoding based on the audio decoding method provided in the embodiments of this application.

[0027] This application provides a method for processing audio bitstreams, wherein the audio bitstreams are decoded based on the audio decoding method provided in this application, or generated according to the audio encoding method provided in this application.

[0028] This application provides an electronic device, the electronic device comprising:

[0029] Memory is used to store executable instructions or computer programs.

[0030] When a processor executes computer-executable instructions or computer programs stored in the memory, it implements the audio encoding method or the audio decoding method provided in the embodiments of this application.

[0031] This application provides a computer-readable storage medium storing a bitstream formed by computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, they implement the audio encoding method or the audio decoding method provided in this application.

[0032] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, they implement the audio encoding method or the audio decoding method provided in this application.

[0033] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the audio encoding method or the audio decoding method provided in this application.

[0034] The embodiments of this application have the following beneficial effects:

[0035] Because the backbone network contains more convolutional layers than the secondary network, it can reconstruct a deeper first reconstructed signal, making it easier to capture deeper information in the audio signal. On the other hand, because the secondary network has relatively fewer convolutional layers than the backbone network, it can reconstruct a shallower second reconstructed signal, making it easier to capture shallower information in the audio signal. By fusing the two signals (i.e., the first reconstructed signal and the second reconstructed signal), both deep and shallow information can be utilized simultaneously, making the reconstructed audio signal contain richer audio information, so as to better recover the audio signal and thus improve the audio decoding quality. Attached Figure Description

[0036] Figure 1 is a schematic diagram of spectrum comparison at different code rates provided in the embodiments of this application;

[0037] Figure 2 is a schematic diagram of the audio encoding and decoding system architecture provided in an embodiment of this application;

[0038] Figure 3A is a first structural schematic diagram of the electronic device provided in an embodiment of this application;

[0039] Figure 3B is a schematic diagram of the second structure of the electronic device provided in an embodiment of this application;

[0040] Figure 4A is a first flowchart of the audio encoding method provided in an embodiment of this application;

[0041] Figure 4B is a second flowchart of the audio encoding method provided in the embodiment of this application;

[0042] Figure 5 is a flowchart illustrating the audio decoding method provided in an embodiment of this application;

[0043] Figure 6 is a schematic diagram of a voice communication link provided in an embodiment of this application;

[0044] Figure 7A is a first flowchart of the speech encoding and decoding method provided in an embodiment of this application;

[0045] Figure 7B is a second flowchart of the speech encoding and decoding method provided in an embodiment of this application; Figure 8A is a schematic diagram of a common convolutional network provided in an embodiment of this application;

[0046] Figure 8B is a schematic diagram of a dilated convolutional network provided in an embodiment of this application;

[0047] Figure 9 is a schematic diagram of the first analysis network provided in an embodiment of this application;

[0048] Figure 10A is a schematic diagram of the residual block structure used in the coding block provided in the embodiment of this application;

[0049] Figure 10B is a schematic diagram of the residual unit structure provided in an embodiment of this application;

[0050] Figure 11A is a first schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in an embodiment of this application;

[0051] Figure 11B is a network structure diagram of the second long-term connection network provided in an embodiment of this application;

[0052] Figure 11C is a schematic diagram of a structure of the first reconstruction network and the second long-term connection network provided in an embodiment of this application;

[0053] Figure 12A is a second schematic diagram of one structure of the first analysis network and the first long-term connection network provided in an embodiment of this application;

[0054] Figure 12B is a second schematic diagram of a structure of the first reconstruction network and the second long-term connection network provided in an embodiment of this application;

[0055] Figure 13A is a third schematic diagram of one structure of the first analysis network and the first long-term connection network provided in an embodiment of this application;

[0056] Figure 13B is a fourth schematic diagram of a structure of the first analysis network and the first long-term connection network provided in an embodiment of this application;

[0057] Figure 13C is a third schematic diagram of a structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application;

[0058] Figure 13D is a fourth schematic diagram of a structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application;

[0059] Figure 13E is a fourth schematic diagram of a structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application;

[0060] Figure 13F is a fourth schematic diagram of a structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application.

[0061] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0064] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0065] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0066] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0067] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0068] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0069] 1) Neural Network (NN): A mathematical model that mimics the behavioral characteristics of animal neural networks to perform distributed parallel information processing. This type of network relies on the complexity of the system to adjust the relationships between a large number of interconnected nodes, thereby achieving the purpose of information processing.

[0070] 2) Deep Learning (DL): This is a new research direction in the field of Machine Learning (ML). Deep learning learns the inherent patterns and representational layers of sample data. The information gained during this learning process greatly helps in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to have analytical and learning capabilities like humans, and to be able to recognize data such as text, images, and sound.

[0071] 3) Quantization: This refers to the process of approximating a signal's continuous values ​​(or a large number of discrete values) into a finite number (or a small number) of discrete values. Quantization includes vector quantization (VQ) and scalar quantization.

[0072] Vector quantization is an effective lossy compression technique based on Shannon's rate distortion theory. The basic principle of vector quantization is to use the index (also called the quantization value) of the codeword in the code table that best matches the input vector for transmission and storage, while decoding only requires a simple table lookup operation. For example, a vector space can be formed by combining several scalar data points, and this vector space can be divided into several small regions. During quantization, the vector falling into a small region is replaced with its corresponding index as the input vector.

[0073] Scalar quantization is the quantization of a scalar, that is, one-dimensional vector quantization. It divides the dynamic range into several small intervals, each with a representative value (i.e., index). When the input signal falls into a certain interval, the input signal is quantized into that representative value.

[0074] 4) Entropy coding: A lossless coding method that does not lose any information during the encoding process according to the entropy principle. It is also a key module in lossy coding and is located at the end of the encoder. Entropy coding includes Shannon coding, Huffman coding, Exponential Golomb coding, and Arithmetic coding.

[0075] 5) Broadband: Reflects the resolution or sampling rate of an audio signal. According to standards organizations such as the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) and the 3rd Generation Partnership Project (3GPP), broadband has a sampling rate of 16000 Hz (Hertz) and an effective bandwidth of up to 8000 Hz. Broadband transmission rates are typically above 1.54 Mbps (megabits per second).

[0076] 6) Ultra-wideband (UWB): Reflects the resolution or sampling rate of an audio signal. According to standards organizations such as ITU-T and 3GPP, UWB has a sampling rate of 32000Hz and an effective bandwidth of up to 16000Hz. UWB is a wireless communication technology; generally, it refers to an audio signal bandwidth of at least 500MHz, or a bandwidth-to-center frequency ratio exceeding 20%.

[0077] Speech coding technology aims to transmit as much speech information as possible with minimal network bandwidth. Speech codecs can achieve compression ratios of over 10 times, meaning that 10MB of speech data can be compressed to only 1MB for transmission, significantly reducing the bandwidth required for information transmission. For example, for a wideband speech signal with a sampling rate of 16000Hz, using a 16-bit sampling depth (the fineness of speech intensity recording during sampling), the uncompressed version has a bitrate (data transmission rate per unit time) of 256kbps. However, with speech coding technology, even lossy coding, the reconstructed speech signal quality can approach that of the uncompressed version within a bitrate range of 10-20kbps, and may even be perceived as indistinguishable. For services requiring higher sampling rates, such as 32000Hz ultrawideband speech, the bitrate range must be at least 30kbps.

[0078] In communication systems, to ensure smooth communication, the industry deploys standard voice codec protocols, such as standards from international and domestic standards organizations like ITU-T, 3GPP, IETF, AVS, and CCSA, including standards like G.711, G.722, AMR series, EVS, and OPUS. Figure 1 shows a spectrum comparison diagram at different bit rates to illustrate the relationship between compression bit rate and quality. Curve 101 is the spectrum curve of the original speech, i.e., the uncompressed signal; curve 102 is the spectrum curve of the OPUS encoder at a bit rate of 20kbps; and curve 103 is the spectrum curve of the OPUS encoding at a bit rate of 6kbps. As shown in Figure 1, as the encoding bit rate increases, the compressed signal is closer to the original signal.

[0079] The principle of speech coding is roughly as follows: speech coding can directly encode speech waveform samples one by one; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.

[0080] The above-mentioned encoding principles are all derived from speech signal modeling, i.e., compression methods based on signal processing, which cannot guarantee the encoding quality of audio. To improve encoding efficiency while ensuring speech quality, embodiments of this application provide an audio encoding method, an audio decoding method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as a terminal device, a server, or a combination of both. The following description uses the example of an electronic device implemented as a terminal device.

[0081] For example, see Figure 2, which is a schematic diagram of the architecture of the audio codec system 100 provided in the embodiment of this application. The audio codec system 100 includes: server 200, network 300, terminal device 400 (i.e., encoding end) and terminal device 500 (i.e., decoding end). The network 300 can be a local area network, a wide area network, or a combination of the two.

[0082] In some embodiments, a client 410 runs on the terminal device 400. The client 410 can be various types of clients, such as instant messaging clients, web conferencing clients, live streaming clients, browsers, etc. In response to an audio acquisition command triggered by the sender (e.g., the initiator of a web conference, a broadcaster, the initiator of satellite communication, etc.), the client 410 calls the microphone built into the terminal device 400 to acquire audio signals, and performs audio encoding processing on the acquired audio signals to obtain an audio bitstream.

[0083] For example, client 410 calls the audio encoding method provided in the embodiments of this application to encode the acquired audio signal, that is, to extract features from the audio signal to obtain audio features of the audio signal; and to encode the audio features to obtain the audio bitstream of the audio signal, wherein the audio bitstream is used for decoding based on the audio decoding method provided in the embodiments of this application.

[0084] Client 410 can send the audio stream to server 200 via network 300, so that server 200 can send the audio stream to the terminal device 500 associated with the recipient (such as participants in a web conference, viewers, or receivers of satellite communications).

[0085] After receiving the audio stream sent by the server 200, the client 510 (such as an instant messaging client, web conferencing client, live streaming client, browser, etc.) can decode the audio stream to obtain a synthesized audio signal, thereby realizing audio communication.

[0086] For example, client 510 calls the audio decoding method provided in this application embodiment to decode the received audio bitstream, that is, to perform signal decoding on the audio bitstream to obtain the audio feature estimation value corresponding to the audio bitstream; to perform a first reconstruction on the audio feature estimation value through the backbone network to obtain a first reconstructed signal corresponding to the audio bitstream; to perform a second reconstruction on the audio feature estimation value through the secondary network to obtain a second reconstructed signal corresponding to the audio bitstream; wherein, the number of convolutional layers included in the backbone network is greater than the number of convolutional layers included in the secondary network, and the dimension of the first reconstructed signal is the same as the dimension of the second reconstructed signal; and the first reconstructed signal and the second reconstructed signal are fused to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0087] For example, the server 200 shown in Figure 2 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal devices 400 and 500 shown in Figure 2 can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, in-vehicle terminals, etc., but are not limited to these. The terminal devices (e.g., terminal devices 400 and 500) and server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0088] In some embodiments, the terminal device or server 200 can also implement the audio encoding or decoding method provided in this application embodiment by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as a live streaming APP, a web conferencing APP, or an instant messaging APP; it can also be a mini-program, that is, a program that only needs to be downloaded into a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0089] Referring to Figure 3A, which is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of this application, and taking a terminal device as an example, the electronic device 500 shown in Figure 3A includes: at least one processor 520, a memory 550, at least one network interface 530, and a user interface 540. The various components in the electronic device 500 are coupled together through a bus system 550. It is understood that the bus system 550 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 550 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 550 in Figure 3A.

[0090] The processor 520 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0091] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 520.

[0092] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0093] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0094] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0095] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 530, exemplary network interfaces 530 including Bluetooth, WiFi, and Universal Serial Bus (USB).

[0096] In some embodiments, the audio encoding device provided in this application can be implemented in software. FIG3A shows an audio encoding device 555 stored in memory 550, which can be software in the form of programs and plug-ins, including the following software modules: feature extraction module 5551 and signal encoding module 5552. The feature extraction module 5551 and signal encoding module 5552 are used to implement audio encoding functions. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented.

[0097] Referring to Figure 3B, which is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of this application, and taking the electronic device 600 as a terminal device as an example, the electronic device 600 shown in Figure 3B includes: at least one processor 620, a memory 650, at least one network interface 630, and a user interface 640. The various components in the electronic device 600 are coupled together through a bus system 650. The memory 650 includes an operating system 651 and a network communication module 652. It should be noted that the function of the structure in Figure 3B is similar to that of the structure in Figure 3A. The audio decoding device provided in this application embodiment can be implemented in software. Figure 3B shows the audio decoding device 655 stored in the memory 650, which can be software in the form of programs and plug-ins, including the following software modules: signal decoding module 6551, first reconstruction module 6552, second reconstruction module 6553, and fusion module 6554. The signal decoding module 6551, first reconstruction module 6552, second reconstruction module 6553, and fusion module 6554 are used to implement the audio decoding function. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented.

[0098] Before introducing the audio encoding and decoding method provided in the embodiments of this application, we will first introduce dilated convolutional networks.

[0099] Referring to Figures 8A and 8B, Figure 8A is a schematic diagram of a conventional convolutional network (e.g., causal convolution) provided in an embodiment of this application, and Figure 8B is a schematic diagram of a dilated convolutional network provided in an embodiment of this application. Compared to conventional convolutional networks, dilated convolution can increase the receptive field while maintaining the same feature map size, and can also avoid errors caused by upsampling and downsampling. Although the kernel size shown in Figures 8A and 8B is 3×3, the receptive field 801 of the conventional convolution shown in Figure 8A is only 3, while the receptive field 802 of the dilated convolution shown in Figure 8B reaches 5. That is, for a convolutional kernel of size 3×3, the receptive field of the conventional convolution shown in Figure 8A is 3 and the dilation rate (the number of intervals between points in the convolutional kernel) is 1; while the receptive field of the dilated convolution shown in Figure 8B is 5 and the dilation rate is 2.

[0100] The convolutional kernel can also move on a plane similar to Figure 8A or Figure 8B, which involves the concept of stride rate. For example, if the convolutional kernel moves 1 unit at a time, the corresponding stride rate is 1.

[0101] In addition, there is the concept of convolution channel count, which refers to the number of parameters corresponding to the convolution kernels used in convolution analysis. Theoretically, the more channels there are, the more comprehensive the signal analysis and the higher the accuracy; however, the more channels there are, the higher the complexity. For example, a 1×320 tensor can be used with a 24-channel convolution operation, and the output will be a 24×320 tensor.

[0102] It should be noted that the size of the dilated convolution kernel (for example, for speech signals, the kernel size can be set to 1×3), dilation rate, shift rate, and number of channels can be defined according to actual application needs. This application does not impose specific limitations on these aspects.

[0103] As mentioned above, the audio encoding method provided in this application embodiment can be implemented by various types of electronic devices. Referring to Figure 4A, which is a first flowchart of the audio encoding method provided in this application embodiment, the audio encoding function is realized through the audio encoding method. The following description is based on steps 101-104 shown in Figure 4A.

[0104] Before explaining the following steps, the two encoding methods involved in the audio encoding method provided in the embodiments of this application are explained (i.e., the signal encoding method in signal processing technology and the feature encoding (i.e., feature extraction) in artificial intelligence technology). For details of feature encoding in artificial intelligence technology, please refer to step 101 below, and for details of the signal encoding method in signal processing technology, please refer to step 102 below.

[0105] In step 101, feature extraction is performed on the audio signal to obtain the audio features of the audio signal.

[0106] For example, audio signals can be obtained by recording in real time using a microphone device, or an audio file can be loaded locally as the audio signal. This application embodiment does not limit the specific method of obtaining the audio signal.

[0107] In some embodiments, feature extraction is implemented through a feature extraction network, which includes multiple cascaded fourth neural network layers. Step 101 can be implemented as follows: feature extraction is performed on the input of the t-th fourth neural network layer through the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer, where t is a positive integer that increases sequentially, 0 < t ≤ T, and T is the number of the fourth neural network layers in the feature extraction network. The input of the first fourth neural network layer is an audio signal. When the t-th fourth neural network layer is not the first fourth neural network layer, the input of the t-th fourth neural network layer is the feature extraction result of the (t-1)-th fourth neural network layer. The feature extraction result of the T-th fourth neural network layer is determined as the audio feature.

[0108] By using a feature extraction network constructed from multiple cascaded fourth neural network layers to extract features from audio signals, the original audio signals can be abstracted and transformed at multiple levels (which can be understood as data compression), making the vector dimension of audio features smaller than that of the audio signal, thereby reducing the amount of data and facilitating subsequent encoding and transmission.

[0109] The embodiments of this application are not limited to the form of the fourth neural network layer, such as causal convolutional layers, deep learning networks, etc. As an example, feature extraction is performed on the audio signal through the first fourth neural network layer to obtain the feature extraction result of the first fourth neural network layer. The feature extraction result of the first fourth neural network layer is used as the input of the second fourth neural network layer. The feature extraction result of the first fourth neural network layer is then extracted through the second fourth neural network layer to obtain the feature extraction result of the second fourth neural network layer, and so on, until the feature extraction result of the last fourth neural network layer is determined as the audio feature.

[0110] Thus, this embodiment of the application explicitly states that the feature extraction network consists of multiple cascaded fourth neural network layers, and that audio features are extracted sequentially by each layer, with the final output determined by the last layer. This hierarchical, progressive feature extraction method gradually removes redundant information from the original audio signal and strengthens core features (such as frequency, amplitude, and temporal correlation features), avoiding incomplete or distorted feature extraction caused by single-layer processing. Furthermore, by fixing the cascade logic "from the first fourth neural network layer to the Tth fourth neural network layer," the stability and repeatability of the feature extraction process are ensured, enabling the output audio features to accurately match subsequent signal encoding, laying the foundation for high-quality decoding of the encoded audio bitstream. At the same time, the hierarchical design provides technical adaptation space for flexibly adjusting the feature extraction accuracy (such as increasing or decreasing the number of fourth neural network layers), improving the versatility of the encoding method.

[0111] In some embodiments, feature extraction is performed on the input of the t-th fourth neural network layer through the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer. This can be achieved by performing one of the following processes through the t-th fourth neural network layer of the feature extraction network: performing convolution processing on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer; performing preprocessing on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer; or performing encoding processing on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer.

[0112] Here, on the audio encoding side, convolution processing is a basic feature extraction method of the fourth neural network layer in the feature extraction network. It refers to the fourth neural network layer performing convolution operations (such as causal convolution) on the input (which can be the original audio signal or the output of the previous fourth neural network layer) to convert the input tensor into convolutional features with a specific number of channels and size.

[0113] On the audio encoding side, preprocessing is a way for the fourth neural network layer in the feature extraction network to optimize the input (which can be the original audio signal or the output of the previous fourth neural network layer). For example, it involves performing a combination of convolution operations, pooling operations with specified factors (such as pooling with a factor of 2), and activation functions (such as PReLU) on the results of the previous convolutional processing layer (such as 16×320 causal convolutional features). The purpose is to refine the expanded convolutional features of the previous layer into a more compact and expressive feature tensor (such as generating a 16×160 tensor) through dimensionality adjustment (such as reducing the feature size from 320 to 160) and nonlinear transformation, so as to provide optimized input for subsequent feature processing.

[0114] On the audio encoding side, the encoding process is a feature deepening method adopted by the fourth neural network layer in the feature extraction network. It processes the input (which can be the original audio signal or the output of the previous fourth neural network layer). For example, as an intermediate link between preprocessing and final convolution processing, its role is to gradually extract and deepen the feature representation of the audio signal, transform the primary / optimized features output by the previous layer into more abstract and audio-essential hierarchical features, and is the core intermediate step in constructing a "feature increasing chain" from the original audio signal to the final audio features.

[0115] For example, refer to the schematic diagram of the first analysis network (i.e., the feature extraction network, which includes multiple cascaded fourth neural network layers) shown in Figure 9. The multiple cascaded fourth neural network layers correspond to the causal convolution layer, preprocessing layer, and encoding block shown in Figure 9. As shown in Figure 9, when n = 1, the input to the first fourth neural network layer is subjected to convolution processing to obtain the feature extraction result of the first fourth neural network layer. That is, by invoking a 16-channel causal convolution layer, the input tensor (i.e., the audio signal x(n)) can be expanded into a 16×320 causal convolution feature (i.e., the feature extraction result of the first fourth neural network layer). When n = 2, the input to the second fourth neural network layer is preprocessed to obtain the feature extraction result of the second fourth neural network layer. That is, after expanding the input tensor (i.e., the audio signal) into a 16×320 causal convolution feature (i.e., the feature extraction result of the first fourth neural network layer) by invoking a 16-channel causal convolution layer, a convolution operation, pooling (Pooling) processing with a factor of 2, and the activation function can be a parametric rectified linear unit (PReLU) are performed to generate a 16×160 tensor (i.e., the feature extraction result of the second fourth neural network layer). When 2 < n < N, the input to the nth fourth neural network layer is encoded (implemented through an encoding block) to obtain the feature extraction result of the nth fourth neural network layer. When n = N, the input to the Nth fourth neural network layer is subjected to convolution processing to obtain the first audio feature (i.e., F1(n) in Figure 9).

[0116] Thus, this embodiment of the application explicitly allows the t-th fourth neural network layer of the feature extraction network to selectively perform convolution processing, preprocessing, or encoding processing to achieve feature extraction. This flexibly adapts to different needs of hierarchical feature extraction (convolution processing can enhance local feature aggregation, preprocessing can optimize the purity of the input signal (such as noise reduction and normalization), and encoding processing can achieve feature compression and semantic enhancement). This allows each layer to select the optimal processing method according to the characteristics of the input signal, avoiding the limitations of a single processing method in adapting to complex audio signals. Through the flexible switching of multiple processing methods, the extracted audio features are more targeted and accurate, ensuring that the audio features can fully carry the key information of the original audio signal, providing high-quality support for feature restoration after subsequent signal encoding and decoding. At the same time, it enriches the technical implementation flexibility of the feature extraction network and can adapt to the personalized encoding needs of different audio types.

[0117] In some embodiments, encoding the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer can be achieved in the following way: performing residual processing on the input of the t-th fourth neural network layer through the residual block included in the t-th fourth neural network layer to obtain residual features; performing convolution processing on the residual features through the convolutional layer included in the t-th fourth neural network layer to obtain convolutional features; and performing downsampling on the convolutional features through the downsampling layer included in the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer.

[0118] The residual processing involves introducing skip connections through residual blocks. This decomposes the input (which can be the original audio signal or the output of the preceding fourth neural network layer) into an "identity mapping path" (the input is directly passed to the output) and a "residual mapping path" (the input undergoes convolution, activation, and other operations to generate residuals). Finally, the results of the two paths are added together to obtain the output (i.e., the residual feature). Essentially, the residual block (a type of deep network) learns the "difference between the input and the output (residual)" rather than directly learning the complex mapping from the input to the output. This ensures effective gradient backpropagation and supports the stable training of deep networks (such as those with more than 100 layers).

[0119] Downsampling is an operation that compresses the spatial dimensions of feature maps (i.e., convolutional features). It reduces the spatial dimensions of feature maps, such as width and height, through pooling (e.g., max pooling, average pooling) or convolution with a stride greater than 1. The core objectives are: to reduce subsequent computation (reduce feature size to reduce parameters and computation), expand the receptive field (allow feature points to cover a larger area of ​​the original input), and extract high-level semantic features (highlight more abstract global information by compressing spatial dimensions). It is a key step in the feature extraction process, transitioning from local details to global semantics.

[0120] For example, convolutional features are obtained by sliding a convolutional kernel (filter) on the feature map of the residual features and calculating a weighted sum of local regions. Convolutional processing can capture local patterns and structural information in audio signals, thereby extracting higher-level features.

[0121] The fourth neural network layer includes residual blocks comprising multiple cascaded residual units. The residual block of the t-th fourth neural network layer is used to process the input of the t-th fourth neural network layer to obtain residual features. This can be achieved as follows: the first residual unit of the multiple cascaded residual units processes the input of the t-th fourth neural network layer; the residual result output from the first residual unit is output to subsequent cascaded residual units, and residual processing and output of residual results continue through these subsequent cascaded residual units; the residual result output from the last residual unit is used as the residual feature. The embodiments of this application do not limit the number of residual blocks, which can be any positive integer such as 2, 3, 4, or 5. The embodiments of this application also do not limit the number of residual units in a residual block, which can be any positive integer such as 2, 3, 4, 5, or 6. The number of residual units in multiple residual blocks can be the same or different; for example, one residual block may contain 4 residual units, and another residual block may contain 5 residual units.

[0122] The processing of the residual units in the fourth neural network layer is similar to that in the audio decoding method described below. For details, please refer to the processing of the residual units in the audio decoding method described below.

[0123] In neural network models, a residual unit is a special structure used to build residual networks (ResNets). Residual units aim to address the vanishing and exploding gradient problems during deep neural network training, and to help the network learn features more effectively. Residual units introduce skip connections, where the input is directly added to the output, rather than simply passed between layers. This skip connection allows the network to learn the residual function—that is, the difference between the input and output—rather than a direct mapping. This design makes the network easier to optimize and also helps mitigate the vanishing gradient problem.

[0124] Here, through residual processing on the encoding side, based on the characteristics of residual processing, it is ensured that the input of the residual unit is fully learned, while also making better use of the shallow feature information of the input of the residual unit, thus avoiding the omission of shallow feature information.

[0125] For example, the embodiments of this application are not limited to the processing method of the downsampling layer included in the fourth neural network layer. Downsampling can be performed by pooling methods such as max pooling and average pooling, or by methods such as bilinear interpolation and nearest neighbor interpolation. The embodiments of this application do not limit the specific downsampling method.

[0126] It should be noted that step 101 is similar to step 11.

[0127] Thus, this embodiment of the application alleviates the gradient vanishing problem in deep networks and preserves effective features through residual processing, further aggregates key feature information through convolution processing, and achieves feature dimension compression and core information focus through downsampling. The three work together to form a complete link of "feature optimization - information aggregation - dimension simplification", so that the feature extraction results output by this level have purity, relevance and compactness. It can not only meet the high-efficiency compression requirements of subsequent signal encoding, but also ensure that the encoded audio bitstream is accurately restored in the decoding method. At the same time, the structured processing flow improves the stability and efficiency of feature extraction, avoids the increase in encoding complexity caused by feature redundancy, further strengthens the synergistic adaptability of encoding and decoding, and ensures the balance of quality and efficiency of the entire "encoding-decoding" link of audio signals.

[0128] Referring to Figure 4B, Figure 4B shows that step 101 in Figure 4A can be achieved through steps 1011-1013:

[0129] In step 1011, the first feature of the audio signal is extracted through the backbone network to obtain the first audio feature of the audio signal.

[0130] The processing of the backbone network in the encoding end is similar to that of the feature extraction network described above, and will not be repeated here.

[0131] Following step 1011, in step 1012, the audio signal is subjected to second feature extraction through a secondary network to obtain the second audio feature of the audio signal.

[0132] It should be noted that the secondary network has fewer convolutional layers than the backbone network. This design facilitates the capture of shallow details in the audio signal, reduces computational resource consumption, and improves processing efficiency.

[0133] In some embodiments, the secondary network includes multiple cascaded fifth neural network layers; the second audio feature of the audio signal is obtained by extracting a second feature from the audio signal through the secondary network, which can be achieved as follows: obtaining the feature extraction result of the a-th fourth neural network layer in the backbone network, where the feature extraction result of the a-th fourth neural network layer is obtained based on the audio signal, 0 < a < N, and a is a positive integer; performing feature extraction on the input of the b-th fifth neural network layer through the b-th fifth neural network layer of the secondary network to obtain the feature extraction result of the b-th fifth neural network layer, where b is a positive integer that increases sequentially, 0 < b ≤ B, and B is the number of the fifth neural network layers in the secondary network, and the input of the first fifth neural network layer is the feature extraction result of the a-th fourth neural network layer; and determining the feature extraction result of the B-th fifth neural network layer as the second audio feature.

[0134] Continuing from the previous example, to extract features from the input of the b-th fifth neural network layer in the secondary network, the feature extraction result of the b-th fifth neural network layer can be obtained in the following ways: Perform one of the following processes on the b-th fifth neural network layer in the secondary network: perform convolution processing on the input of the b-th fifth neural network layer to obtain the feature extraction result of the b-th fifth neural network layer; or perform encoding processing on the input of the b-th fifth neural network layer to obtain the feature extraction result of the b-th fifth neural network layer.

[0135] The convolution process in "convolution processing the input of the b-th fifth neural network layer" is similar to the convolution process in "convolution processing the input of the t-th fourth neural network layer" mentioned above. The encoding process in "encoding processing the input of the b-th fifth neural network layer" is similar to the encoding process in "encoding processing the input of the t-th fourth neural network layer" mentioned above, and will not be repeated here.

[0136] Continuing from the previous example, encoding the input of the b-th fifth neural network layer to obtain the feature extraction result of the b-th fifth neural network layer can be achieved as follows: At least one convolutional process is performed on the input of the b-th fifth neural network layer using at least one convolutional layer to obtain convolutional features; then, pooling is performed on the convolutional features using a pooling layer included in the b-th fifth neural network layer to obtain the feature extraction result of the b-th fifth neural network layer.

[0137] In other embodiments, the secondary network includes multiple cascaded fifth neural network layers. The second audio feature of the audio signal is extracted through the secondary network, which can be achieved as follows: The feature extraction result of the a-th fourth neural network layer in the backbone network is obtained, where the feature extraction result of the a-th fourth neural network layer is based on the audio signal, 0 < a < N, and a is a positive integer; the feature extraction result of the a-th fourth neural network layer is encoded through the first fifth neural network layer to obtain the feature extraction result of the first fifth neural network layer; the feature extraction result of the (c-1)-th fifth neural network layer is encoded through the c-th fifth neural network layer to obtain the feature extraction result of the c-th fifth neural network layer, where c is a sequentially increasing positive integer, 2 ≤ c < B; the feature extraction result of the (B-1)-th fifth neural network layer is convolved through the B-th fifth neural network layer to obtain the feature extraction result of the B-th fifth neural network layer; the feature extraction result of the B-th fifth neural network layer is determined as the second audio feature.

[0138] It should be noted that step 1012 is similar to step 12.

[0139] Following step 1012, in step 1013, the first audio feature and the second audio feature are fused to obtain a fused audio feature, which is then used as the audio feature for signal encoding.

[0140] It should be noted that the first and second audio features have the same feature dimensions. The fused audio features contain information at different levels, which can provide higher quality feature input for subsequent signal encoding, thereby improving the encoding quality of the audio.

[0141] In some embodiments, the backbone network includes a first subnetwork and a second subnetwork. The first subnetwork is used to perform a first feature extraction. The fusion of the first audio feature and the second audio feature to obtain a fused audio feature can be achieved by adding the first audio feature and the second audio feature to obtain a third audio feature. The third audio feature is then extracted by the second subnetwork to obtain the fused audio feature.

[0142] The specific implementation of the third feature extraction can be found in the explanation of causal convolution in the coding block above, and will not be repeated here.

[0143] In other embodiments, the first audio feature and the second audio feature are added together to obtain the third audio feature; the third audio feature is then nonlinearly mapped using an activation function to obtain the fused audio feature.

[0144] It should be noted that step 1013 is similar to step 13.

[0145] In step 102, the audio features are encoded to obtain the audio bitstream of the audio signal.

[0146] Here, the signal encoding in step 102 is, in the field of digital signal processing, using traditional signal encoding methods to encode audio features to obtain an audio bitstream.

[0147] Here, signal encoding of audio features refers to converting audio features into a format suitable for storage, transmission, or further processing. In this process, audio features are encoded into a specific data stream, namely an audio bitstream, which is more suitable for transmission in network environments with limited bandwidth.

[0148] In some embodiments, before signal encoding the audio features to obtain the audio bitstream of the audio signal, the following processing may be performed: obtaining a target bitrate mode for the audio signal from multiple bitrate modes; correspondingly, step 102 may be implemented in the following way: signal encoding the audio features using the target bitrate mode to obtain the audio bitstream of the audio signal.

[0149] Continuing from the previous example, the target bitrate mode is used to indicate the use of a target code table for signal encoding. Encoding audio features using the target bitrate mode to obtain the audio bitstream can be achieved as follows: Quantizing the audio features using the target code table to obtain quantized values ​​of the fused audio features; then, using the target bitrate corresponding to the target code table, entropy encoding is performed on the quantized values ​​to obtain the audio bitstream. Entropy encoding is a lossless compression technique used to further reduce the amount of data in the index values. This application's embodiments are not limited to any particular form; for example, it could be Huffman coding, arithmetic coding, Shannon-Fano coding, etc.

[0150] As an example, scalar quantization (quantizing each component individually) and entropy coding methods can be used for audio features. Furthermore, this application does not limit the combination of vector quantization (combining multiple adjacent components into a vector for joint quantization) and entropy coding techniques.

[0151] For example, the audio features F(n) of an audio signal can be quantized using scalar quantization (each component is quantized individually) and entropy coding. Furthermore, this application does not limit the combination of vector quantization (multiple adjacent components are combined into a vector for joint quantization) and entropy coding techniques. In addition to entropy coding, direct coding can also be used; for example, if a codebook has 8 elements, 3 bits are used for encoding; if a codebook has 16 elements, 4 bits are used for encoding, and so on.

[0152] As mentioned above, the audio decoding method provided in this application embodiment can be implemented by various types of electronic devices. Referring to Figure 5, which is a first flowchart of the audio decoding method provided in this application embodiment, the audio decoding function is realized through the audio decoding method. The following description is based on steps 201-204 shown in Figure 5.

[0153] Before explaining the following steps, the two decoding methods involved in the audio decoding method provided in the embodiments of this application are explained (i.e., the signal decoding method in signal processing technology and the feature decoding (i.e., the reconstruction method) in artificial intelligence technology). The signal decoding method in signal processing technology is detailed in step 201 below, and the feature decoding in artificial intelligence technology is detailed in step 202 or step 203 below.

[0154] In step 201, the audio bitstream is decoded to obtain the estimated audio features corresponding to the audio bitstream.

[0155] The audio bitstream is generated according to the audio encoding method provided in the embodiments of this application.

[0156] For example, an audio stream file can be read from a local file system or network storage, or a real-time audio stream (such as an audio stream in a live streaming broadcast or video conference) can be received over a network. This application does not limit the specific method of obtaining the audio stream.

[0157] It's important to note that signal decoding is the inverse process of signal encoding. Since the decoding process at the decoding end is the reverse of the encoding process at the encoding end, the values ​​generated during decoding are estimates relative to the values ​​generated during encoding. For example, the data values ​​representing audio features generated during decoding are estimates relative to the audio features in the encoding process. That is, the former is not necessarily equivalent to the original audio feature data values. For instance, there may be subtle differences due to the encoding and decoding processes, and there may be situations where decoding cannot completely reconstruct the original audio feature data values. Therefore, the former is also called an "audio feature estimate."

[0158] As an example, in response to a decoding request for an audio stream encapsulation, the audio stream encapsulation is obtained, and a target bitrate mode is obtained from the frame header included in the audio stream encapsulation. The audio stream included in the audio stream encapsulation is obtained by audio encoding the audio signal using the target bitrate mode and the audio encoding method provided in this embodiment. The target bitrate mode is obtained from multiple bitrate modes. Here, the decoding request is used to instruct audio decoding of the audio stream.

[0159] The target bitrate mode is used to indicate the use of a target code table for signal decoding of the audio bitstream. Decoding the audio bitstream using the target bitrate mode yields estimated audio features, which can be achieved as follows: entropy decoding of the audio bitstream using the target bitrate corresponding to the target code table to obtain the quantized value; and inverse quantization of the quantized value using the target code table to obtain the estimated audio features.

[0160] Inverse quantization is achieved by querying a quantization table, which is a mapping table generated during the encoding process. For example, upon receiving a bitstream, entropy decoding is first performed, and then the estimated value of the feature vector, i.e., the estimated audio feature value corresponding to the audio bitstream, is obtained by querying the quantization table (i.e., inverse quantization, which is a mapping table generated during the encoding process).

[0161] For example, for an audio bitstream, first, entropy decoding is performed on the audio bitstream using the bitrate corresponding to the target code table to obtain the quantization value corresponding to the audio bitstream. Then, by looking up the target code table, the estimated audio feature value F corresponding to the audio bitstream is obtained. ′ (n).

[0162] In step 202, the audio feature estimates are reconstructed using the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream.

[0163] The first reconstruction method is a reconstruction method based on a backbone network. This method uses audio feature estimates obtained after signal decoding through signal processing techniques. These estimates are then processed by a pre-defined artificial intelligence network (such as a backbone network and secondary networks) to perform hierarchical feature restoration and detail supplementation. Its core is to utilize the feature learning and mapping capabilities of neural networks to convert abstract audio feature estimates into reconstructed signals with real audio physical properties (such as amplitude, frequency, and temporal correlation). This compensates for potential feature loss or distortion during signal decoding, ultimately outputting a decoding result that closely matches the listening experience and quality of the original audio signal. It is a key artificial intelligence technology connecting feature estimation at the signal processing level with the actual perceptible audio output.

[0164] In some embodiments, the backbone network includes multiple cascaded first neural network layers; the first reconstruction of the audio feature estimates through the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream can be achieved in the following way: through the nth first neural network layer of the backbone network, feature reconstruction is performed on the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer, where n is a positive integer that increases sequentially, 0 < n ≤ N, N is the number of the first neural network layers in the backbone network, the input of the first first neural network layer is the audio feature estimate, and when the nth first neural network layer is not the first first neural network layer, the input of the nth first neural network layer is the feature reconstruction result of the (n-1)th first neural network layer; the feature reconstruction result of the Nth first neural network layer is determined as the first reconstructed signal.

[0165] The embodiments of this application are not limited to the form of the first neural network layer, such as a causal convolutional layer, a deep neural network layer, etc. As an example, the audio feature estimate is reconstructed through the first first neural network layer to obtain the feature reconstruction result of the first first neural network layer. The feature reconstruction result of the first first neural network layer is used as the input of the second first neural network layer. The feature reconstruction result of the first first neural network layer is reconstructed through the second first neural network layer to obtain the feature reconstruction result of the second first neural network layer, and so on, until the feature reconstruction result of the last first neural network layer is determined as the first reconstructed signal.

[0166] Thus, this embodiment of the application, by clearly defining that the backbone network consists of multiple cascaded first neural network layers, and that the first reconstructed signal undergoes feature reconstruction sequentially through each layer, with the final result determined by the output of the last layer, fully leverages the structural advantage of the backbone network's "large number of convolutional layers." Through hierarchical progressive feature reconstruction, it gradually deepens the mining, refinement, and calibration of deep-level information in the audio feature estimates, avoiding feature loss or insufficient restoration caused by single reconstruction. Furthermore, by fixing the cascaded processing logic "from the first first neural network layer to the Nth first neural network layer," it ensures the stability and repeatability of the first reconstructed signal generation process, enabling the output first reconstructed signal to more accurately and comprehensively carry deep-level audio features. This provides a high-quality, highly reliable foundation of deep features for subsequent fusion with the second reconstructed signal. At the same time, the hierarchical design provides technical adaptation space for subsequent flexible adjustment of reconstruction accuracy (such as increasing or decreasing the number of first neural network layers).

[0167] In some embodiments, feature reconstruction is performed on the input of the nth first neural network layer through the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer. This can be achieved by performing one of the following processes through the nth first neural network layer of the backbone network: performing convolution processing on the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer; decoding the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer; or performing post-processing on the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer.

[0168] Here, on the audio decoding side, convolution processing refers to the process of performing local feature extraction, weight mapping, and dimensionality transformation on the audio feature-related data (including audio feature estimates or feature reconstruction results output by previous neural network layers) in the nth first neural network layer of the backbone network through a preset convolution kernel. Its core is to use the parameter sharing characteristics of the convolution kernel to aggregate and enhance the spatial or temporal correlation features of the input data, and output a feature matrix (i.e., feature reconstruction result) that retains key audio feature information and meets the requirements of network layer progression, thus providing a foundation for subsequent feature reconstruction or final reconstructed signal generation.

[0169] On the audio decoding side, decoding refers to the process in the nth first neural network layer of the backbone network that, for the input (including audio feature estimates or feature reconstruction results output by previous neural network layers), converts the input (abstract encoded features) into a feature form with physical meaning or that can be directly used for audio reconstruction through inverse encoding operations, codebook mapping, feature decompression, or semantic restoration. Its core is to restore the key audio information (such as amplitude, frequency, and phase correlation features) compressed or converted during the encoding process, eliminate the redundancy or distortion introduced by the encoding, and output reconstruction intermediate data that can accurately reflect the features of the original audio signal.

[0170] On the audio decoding side, post-processing refers to the optimization process performed on the input of the nth first neural network layer of the backbone network (including audio feature estimates or feature reconstruction results output by previous neural network layers), such as noise reduction, artifact removal, feature smoothing, dimensionality calibration, or outlier correction. Its core is to correct problems such as noise interference, distortion introduced by network operations, or feature discontinuity that may exist in the input data, improve the purity and consistency of feature data, and ensure that the output feature reconstruction results are closer to the real features of the original audio signal, providing high-quality feature input for the subsequent fusion process.

[0171] Exemplarily, refer to the schematic diagram of the first reconstruction network shown in FIG. 11C (i.e., the backbone network at the decoding end, and the backbone network includes multiple cascaded first neural network layers). The multiple cascaded first neural network layers correspond to the causal convolution, decoding block, and post-processing shown in FIG. 11C. When n = 1, the audio feature estimate is convolved to obtain the feature reconstruction result of the first first neural network layer. That is, by invoking the causal convolution layer, the input tensor (i.e., the audio feature estimate) can be expanded into a causal convolution feature of 256×1. When 1 < n < N - 1, the input of the nth first neural network layer (the feature reconstruction result of the previous first neural network layer) is decoded to obtain the feature reconstruction result of the nth first neural network layer. When n = N - 1, the input of the (N - 1)th first neural network layer is post-processed to obtain the feature reconstruction result of the (N - 1)th first neural network layer (the 16×320 tensor obtained after post-processing in FIG. 11C). When n = N, the input of the Nth first neural network layer (the feature reconstruction result of the (N - 1)th first neural network layer) is convolved to obtain the reconstructed audio signal of the Nth first neural network layer (the 16×320 tensor x ′ (n)) in FIG. 11C).

[0172] Thus, in the embodiments of the present application, by clearly specifying that any one of convolution processing, decoding, or post-processing can be selectively performed on the nth first neural network layer of the backbone network to achieve feature reconstruction, it not only flexibly adapts to different requirements of hierarchical feature extraction of the backbone network (convolution processing can strengthen local feature aggregation, decoding can restore the core information of the encoded features, and post-processing can optimize feature purity and consistency), enabling each layer to select the optimal processing method according to the characteristics of the input features, avoiding the adaptation limitations of a single processing method to complex audio features, and making the generation process of the first reconstruction signal more targeted and accurate through flexible switching of multiple processing methods, ensuring that the output first reconstruction signal can more comprehensively and high-qualityly carry the deep audio information, laying a solid deep feature foundation for the subsequent fusion with the second reconstruction signal, and at the same time enriching the technical implementation flexibility of the backbone network, which can adapt to the personalized requirements of different audio types (such as speech, music) or decoding scenarios.

[0173] In some embodiments, decoding the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer can be achieved through the following method: performing deconvolution processing on the input of the nth first neural network layer through the deconvolution layer included in the nth first neural network layer to obtain deconvolution features; performing upsampling on the deconvolution features through the upsampling layer included in the nth first neural network layer to obtain upsampled features; performing feature residual processing on the upsampled features through the residual block included in the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer.

[0174] Here, on the audio decoding side, deconvolution processing refers to the process of performing inverse convolution operations on the input audio feature data (including audio feature estimates, feature reconstruction results output by preceding neural network layers, or intermediate features output by the previous layer) through a preset deconvolution kernel in the deconvolution layer. Its core is to map low-dimensional abstract features into high-dimensional detailed features by expanding the feature map size and restoring the feature space dimension, while preserving the correlation between features, and outputting deconvolution features that can reflect more audio detail information, providing basic feature data with dimension adaptation for subsequent upsampling processing.

[0175] On the audio decoding side, upsampling refers to the process of increasing the dimensionality and interpolating the deconvolution features output by the deconvolution process in the upsampling layer. Its core is to increase the spatial resolution of the feature map by using interpolation algorithms (such as bilinear interpolation) or feature copying methods, fill in the detail gaps in the feature map after deconvolution processing, and make the feature dimension more closely match the feature scale of the original audio signal, outputting upsampled features with higher dimensionality and richer details, providing more comprehensive feature input for feature residual processing.

[0176] On the audio decoding side, feature residual processing refers to the process of feature extraction and residual fusion of the input upsampled features (or the residual results output by the preceding residual unit) in the residual block. Its core is to extract feature information through the convolution operation of the residual unit and directly add the convolution result to the input feature (or the preceding residual result) to build a residual connection path. This effectively alleviates the gradient vanishing problem in deep networks, while retaining the effective information of the original features, filtering redundant noise, and outputting residual results with higher accuracy and stronger stability. Finally, it forms the feature reconstruction result of the nth first neural network layer.

[0177] Thus, this embodiment of the application effectively restores the spatial dimension and correlation information of the input features through deconvolution processing, further improves the feature resolution by upsampling to fill in the details, and alleviates the gradient vanishing of deep networks by feature residual processing while retaining effective features. The three work together to form a complete link of "dimensional restoration - detail supplementation - feature optimization", making the feature reconstruction results output by this level more complete and accurate. At the same time, this structured processing flow fully adapts to the core requirements of the backbone network of "multiple convolutional layers and deep reconstruction", further strengthening the ability of the first reconstructed signal to capture and restore the deep information of the audio, and providing higher quality deep feature support for subsequent fusion with the second reconstructed signal.

[0178] Continuing from the previous example, the residual block includes multiple cascaded residual units. The upsampled features are processed using the residual block included in the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer. This can be achieved as follows: The upsampled features are processed using the first residual unit of the multiple cascaded residual units; the residual result output from the first residual unit is passed to subsequent cascaded residual units for further residual processing and output; the residual result output from the last residual unit is used as the feature reconstruction result of the nth first neural network layer.

[0179] In neural network models, a residual unit is a special structure used to build residual networks (ResNets). Residual units aim to address the vanishing and exploding gradient problems during deep neural network training, and to help the network learn features more effectively. Residual units introduce skip connections, where the input is directly added to the output, rather than simply passed between layers. This skip connection allows the network to learn the residual function—that is, the difference between the input and output—rather than a direct mapping. This design makes the network easier to optimize and also helps mitigate the vanishing gradient problem.

[0180] As shown in Figure 11C, the first reconstruction network comprises four cascaded decoding blocks with different upsampling factors. Each decoding block contains a residual block (including at least one residual unit), a convolutional layer, and an upsampling layer. Each residual block includes five dilated convolution-based residual units (the input and output feature dimensions of the residual units remain unchanged). The convolutional layer halves the number of input channels, and the activation function can be PReLU to ensure sufficient data volume and avoid data loss. The upsampling layer is an upsampling operation with an up factor (e.g., using a repeat operation to perform upsampling through repeated padding) to restore the data. Here, the up factors of the four decoding blocks are set to 5, 4, 4, and 2, respectively. Therefore, the number of output channels of the four decoding blocks are set to 128, 64, 32, and 16, respectively. After processing through four decoding blocks, the input 256×1 tensor is converted into tensors of 128×5, 64×20, 32×80, and 16×160, respectively. The embodiments of this application are not limited to the number of decoding blocks; it can be any positive integer such as 2, 3, 4, or 5.

[0181] Thus, the embodiments of this application realize feature residual processing through multi-level cascaded residual units, which not only alleviates the gradient vanishing problem in deep networks and reduces feature loss and distortion, but also optimizes upsampling features layer by layer, improves the accuracy of feature reconstruction results of the nth first neural network layer, provides support for high-quality first reconstruction signal, and thus enhances the core decoding quality. Furthermore, different audio reconstruction needs can be adapted by adding or removing residual units.

[0182] Continuing from the previous example, the first residual unit of a series of cascaded residual units can be used to perform residual processing on the input of the nth first neural network layer. This can be achieved as follows: The first residual unit of the series of cascaded residual units performs the following processing: it performs convolution processing on the input of the nth first neural network layer to obtain the convolution result of the first residual unit; it adds the convolution result of the first residual unit to the residual features to obtain the residual result output by the first residual unit.

[0183] In deep networks, gradients can gradually diminish during backpropagation, making it difficult to update parameters near the input layer and causing network training to stagnate. Residual units, by introducing skip connections (corresponding to adding the convolution result of the first residual unit to the residual features as mentioned above), allow gradients to propagate directly through the residual path, avoiding the vanishing gradient problem during multi-layer propagation. This makes deep networks easier to optimize, and their performance does not decrease with increasing layer count.

[0184] Continuing from the previous example, further residual processing and output of residual results through subsequent cascaded residual units can be achieved as follows: The j-th residual unit performs the following processing through multiple cascaded residual units: Convolve the residual result output by the (j-1)-th residual unit to obtain the convolution result of the j-th residual unit; Add the convolution result of the j-th residual units to the residual result output by the (j-1)-th residual unit to obtain the residual result output by the j-th residual unit; Output the residual result output by the j-th residual unit to the (j+1)-th residual unit; where j is a sequentially increasing positive integer, 1 < j < J, and J is the number of residual units.

[0185] Continuing the example above, each residual unit includes a dilated convolution operator. The input of the nth first neural network layer is convolved through the first residual unit of a cascaded network, which can be achieved as follows: the input of the nth first neural network layer is dilated using the dilated convolution operator included in the first residual unit. That is, the j-th residual unit performs the following processing: the residual result output by the (j-1)-th residual unit is dilated using the dilated convolution operator included in the j-th residual unit, resulting in the dilated convolution result of the j-th residual unit, where j is a sequentially increasing positive integer, 1 < j ≤ J, and J is the number of residual units. It should be noted that each residual unit contains a dilated convolution operator with a specified dilation rate. Using a dilated convolution operator with an asymptotic dilation rate is equivalent to using different receptive fields to extract features of the input at different resolutions, allowing for a more comprehensive analysis of the data. After each residual unit is convolved by the dilated convolution operator with the dilation rate, it is added to the shallow features (i.e., the input of each residual unit) from the jump connection, thereby directly utilizing the shallow feature information, so that the network can make full use of the shallow feature information during the learning process.

[0186] In some embodiments, each residual unit further includes at least one causal convolution operator. After performing dilated convolution processing on the input of the nth first neural network layer using the dilated convolution operator included in the first residual unit, the following processing can be performed: causal convolution processing is applied to the obtained dilated convolution result using the at least one causal convolution operator included in the first residual unit, and the obtained causal convolution result is used as the convolution result of the first residual unit. That is, after performing dilated convolution processing on the residual result output of the (j-1)th residual unit using the dilated convolution operator included in the jth residual unit to obtain the dilated convolution result of the jth residual unit, causal convolution processing is applied to the dilated convolution result of the jth residual unit using the at least one causal convolution operator included in the jth residual unit, and the causal convolution result of the jth residual unit is used as the convolution result of the jth residual unit. It should be noted that each residual unit further includes at least one causal convolution operator, through which local information of the features input to the causal convolution operator is further extracted.

[0187] In some embodiments, grouped convolution can be applied to the convolution operators (including dilated convolution operators and causal convolution operators) of the residual units. Grouped convolution divides the input channels into multiple groups for convolution operations, with the input and output channels only associated within each group. It should be noted that when the input channels are divided into multiple groups, the corresponding output channels are also divided into multiple groups; that is, the number of input channel groups is the same as the number of output channel groups, thus ensuring that after convolution within a group, the input and output channels are associated only within that group. Here, it is assumed that the input channels and output channels of a feature input to a certain convolution operator are 4. If the number of groups is 1, each input channel is associated with all 4 output channels. If the number of groups is 2, the 4 input channels are first divided into two groups, 0-1 and 2-3. Within each group, the input channel is associated with the output channel within that group. For example, input channel 0-1 in the first group is associated with output channel 0-1, and input channel 2-3 in the second group is associated with output channel 2-3.

[0188] For example, when applying grouped convolution to the dilated convolution operator included in the residual unit, dilated convolution processing of the input of the nth first neural network layer can be achieved as follows: the input channels of the input of the nth first neural network layer are grouped to obtain multiple groups, wherein each group includes the first element corresponding to at least two channels in the input of the nth first neural network layer; dilated convolution processing is performed on the first element within each group. Similarly, when applying grouped convolution to the causal convolution operator included in the residual unit, causal convolution processing of the resulting dilated convolution result can be achieved as follows: the input channels of the dilated convolution result are grouped to obtain multiple groups, wherein each group includes the second element corresponding to at least two channels in the dilated convolution result; causal convolution processing is performed on the second element within each group.

[0189] In step 203, the audio feature estimates are reconstructed using a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream.

[0190] The backbone network has a greater number of convolutional layers than the secondary network, and the dimensions of the first reconstructed signal are the same as those of the second reconstructed signal.

[0191] It should be noted that the process of the second reconstruction in step 203 is similar to the process of the first reconstruction in step 202.

[0192] In some embodiments, the secondary network includes multiple cascaded second neural network layers; the second reconstruction of the audio feature estimates through the secondary network to obtain the second reconstructed signal corresponding to the audio bitstream can be achieved in the following way: obtaining the feature reconstruction result of the g-th first neural network layer in the backbone network, wherein the feature reconstruction result of the g-th first neural network layer is obtained based on the audio feature estimates, 0 < g < N, and g is a positive integer; performing feature reconstruction on the input of the k-th second neural network layer through the k-th second neural network layer to obtain the feature reconstruction result of the k-th second neural network layer, wherein k is a positive integer that increases sequentially, 0 < k ≤ K, K is the number of the second neural network layers in the secondary network, and the input of the first second neural network layer is the feature reconstruction result of the g-th first neural network layer; and determining the feature reconstruction result of the k-th second neural network layer as the second reconstructed signal.

[0193] The process of "reconstructing features from the input of the kth second neural network layer" is similar to the process of "reconstructing features from the input of the nth first neural network layer" described above.

[0194] Thus, this embodiment of the application explicitly states that the secondary network consists of multiple cascaded second neural network layers, and uses the feature reconstruction result of the g-th first neural network layer in the backbone network that has not reached the final level (0 < g < N) as the input of the first second neural network layer of the secondary network. This allows the secondary network to avoid processing audio feature estimates from scratch, and instead perform lightweight and targeted feature reconstruction based on the intermediate depth features initially extracted by the backbone network. While ensuring that the second reconstructed signal can capture the basic features of the audio at a shallow level, it improves the feature correlation and adaptability between the second reconstructed signal and the first reconstructed signal. Furthermore, the hierarchical processing of the secondary network further refines the basic features, making the output second reconstructed signal of better quality. When fused with the first reconstructed signal, it can more efficiently achieve the complementarity of deep and shallow features, while shortening the processing link of the secondary network. It balances decoding efficiency and feature adaptability, and is suitable for more application scenarios that balance real-time performance and quality.

[0195] Continuing with the example above, feature reconstruction of the input of the kth second neural network layer through the kth second neural network can be achieved in the following ways: Perform one of the following processes through the kth second neural network layer: perform deconvolution on the input of the kth second neural network layer to obtain the feature reconstruction result of the kth second neural network layer; or decode the input of the kth second neural network layer to obtain the feature reconstruction result of the kth second neural network layer.

[0196] For example, referring to Figure 13D, Figure 13D is a schematic diagram of an optional structure of the first reconstruction network (i.e., the backbone network, which includes multiple cascaded second neural network layers) and the second long-term connection network (i.e., the secondary network) provided in the embodiments of this application. The second neural network layer corresponds to a convolution-copy layer. When K=4, the feature extraction result of the first first neural network layer is convolved and copied through the first second neural network to obtain the feature reconstruction result of the first second neural network layer. That is, the feature reconstruction result of the first first neural network layer (the 256×1 tensor obtained after causal convolution in Figure 13D) is first convolved and then upsampled by a factor of 5 to generate a 128×5 tensor. The feature reconstruction result of the first second neural network layer is convolved and copied through the second second neural network to obtain a 64×20 tensor. The feature reconstruction result of the second second neural network layer is convolved and copied through the third second neural network to obtain a 32×80 tensor. The feature reconstruction results of the third second neural network layer are convolutionally copied through the fourth second neural network to obtain a 16×160 tensor, which is the second reconstructed signal.

[0197] Thus, this embodiment of the application explicitly allows the k-th second neural network layer of the secondary network to selectively perform deconvolution processing or decoding processing to achieve feature reconstruction. This not only meets the core requirement of the secondary network to perform lightweight reconstruction based on the intermediate features of the backbone network (deconvolution processing can quickly restore the feature dimension, and decoding processing can accurately extract the basic features), but also allows the secondary network to efficiently generate a second reconstructed signal that fits the requirements with a small number of convolutional layers. Furthermore, it improves the flexibility and targeting of the feature reconstruction of the secondary network, making the feature complementarity between the second reconstructed signal and the first reconstructed signal more accurate. At the same time, it simplifies the processing logic of the secondary network and balances decoding efficiency and feature adaptability.

[0198] In some embodiments, decoding the input of the kth second neural network layer to obtain the feature reconstruction result of the kth second neural network layer can be achieved by: performing at least one deconvolution process on the input of the kth second neural network layer through at least one deconvolution layer included in the kth second neural network layer to obtain deconvolution features; and upsampling the convolution features through an upsampling layer included in the kth second neural network layer to obtain the feature reconstruction result of the kth second neural network layer.

[0199] Figure 11B shows the network structure of the second long-term connection network (i.e., the secondary network). The second neural network layer is implemented through a convolution-repeat layer. The second long-term connection network includes four cascaded convolution-repeat layers with different upsampling factors. Each convolution-repeat layer contains one convolutional layer and one upsampling layer. One convolutional layer is used to halve the number of input channels; one upsampling layer contains a specific up factor to complete the upsampling. The number of convolution-repeat layers in this embodiment is not limited and can be any positive integer such as 2, 3, 4, or 5.

[0200] Thus, this embodiment of the application clearly defines the structured implementation path of "at least one deconvolution process + upsampling" for decoding processing in the secondary network. It also uses multiple deconvolution processes to gradually deepen the dimensionality restoration and detail mining of the intermediate features of the backbone network. Combined with upsampling, it further improves feature resolution and fills the gaps in basic features. Under the premise of a lightweight design with fewer convolutional layers in the secondary network, it efficiently generates a second reconstructed signal that is both complete and adaptable. This not only enhances the accuracy of feature complementarity with the first reconstructed signal, but also takes into account decoding efficiency and the quality of basic feature restoration. At the same time, the structured processing flow improves the feasibility and stability of the solution.

[0201] In some embodiments, the secondary network includes multiple cascaded third neural network layers; the second reconstruction of the audio feature estimates through the secondary network to obtain the second reconstructed signal corresponding to the audio bitstream can be achieved in the following way: the input of the q-th third neural network layer is reconstructed through the q-th third neural network layer of the secondary network to obtain the feature reconstruction result of the q-th third neural network layer, where q is a positive integer that increases sequentially, 0 < q ≤ Q, Q is the number of the third neural network layers in the secondary network, and the input of the first third neural network layer is the audio feature estimate; the feature reconstruction result of the Q-th third neural network layer is determined as the second reconstructed signal.

[0202] For example, as shown in Figure 12B, when the secondary network (i.e., the second long-time connection network, which includes multiple cascaded third neural network layers) is called based on the audio feature estimation value, the audio signal is convolved through the first third neural network layer in the secondary network to obtain the feature reconstruction result of the first third neural network layer (a 128×1 tensor obtained after causal convolution in Figure 12B). The feature reconstruction result of the first third neural network layer is then reconstructed (convolutional replication) through the second third neural network layer to obtain the feature extraction result of the second third neural network layer (a 64×4 tensor obtained after convolutional replication in Figure 12B). This process continues until the Qth third neural network layer is used to reconstruct the feature reconstruction result of the Q-1th third neural network layer to obtain the feature reconstruction result of the Qth third neural network layer (a 16×320 tensor obtained after post-processing in Figure 12B).

[0203] Thus, this embodiment of the application explicitly states that the secondary network consists of multiple cascaded third neural network layers, and that the audio feature estimation value is the input of the first third neural network layer. After progressive feature reconstruction, the second reconstructed signal is output. This not only adapts to the core design of the secondary network, which is characterized by "few convolutional layers and lightweight design," but also refines the basic audio features through hierarchical processing, avoiding the problem of insufficient feature extraction caused by single-layer processing. Furthermore, it ensures that the second reconstructed signal can independently and accurately capture the shallow core information of the audio, forming an efficient complement to the first reconstructed signal of deep features generated by the backbone network. At the same time, the hierarchical structure improves the stability and controllability of the secondary network feature reconstruction (such as adapting to different basic feature requirements by increasing or decreasing the number of third neural network layers). While ensuring decoding efficiency, it enhances the quality of the second reconstructed signal, providing reliable basic feature support for the subsequent fusion process.

[0204] In step 204, the first reconstructed signal and the second reconstructed signal are fused to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0205] The embodiments of this application are not limited to the fusion method of the first reconstructed signal and the second reconstructed signal. For example, one fusion method is direct addition fusion, which is based on the high-dimensional and refined first reconstructed signal (focusing on global audio features and detail restoration) extracted by the backbone network (containing more convolutional layers) and the lightweight and real-time second reconstructed signal (focusing on basic audio features and efficient restoration) generated quickly by the secondary network (containing fewer convolutional layers). The feature complementarity is achieved by adding elements one by one, and a third reconstructed signal that takes into account both deep features and basic features is obtained directly. This method can quickly aggregate the advantages of the two types of signals and is suitable for scenarios with high real-time requirements for decoding.

[0206] In some embodiments, the backbone network includes a first subnetwork and a second subnetwork. The first subnetwork is used to perform a first reconstruction. The first reconstructed signal and the second reconstructed signal are fused to obtain the reconstructed audio signal corresponding to the audio bitstream. This can be achieved by adding the first reconstructed signal and the second reconstructed signal to obtain a third reconstructed signal. The third reconstructed signal is then reconstructed by the second subnetwork to obtain the reconstructed audio signal.

[0207] The third reconstruction is a reconstruction method, and the embodiments of this application are not limited to the processing of the third reconstruction.

[0208] For example, referring to Figure 13D, the first reconstructed signal (the 16×160 tensor after decoding by the decoding block in Figure 13D) and the second reconstructed signal (the 16×160 tensor after convolution and copying in Figure 13D) are added together to form the third reconstructed signal (the 16×160 tensor after feature fusion in Figure 13D). The third reconstructed signal is then reconstructed by the second sub-network (corresponding to post-processing and causal convolution in Figure 13D) to obtain the reconstructed audio signal (the 1×320 tensor in Figure 13D).

[0209] The specific implementation of the third reconstruction can be found in the explanation of the first neural network layer above, and will not be repeated here.

[0210] Thus, this embodiment of the application splits the backbone network into a first sub-network (for the first reconstruction) and a second sub-network (for the third reconstruction). This not only enables the rapid aggregation of features at different levels by adding elements one by one, but also effectively solves the feature incompatibility problem that may exist in direct fusion through the processing of the second sub-network. This results in a reconstructed audio signal with a more natural sound and higher accuracy. At the same time, the second sub-network relies on the structural advantages of the backbone network and does not require the construction of an additional independent optimization network, thus balancing feature optimization effect and system lightweighting.

[0211] In other embodiments, the first reconstructed signal and the second reconstructed signal are added together to obtain the third reconstructed signal; the third reconstructed signal is then nonlinearly mapped using an activation function to obtain the reconstructed audio signal.

[0212] For example, referring to Figure 13C, the first reconstructed signal (a 1×320 tensor output by the causal convolution in Figure 13C) and the second reconstructed signal (a 1×320 tensor output by the second temporal connection network in Figure 13C) are added together to obtain the third reconstructed signal. Then, the third reconstructed signal is nonlinearly mapped by a pre-set activation function (e.g., tanh) to obtain the reconstructed audio signal (a 1×320 tensor obtained after feature fusion in Figure 13C).

[0213] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0214] The embodiments of this application can be applied to various audio scenarios, such as voice calls, instant messaging, remote audio and video calls, and even low-bitrate voice scenarios such as emergency support (i.e., walkie-talkies in emergency scenarios) and satellite calls. The following explanation uses voice calls as an example:

[0215] In related technologies, the principle of speech coding is roughly as follows: speech coding can directly encode speech waveform samples one by one; or, based on the principle of human vocalization, relevant low-dimensional features are extracted, the encoding end encodes the features, and the decoding end reconstructs the speech signal based on these parameters.

[0216] The above encoding principles all originate from speech signal modeling, i.e., signal processing-based compression methods. To improve encoding quality while maintaining speech encoding efficiency compared to signal processing-based compression methods, this application provides a speech encoding / decoding method (i.e., an audio encoding method and an audio decoding method). For an input speech signal, the encoder obtains a first feature vector based on a first analysis network (i.e., the backbone network of the encoder); in parallel, it obtains a first long-time connection feature vector using a first long-time connection network (i.e., the secondary network of the encoder). The first feature vector and the first long-time connection feature vector have the same dimension and format. Finally, the first feature vector and the first long-time connection feature vector are added together, and an activation function (such as tanh) is used to obtain the final feature vector. The final feature vector is then compressed and encoded. Specifically, code tables with different quantization precipitates can be used for quantization and encoding of the feature vector to achieve multi-rate encoding / decoding effects. The neural network employs a "block-based" operation, which reduces algorithm complexity and improves encoding performance. The decoding end decodes the received bitstream to obtain the estimated feature vector values, and then calls the inverse process of the corresponding encoding end to complete the generation of the speech signal.

[0217] This application embodiment can be applied to the voice communication link in the satellite communication scenario shown in Figure 6. Taking a Voice over Internet Protocol (VoIP) satellite communication system as an example, the voice encoding and decoding technology involved in this application embodiment is deployed in the encoding and decoding parts to solve the basic function of voice compression. Generally, both the encoder and decoder are deployed on the client or terminal (such as a mobile phone). The encoder is deployed on the uplink client 601, and the decoder is deployed on the downlink client 602. Voice is acquired through the uplink client and pre-processed, enhanced, encoded, etc. The encoded bitstream is transmitted to the downlink client 602 through the network. The downlink client 602 performs decoding, enhancement, etc., so that the decoded voice can be played back on the downlink client 602.

[0218] To ensure forward compatibility (i.e., compatibility between the new encoder and existing encoders), transcoders are deployed at relevant nodes in the operator's network to address the interoperability issue between the new and existing encoders. For example, a mobile phone uses the encoder provided in this embodiment to send a very low bitrate stream to a satellite; the satellite may simply forward the stream; after receiving the stream, the ground receiving station converts it into a G.711 stream at the media gateway; considering that G.711 is a widely supported standard, it can be directly forwarded to the receiving end through the terrestrial operator network. On the other hand, the G.711 stream received by the receiving end is converted into a stream generated by the encoder provided in this embodiment (i.e., a very low bitrate stream) at the media gateway; the converted stream is then transmitted to the satellite through the operator; the satellite may simply forward the stream and transmit it to the mobile phone; the mobile phone, equipped with the decoder provided in this embodiment, can play back the audio. Therefore, through the above methods, the voice encoding and decoding technology provided in this embodiment can be deployed on mobile satellite calling and other services provided by operators.

[0219] The speech encoding and decoding method provided in the embodiments of this application is described below with reference to Figure 7A:

[0220] The following processing is performed on the encoding end:

[0221] For the input signal x(n) of the nth frame (i.e., the audio signal mentioned above), the first analysis network is invoked to obtain the first feature vector F1(n) (i.e., the first audio feature mentioned above). Specifically, the dimension of the first feature vector is smaller than the dimension of the input signal (to reduce the amount of data). A specific implementation is that for each frame x(n), a dilated convolutional neural network (Dilated CNN) (encoder part) is invoked to generate a lower-dimensional feature vector F1(n). This application does not limit the neural network structure of the first analysis network, such as an autoencoder, a fully connected (FC) network, a long short-term memory (LSTM) network, a convolutional neural network (CNN) + LSTM, etc. Internally, the neural network employs a "block-based" operation to reduce algorithm complexity.

[0222] For x(n), the first long-term connection network is invoked in parallel to obtain the first long-term connection feature vector F. LoRa (n) (i.e., the second audio feature mentioned above), in particular, the feature dimension of the first long-time connection feature vector is consistent with the feature dimension of the first feature vector.

[0223] Using the feature merging module, the first feature vector and the first long-time connection feature vector are fused to obtain the first fused feature vector (i.e., the fused audio feature mentioned above). Specifically, F1(n) and F... LoRa After adding the corresponding dimensions (n), the final first fused feature vector F(n) is output through the activation function.

[0224] The first fused feature vector is quantized using either vector or scalar methods to obtain an index value. This index value is then directly encoded or entropy encoded to obtain the bitstream (i.e., the audio bitstream mentioned above), which is then transmitted to the decoder. Specifically, for F(n), code tables with different quantization precisions are used for quantization and encoding to achieve multi-rate encoding and decoding effects.

[0225] The following processing is performed on the decoding end:

[0226] The bitstream encoded by the decoding end obtains the estimated value F′(n) of the first fused feature vector (i.e., the audio feature estimate mentioned above).

[0227] The second long-time connection network (i.e., the secondary network at the encoding end) is invoked to decode the bitstream to obtain the second long-time connection feature vector L(n) (i.e., the second reconstructed signal mentioned above).

[0228] The first reconstruction network (i.e., the backbone network at the encoding end) is invoked, combined with the second long-time connection feature vector L(n), to generate the reconstructed signal estimate x′(n) (i.e., the reconstructed speech signal, also known as the reconstructed audio signal). The second long-time connection feature vector, as one of the inputs to the first reconstruction network, can be fused and further processed at intermediate nodes of the first reconstruction network to reconstruct the speech signal. Specifically, the structure of the first reconstruction network can refer to the structure of the first analysis network, similar to its "inverse" process. Furthermore, depending on the actual codeword length in the bitstream, code tables with different quantization precipitates can be used to generate F′(n) of different precisions to achieve multi-rate encoding and decoding effects.

[0229] The speech encoding and decoding method provided in the embodiments of this application will be described in detail below.

[0230] In some embodiments, a narrowband speech signal with a sampling rate of Fs = 8000Hz is taken as an example (it should be noted that the method provided in this application embodiment is also applicable to scenarios with other sampling rates, including but not limited to: 16000Hz, 32000Hz, and 48000Hz). Similarly, this application embodiment does not limit the frame length combination under other sampling rates. For example, the frame length is set to 40ms; for Fs = 8000Hz, this is equivalent to each frame still containing 320 sample points.

[0231] The encoding and decoding ends will be described in detail below with reference to the flowchart shown in Figure 7A.

[0232] The encoding process is as follows:

[0233] Step 11: Call the first analysis network based on x(n).

[0234] As mentioned earlier, the 320 sample points in the nth frame are denoted as x(n). The purpose of the first analysis network is to generate a lower-dimensional feature vector F1(n) based on the signal x(n) by calling a deep network. In this embodiment, x(n) has a dimension of 320, and F1(n) has a dimension of 12. From the perspective of data volume, it is clear that the first analysis network plays a role in "dimensionality reduction," which is essentially a form of data compression.

[0235] Referring to the network structure diagram of the first analysis network shown in Figure 9, the process of data compression (i.e., feature extraction) by the first analysis network is explained in detail below:

[0236] First, calling a 16-channel causal convolution can generate a 16×320 tensor from the input tensor (i.e., the feature vector, corresponding to the 1×320 feature vector x(n) in the figure).

[0237] Then, the 16×320 tensor is preprocessed. For example, after performing a convolution operation on the 16×320 tensor, a pooling operation with a factor of 2 is performed, and the activation function can be PReLU, to generate a 16×160 tensor.

[0238] Next, four encoding blocks with different downsampling factors are concatenated. Each encoding block contains a residual block, a convolutional layer, and a pooling layer. Each residual block includes five dilated convolution-based residual units (the input and output feature dimensions of the residual units remain unchanged). The convolutional layer doubles the number of input channels, and the activation function can be PReLU to ensure sufficient data volume and avoid data loss. The pooling layer is a pooling operation with a down factor to perform downsampling and achieve data compression. Here, the down factors of the four encoding blocks are set to 2, 4, 4, and 5, respectively. Therefore, the number of output channels of the four encoding blocks are set to 32, 64, 128, and 256, respectively. After processing by the four encoding blocks, the input 16×160 tensor is transformed into tensors of 32×80, 64×20, 128×5, and 256×1, respectively. In this application embodiment, the number of coding blocks is not limited, and can be any positive integer such as 2, 3, 4, 5, etc. Furthermore, this application embodiment does not limit the number of residual units in a coding block, and can be any positive integer such as 2, 3, 4, 5, 6, etc. The number of residual units in multiple coding blocks can be the same or different; for example, one coding block contains 4 residual units, and another coding block contains 5 residual units.

[0239] This section further introduces the Residual Unit. A Residual Unit is a module in a deep neural network that introduces cross-layer connections, making the neural network easier to optimize during training and avoiding problems such as vanishing or exploding gradients. Its core idea is to perform residual learning on the input within the module, that is, to bypass a portion of the layers and directly pass the input information to the output through a direct path, allowing the network to better utilize shallow feature information during the learning process. Figure 10A is a schematic diagram of the residual block structure used in the encoding block of the first analysis network. This residual block includes five dilated convolution-based residual units, each containing a dilated convolution block with a specified dilation rate (e.g., dilation rate = 3). In this embodiment, using five dilated convolution blocks with progressively increasing dilation rates is equivalent to using different receptive fields to extract features of the input at different resolutions, allowing for a more comprehensive analysis of the data. After residual processing through 5 dilated convolutional blocks with specified dilation rates, the residual is added to the input from the jump connection to obtain the output of the residual block, and then the output is output to the convolutional layer connected to the residual block.

[0240] Here, we further describe any residual unit in Figure 10A, as shown in Figure 10B. For any residual unit, it contains at least one dilated convolution with a specified dilation rate (used to expand the receptive field), which can use PReLU as the activation function. Furthermore, one or more causal convolutions (used to extract local information) can be cascaded, which can also use PReLU as the activation function. The kernel size of the dilated convolution with the specified dilation rate can be 3, 5, 7, 9, etc., and the kernel size of the causal convolution can be 1, 3, etc. This application and this embodiment do not limit the kernel size of the dilated convolution with the specified dilation rate or the causal convolution. In addition, the causal convolution or dilated convolution in this embodiment can also be implemented by other convolution units with similar or equivalent functions.

[0241] Furthermore, for residual units, a grouped convolution algorithm is introduced to reduce algorithm complexity. Grouped convolution divides the input channels into multiple groups for convolution operations, associating input and output channels only within each group. Here, we assume there are 16 input channels and 32 output channels. If the number of groups is 1, each input channel is associated with all 32 output channels. If the number of groups is 2, the 16 input channels are first divided into two groups: 0-7 and 8-15. Within each group, the input channel is associated with the output channel within that group. For example, input channels 0-7 in the first group are associated with output channels 0-15, and input channels 8-15 in the second group are associated with output channels 16-31. For instance, the 0th output channel is associated only with input channels 0-7 and not with input channels 8-15, and the 25th output channel is associated only with input channels 8-15 and not with input channels 0-7. This comparison shows that introducing grouped convolutions can avoid any correlation between input channels and all output channels, reducing the number of connections and thus lowering complexity. However, since a larger number of groups reduces the correlation between input and output channels, it also affects the encoding performance; therefore, a larger number of groups is not always better. In this embodiment, the dilated convolutions contained in the five residual blocks corresponding to the four coding blocks can use different group number configurations, as shown in Table 1.

[0242] Table 1. Configuration of the number of blocks used by residual units in different coding blocks

[0243] Finally, the 256×1 tensor is further processed by causal convolution and Tahn activation to output a 12-dimensional first feature vector. Specifically, according to the calculations of the first analysis network, each value in the 12-dimensional first feature vector is between [-1, 1]. Furthermore, this embodiment does not limit the use of two or more causal convolutions. For example, the 256×1 tensor can be processed by one causal convolution to output a 128×1 tensor; then, the 128×1 tensor can be processed by another causal convolution to output a 12-dimensional feature vector (i.e., F1(n)). This increases computational complexity only slightly while achieving better feature vector extraction.

[0244] Step 12: Call the first long-term connection network based on x(n).

[0245] The purpose of the first long-term connection network is to generate another lower-dimensional feature vector F based on the signal x(n) by calling a deep network. LoRa (n). Where F LoRa The dimensions of F1(n) are the same as those of F1(n).

[0246] The role of the first long-term connection network is to "transmit" the characteristics of the input signal through a second path via the first long-term feature vector, and then fuse it with the first feature vector output by the first analysis network via the feature fusion module. Referring to Figures 10A and 10B, the residual units are mostly "single-layer" connections, while the first long-term connection network is a "cross-layer" or even multi-layer connection; that is, compared to the first analysis network, the first long-term connection network spans at least one neural network layer in the first analysis network. From an information transmission perspective, establishing long-term connections between the input signal and the feature vector achieves better feature extraction results.

[0247] In implementation, the topology of the first long-term connection network is similar to that of the first analysis network. However, to control computational complexity, the internal structure of the first long-term connection network is simplified, containing only simple operations such as causal convolution and pooling. Furthermore, the input and output dimensions of each neural network layer in the first long-term connection network are referenced from the corresponding neural network layer in the first analysis network, and are consistent with the input and output dimensions of the corresponding neural network layer in the first analysis network. Therefore, in terms of results, the properties of the first long-term connection feature vector output by the first long-term connection network are consistent with the properties of the first feature vector output by the first analysis network, and the feature dimensions of the first long-term connection feature vector output by the first long-term connection network are consistent with the feature dimensions of the first feature vector output by the first analysis network, making it suitable for subsequent feature fusion operations.

[0248] Referring to Figure 11A, which is a first schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application, the following describes the feature extraction process of the first long-term connection network, referring to the network structure diagram of the first long-term connection network shown in Figure 11A (the first long-term connection network includes multiple cascaded neural network layers):

[0249] 1) First, a 16-channel causal convolution (i.e., the first neural network layer) is called, which can generate a 16×320 tensor from the input tensor (i.e. the feature vector, corresponding to the 1×320 feature vector x(n) in the figure).

[0250] 2) Next, the 16×320 tensor is preprocessed through the first neural network layer (i.e., the first convolution-pooling layer in the first long-term connected network shown in Figure 11A, which is the second neural network layer). For example, after a convolution operation through the convolution-pooling layer, a pooling operation with a down factor of 2 is performed, and the activation function can be PReLU, to generate a 16×160 tensor.

[0251] 3) Similarly, the second to fifth convolutional-pooling layers (i.e., the third to sixth neural network layers) correspond to the four different downsampling factors of the encoding blocks in the first analysis network. For example, the second convolutional-pooling layer corresponds to the first encoding block in the first analysis network, and the third convolutional-pooling layer corresponds to the second encoding block in the first analysis network. The down_factors of the above four convolutional-pooling layers are set to 2, 4, 4, and 5, respectively, and the number of output channels are set to 32, 64, 128, and 256, respectively. Specifically, the second to fifth convolutional-pooling layers use operations similar to those of the first convolutional-pooling layer. Obviously, the calculation of the convolutional-pooling layers is greatly simplified compared to the encoding blocks in the first analysis network.

[0252] 4) Finally, the 256×1 tensor is passed through a causal convolution (i.e., the last neural network layer) and activated by Tahn, which outputs a 12-dimensional first long-term connection feature vector. Therefore, the first long-term connection feature vector has completely identical attributes and feature dimensions to the aforementioned first feature vector. In this embodiment, the number of causal convolutions used is not limited; two or more causal convolutions can be used. For example, the 256×1 tensor can be passed through a causal convolution to output a 128×1 tensor; then, the 128×1 tensor can be passed through another causal convolution to output a 12-dimensional feature vector (i.e., F). LoRa (n)). In this way, with only a small increase in computational complexity, feature vector extraction can be performed more effectively.

[0253] Step 13: Feature fusion.

[0254] Obtain the first feature vector F1(n) and the first long-term connection feature vector F LoRa After (n), a feature fusion operation is performed. For example, the first feature vector output from the first analysis network is added to the first long-term connection feature vector output from the first long-term connection network along the corresponding dimension, and then tanh activation is performed. In this way, the final feature vector F(n) is output, and each value in the final feature vector F(n) is between [-1, 1].

[0255] Step 14: Quantization encoding.

[0256] For the feature vector F(n), scalar quantization (quantizing each component individually) and entropy coding methods can be used. Furthermore, the embodiments of this application do not limit the combination of vector quantization (combining multiple adjacent components into a single vector for joint quantization) and entropy coding techniques. In addition, besides entropy coding, direct coding can also be used.

[0257] This section focuses on the implementation description of the feature vector F(n). As described above, the 320-point input x(n) of each frame, after passing through the first analysis network, the first long-term connection network, and feature fusion processing, yields a 12-dimensional feature vector. Therefore, for narrowband ultra-low bitrate scenarios, the required bitrate is 1kbps or even lower. This application provides a method based on scalar quantization and entropy coding. For example, code table-1 is [-1.0, -0.66, -0.33, 0, 0.25, 0.5, 0.75, 1.0], corresponding to 3 bits; code table-2 is [-1.0, 0, 0.5, 1.0], corresponding to 2 bits. Taking the 3-bit code table-1 as an example, the required bitrate for each frame is 36 bits, and with 25 frames per second, the bitrate is 25 * 36 = 0.9kbps. Similarly, taking the 2-bit code table-2 as an example, the bitrate is 0.6kbps. Therefore, for the selected actual bit rate, 2-bit or 3-bit encoding can be used directly to complete the quantization and direct encoding of the feature vector.

[0258] In some embodiments, for each frame of the bitstream, 1 bit is first used to represent the bitrate used for encoding (corresponding to the codebook used for encoding), and then the corresponding index value obtained by scalar quantization and direct encoding of the 12-dimensional feature vector is written into the bitstream.

[0259] The decoding process is as follows:

[0260] Step 21, Decode.

[0261] Decoding is the reverse process of encoding. The received bitstream is parsed; the first bit is parsed to obtain the encoding bitrate, and the corresponding code table (two code tables in total) is selected. Then, based on the bitstream, 12 index values ​​are parsed out, and by looking up the quantization table, the estimated value F of the feature vector can be obtained.′ (n).

[0262] The following is a detailed description using a configuration with an encoding rate of 0.9kbps as an example:

[0263] 1) Parse the frame header. The encoding mode bit is 0, which indicates that the bandwidth uses code table -1.

[0264] 2) The bitstream is parsed in 12 steps using direct decoding technology. By using code table-1, the quantized value F of the 12-dimensional feature vector can be obtained. ′ (n).

[0265] Step 22: Invoke the second long-term connection network.

[0266] The purpose of the second long-term connection network is to quantize the feature vector values ​​F ′ L(n) is called to generate a second long-term feature vector L(n) by calling a deep neural network. This vector is then used in conjunction with the first reconstruction network to reconstruct the speech signal.

[0267] Referring to Figure 11B, the network structure of the second long-time connection network is highly similar to that of the first long-time connection network. The specific process is as follows:

[0268] 1) Calling a causal convolution can expand the input tensor from 12×1 to 256×1.

[0269] 2) Next, four convolutional-repeat layers with different upsampling factors (Up_factor) are cascaded. Each convolutional-repeat layer contains one convolutional layer and one upsampling layer. One convolutional layer halves the number of input channels; the upsampling layer contains a specific Up_factor to perform upsampling. The Up_factors of the four convolutional-repeat layers are set to 5, 4, 4, and 2, respectively. Therefore, the number of output channels of the four convolutional-repeat layers are set to 128, 64, 32, and 16, respectively. After processing by the four convolutional-repeat layers, the 256×1 tensor is transformed into tensors of 128×5, 64×20, 32×80, and 16×160, respectively. Note that the number of convolutional-repeat layers in this embodiment is not limited and can be any positive integer such as 2, 3, 4, or 5.

[0270] Here, for upsampling layers containing a specific Up_factor, a repeat operation can be used to complete the upsampling operation by repeatedly padding. This saves complexity.

[0271] 3) Post-process the 16×160 tensor output by the cascaded convolution-Repeat layer. For example, perform a factor of 2 Repeat operation on the 16×160 tensor output by the cascaded convolution-Repeat layer to complete upsampling, then perform convolution operation and use the PReLU activation function to generate a 16×320 tensor.

[0272] Other details of the second long-time connection network are the reverse process of the first long-time connection network, and will not be repeated here.

[0273] Step 23: Invoke the first reconstruction network.

[0274] The purpose of the first reconstruction network is to quantize the feature vector values ​​F ′ (n), combine the output L(n) from the second long-term connection network, call the deep neural network, and generate the estimated value of the signal x′(n).

[0275] Referring to the network structure diagram of the first reconstruction network shown in Figure 11C, the network structure of the first reconstruction network is highly similar to that of the first analysis network, for example, causal convolution. The post-processing structure in the first reconstruction network is similar to the pre-processing in the first analysis network. The decoding block structure is symmetrical to the encoding block on the encoding side. The encoding block on the encoding side first undergoes dilated convolution followed by pooling to complete downsampling, while the decoding block on the decoding side first undergoes pooling to complete upsampling, followed by dilated convolution. The specific process is as follows:

[0276] 1) Calling a causal convolution can expand the input tensor from 12×1 to 256×1.

[0277] 2) Next, four decoding blocks with different upsampling factors (Up_factor) are concatenated. Each decoding block contains a convolutional layer, an upsampling layer, and a residual block. The convolutional layer halves the number of input channels; the upsampling layer contains a specific Up_factor to perform upsampling; and the residual block contains five dilated convolution-based residual units. The Up_factors of the four decoding blocks are set to 5, 4, 4, and 2, respectively. Therefore, the number of output channels for the four decoding blocks are set to 128, 64, 32, and 16, respectively. After processing by the four decoding blocks, the 256×1 tensor is converted into tensors of 128×5, 64×20, 32×80, and 16×160, respectively. Note that the number of decoding blocks in this embodiment is not limited and can be any positive integer such as 2, 3, 4, or 5.

[0278] Here, for upsampling modules that contain a specific Up_factor, a repeat operation can be used to complete the upsampling operation by repeatedly filling in the data. This saves complexity.

[0279] Here, the configuration of the five dilated convolution-based residual units at the decoding end is similar to that at the encoding end, including but not limited to the internal structure of the residual units, convolution kernel size, and dilation rate. The number of groups used in the dilated convolution in the decoding block is configured as shown in Table 2. Here, a higher group size of 2 is used in the decoding block to associate more input and output channels, thereby improving the quality of speech reconstruction.

[0280] Table 2. Configuration of the number of groups used by residual units in different decoding blocks

[0281] 3) Post-process the 16×160 tensor output by the concatenated decoder block. For example, perform a factor-2 Repeat operation on the 16×160 tensor output by the concatenated decoder block to complete upsampling, then perform convolution operation and use the PReLU activation function to generate a 16×320 tensor.

[0282] 4) The 16×320 tensor output from the second long-term connected network is added dimension-by-dimensionally to the 16×320 tensor obtained after post-processing to obtain a new 16×320 tensor through feature fusion. A causal convolution is then applied to convert the input 16×320 tensor into a 1×320 tensor, thus completing signal generation. However, this embodiment is not limited to the feature fusion method. For example, the 16×320 tensor output from the second long-term connected network can also be added dimension-by-dimensionally to the 16×320 tensor obtained after post-processing, and then an activation function (such as PReLU) can be applied to obtain a new 16×320 tensor.

[0283] Other details of the first reconstruction network are the reverse process of the first analysis network mentioned above, and will not be repeated here.

[0284] The speech encoding and decoding method provided in the embodiments of this application is described below with reference to Figure 7B:

[0285] For the encoding end, the processing performed by the encoding end is similar to that shown in Figure 7A, that is, the processing of steps 11-14 shown in Figure 7B is similar to that shown in Figure 7A.

[0286] For the decoding end, most of the processing performed by the decoding end is similar to the processing of the decoding end shown in Figure 7A, that is, the processing of steps 21-22 shown in Figure 7B is similar to the processing of steps 21-22 shown in Figure 7A.

[0287] The processing of the first reconstruction network in step 24 shown in Figure 7B is similar to that in step 23 shown in Figure 7A. However, the first reconstruction network in step 24 of Figure 7B does not need to fuse the output of the second long-time connection network. That is, F′(n) is transformed into the final output S(n) of the first reconstruction network through the first reconstruction network shown in Figure 13C. The specific feature fusion process is implemented through step 25 shown in Figure 7B, which fuses the final output S(n) of the first reconstruction network with L(n) output in step 22 to generate the reconstructed signal estimate x′(n) (i.e., the reconstructed speech signal, also known as the reconstructed audio signal).

[0288] In some other embodiments, see Figure 12A, which is a second schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in the embodiments of this application. The network structure shown in Figure 12A will be described in detail below.

[0289] In the preceding embodiment (corresponding to step 12 above), the principle and implementation of the first long-time connection network were introduced; the characteristics of the so-called long-time connection network were explained by comparing it with a general residual unit. Here, another feasible implementation method is given.

[0290] As shown in Figure 12A, the dimensions of the input and output tensors of the first long-term connection network in this embodiment are completely consistent with those of the first analysis network. However, the internal structure of the first long-term connection network in Figure 12A has been further simplified (the simplification method can be found in the differences between Figure 12A and Figure 11A). For example, by modifying the number of convolutional-pooling layers in the first long-term connection network and matching the corresponding down_factor, it is ensured that a 1×320 tensor is mapped to a 12×1 tensor. The structure of the first long-term connection network in Figure 12A is more streamlined than that of the first analysis network, resulting in lower computational complexity; for example, the number of channels is at most 128 instead of 256.

[0291] Similarly, referring to Figure 12B, which is a second schematic diagram of an optional structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application, the second long-term connection network in the decoding end has also been further simplified, mapping the input 12×1 tensor to a 16×320 tensor. In this process, the maximum number of channels of the second long-term connection network shown in Figure 12B is also 128 instead of 256. Then, through feature fusion, the output of the second long-term connection network is fused with the intermediate nodes of the first reconstruction network. For example, after adding the post-processed output of the first reconstruction network to the corresponding dimension of the second long-term connection feature vector, PReLU activation is performed to generate a new 16×320 tensor; finally, a causal convolution is called to convert the newly generated 16×320 tensor into a 1×320 tensor to complete the signal generation.

[0292] Therefore, the embodiments of this application do not mandate that the configuration parameters of various first long-term connection networks (such as the number of neural network layers, the structure of neural network layers, the Down_factor parameter, etc.) be consistent with the first analysis network, nor do they mandate that the configuration parameters of the second long-term connection network be consistent with the first reconstruction network.

[0293] For details on the specific implementation of the decoding end, please refer to the descriptions of steps 21 to 23 above, which will not be repeated here.

[0294] In other embodiments, the data structures (such as tensor dimensions) for the inputs and outputs of the first long-term connection network and the second long-term connection network can be more flexible. Referring to Figures 13A and 13B, Figure 13A is a third schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in an embodiment of this application, and Figure 13B is a fourth schematic diagram of an optional structure of the first analysis network and the first long-term connection network provided in an embodiment of this application. The network structures shown in Figures 13A and 13B will be described in detail below.

[0295] As shown in Figure 13A, the input to the first long-term connection network can be the output of the first causal convolution in the first analysis network, corresponding to a dimension of 16×320. The output of the first long-term connection network is a 12×1 dimensional tensor. Therefore, the output of the first long-term connection network is fused with the final output of the first analysis network. In this configuration, the first long-term connection network can correspond one-to-one with the structures in the first analysis network except for the first causal convolution. Apart from this, the other implementation methods and details are consistent with step 12 above, and will not be repeated here.

[0296] In principle, the first causal convolution of the first analysis network maps the original input to a 16-channel representation, which is equivalent to describing the original signal using multiple neural network channels. Therefore, it is feasible to obtain another form of the first long-term feature vector by using the 16×320 representation as a starting point for the first long-term connection network analysis. This application does not limit other methods of obtaining the input to the first long-term connection network. For example, the preprocessed 16×160 tensor of the first analysis network in Figure 9 can be used as the input to the first long-term connection network; correspondingly, the first long-term connection network reduces one convolution-pooling operation. The principle is similar and will not be described in detail further.

[0297] As shown in Figure 13B, the input of the first long-term connection network is completely identical to that of the first analysis network, i.e., the corresponding dimension is 1×320. However, the output dimension of the first long-term connection network is 128×5. This is achieved through feature fusion with the output of the third encoding block in the first analysis network, followed by further operations in the first analysis network to obtain a 12×1 dimension feature vector. Therefore, it can be shown that the first long-term connection network can perform feature fusion as long as the dimensions of its input and output tensors match those of the neural network layers in the first analysis network. Furthermore, referring to the above-described encoding-side embodiment, this application does not limit the internal structure of the first long-term connection network. Generally, because the first long-term connection network adopts a more streamlined network structure, its computational complexity is lower than that of the first analysis network.

[0298] Similarly, referring to Figure 13C, which is a third schematic diagram of an optional structure of the first reconstruction network and the second long-time connection network provided in the embodiments of this application, the second long-time connection network in the decoding end has also been further simplified, mapping the input 12×1 tensor to a 1×320 tensor. In this process, the input of the second long-time connection network shown in Figure 13C is the decoded 12×1-dimensional feature vector, and the output is a 1×320-dimensional tensor. Therefore, the second long-time connection network shown in Figure 13C generates the final reconstructed speech signal after feature fusion with the final output of the first reconstruction network.

[0299] In some other embodiments, see Figure 13D, which is a fourth schematic diagram of an optional structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application. The network structure shown in Figure 13D will be described in detail below.

[0300] As shown in Figure 13D, the input to the second long-term connection network can be the output of the first causal convolution in the first reconstruction network, corresponding to a dimension of 256×1. The connection method between the second long-term connection network and the first reconstruction network has been changed (see the difference between Figure 11B and Figure 13D for this change). For example, after the second long-term connection network performs three convolution-repeat layers, it transforms the 256×1 tensor into a 64×20 tensor. Then, it reuses the intermediate nodes of the first reconstruction network (not the final output), namely the fourth decoding block (Up_factor=2) and the post-processing layer in the first reconstruction network, to generate a 16×160 first tensor. This 16×160 first tensor is then input into the causal convolution layer of the second long-term connection network, thus obtaining the final output of the second long-term connection network (a 16×160 second tensor). The first reconstruction network quantizes the feature vector F... ′ The tensor (n) (i.e., a 12×1 tensor) is transformed into a 1×320 tensor and then combined with the final output L(n) (i.e., a 16×160 second tensor) from the second long-term connected network to generate an estimate of the signal x′(n).

[0301] Therefore, this embodiment of the application implements the input of the output of the intermediate node of the second long-time connection network (the 32×80 tensor output of the convolution-pooling layer with Up_factor=4 in Figure 13D) to the intermediate node of the first reconstruction network (the fourth coding block in Figure 13D), and the input of the output of the intermediate node of the first reconstruction network (the 16×320 tensor output of the post-processing layer in Figure 13D) to the intermediate node of the second long-time connection network (the causal convolutional layer in Figure 13D), thereby generating the estimated value of the signal x′(n). The specific implementation of the decoding end can be referred to in the description of steps 21 to 23 above, and will not be repeated here.

[0302] In some other embodiments, see Figure 13E, which is a fifth schematic diagram of an optional structure of the first reconstruction network and the second long-term connection network provided in the embodiments of this application. The network structure shown in Figure 13E will be described in detail below.

[0303] As shown in Figure 13E, the input to the second long-term connection network can be the output of the first causal convolution in the first reconstruction network, corresponding to a dimension of 256×1. The connection method between the second long-term connection network and the first reconstruction network has been changed (see the difference between Figure 11B and Figure 13E for this change). For example, after the second long-term connection network performs four convolution-repeat layers, it transforms the 256×1 tensor into a 16×160 tensor. This tensor is then fused with the output of the fourth decoding block (Up_factor=2) of the intermediate node (not the final output) of the first reconstruction network to generate a new 16×160 tensor. After this, it undergoes post-processing of the original first reconstruction network to generate a 16×320 tensor. Following this, the first reconstruction network performs further processing based on causal convolution layers. The remaining operations yield a 1×320 tensor. The first long-term connection network generates a 16×320 tensor as the input to the last causal convolutional layer, converting the 16×320 tensor into a 1×320 tensor to obtain the final output L(n) of the second long-term connection network (i.e., (a 16×320 tensor)). Finally, a second feature fusion is performed based on the final output L(n) of the second long-term connection network and the 1×320 tensor output by the first reconstruction network to obtain the estimated value x′(n) of the signal.

[0304] Therefore, this embodiment of the application implements feature fusion of the output of the intermediate node of the second long-time connection network with the intermediate node of the first reconstruction network, and then performs the remaining operations of the first reconstruction network and the second long-time connection network, as well as a second feature fusion, based on the fused features to obtain the estimated value x′(n) of the signal. The specific implementation of the decoding end can be referred to the description of steps 21 to 23 above, and will not be repeated here.

[0305] As can be seen from the above, the connection between the first reconstruction network (backbone network) and the second long-term connection network (secondary network) can exhibit high flexibility and diversity. Through interaction and fusion at different levels, the feature reconstruction capability of the model is significantly improved. The connection method can be optimized according to specific tasks and data, thereby increasing the flexibility of the model.

[0306] The connection methods between the backbone network and the secondary network include, but are not limited to, the following:

[0307] 1) Independent extraction and fusion: The backbone network and the secondary network independently reconstruct features, and then these features are fused in the final stage of the network to achieve more comprehensive information utilization (as shown in Figure 13C).

[0308] 2) Intermediate Node Connections: The outputs of intermediate nodes in the backbone network are used as inputs to some neural network layers in the secondary network, and the outputs of intermediate nodes in the secondary network can also be used as inputs to some neural network layers in the backbone network (as shown in Figure 13D). There is no limit to the number of times the intermediate nodes of the backbone and secondary networks can be used for input and output, meaning that the secondary network can reuse some neural network layers of the backbone network. After the backbone and secondary networks reconstruct their features, the reconstructed features are fused to obtain fused features. This feature fusion can be performed at the final stage of the network (as shown in Figure 13C), or it can be achieved by fusing the outputs of the secondary network with the outputs of the intermediate nodes of the backbone network (as shown in Figure 13B) to enhance the expressive power of the features.

[0309] 3) Layered fusion: The output of the neural network layer of the secondary network is fused with the output of the intermediate node of the backbone network at least once to achieve in-depth feature mining and integration. For example, the output of the neural network layer of the secondary network is fused with the output of the intermediate node of the backbone network once, and a second feature fusion is performed at the last output of the backbone network (as shown in the connection method in Figure 13E); the output of the neural network layer of the secondary network is fused with the output of the intermediate node of the backbone network twice.

[0310] Therefore, as can be seen from the above embodiments, the second long-term connection network can be fused as long as the dimensions of the input and output tensors match the neural network layers of the first reconstruction network. Furthermore, referring to the above-described decoding-side embodiments, this application does not limit the internal structure of the second long-term connection network. Generally, because a more streamlined network structure is adopted, the computational complexity of the second long-term connection network is less than that of the first reconstruction network.

[0311] This application embodiment can obtain optimal parameters by jointly training the relevant networks of the encoder and decoder through data collection. Users only need to prepare the data and set the corresponding network structure. After training is completed in the background, the trained model can be put into use.

[0312] Therefore, through the process described in the above embodiments, a low bit-rate speech encoding and decoding scheme based on signal processing and deep learning networks can be completed.

[0313] In summary, this application's embodiments introduce a long-term connection analysis network at the encoding end to generate a more complex encoding network. This more complex encoding network ensures that the feature vectors obtained through complex network analysis retain the most authentic information of the input signal at a very low bit rate, thereby improving encoding performance. Then, by introducing a long-term connection analysis network into the decoding network, higher-quality reconstruction is obtained, reconstructing the most authentic information of the input signal. By introducing a long-term connection analysis network, the end-to-end neural network encoding and decoding technology is further improved, enhancing audio decoding quality while maintaining acceptable neural network model complexity.

[0314] The following continues to describe an exemplary structure of the audio encoding device 555 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG3A, the software module stored in the audio encoding device 555 in the memory 550 may include:

[0315] The feature extraction module 5551 is used to extract features from the audio signal to obtain the audio features of the audio signal;

[0316] The signal encoding module 5552 is used to encode the audio features to obtain the audio bitstream of the audio signal, wherein the audio bitstream is used for decoding based on the audio decoding method provided in the embodiments of this application.

[0317] In some embodiments, the feature extraction is implemented through a feature extraction network, which includes multiple cascaded fourth neural network layers; the feature extraction module 5551 is further configured to extract features from the input of the t-th fourth neural network layer through the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer, where t is a positive integer that increases sequentially, 0 < t ≤ T, T is the number of the fourth neural network layers in the feature extraction network, and the input of the first fourth neural network layer is the audio signal; the feature extraction result of the t-th fourth neural network layer is determined as the audio feature.

[0318] In some embodiments, the feature extraction module 5551 is further configured to perform one of the following processes through the t-th fourth neural network layer of the feature extraction network: performing convolution processing on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer; performing preprocessing on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer; or performing encoding processing on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer.

[0319] In some embodiments, the feature extraction module 5551 is further configured to perform residual processing on the input of the t-th fourth neural network layer through the residual block included in the t-th fourth neural network layer to obtain residual features; perform convolution processing on the residual features through the convolutional layer included in the t-th fourth neural network layer to obtain convolutional features; and perform downsampling on the convolutional features through the downsampling layer included in the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer.

[0320] The following continues to describe an exemplary structure of the audio decoding device 655 provided in this application embodiment as a software module. In some embodiments, as shown in FIG3B, the software module stored in the audio decoding device 655 in the memory 650 may include:

[0321] The signal decoding module 6551 is used to perform signal decoding on the audio bitstream to obtain the audio feature estimation value corresponding to the audio bitstream;

[0322] The first reconstruction module 6552 is used to perform a first reconstruction on the audio feature estimate through the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream.

[0323] The second reconstruction module 6553 is used to perform a second reconstruction on the audio feature estimate through a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream;

[0324] Wherein, the number of convolutional layers in the backbone network is greater than the number of convolutional layers in the secondary network, and the dimension of the first reconstructed signal is the same as the dimension of the second reconstructed signal;

[0325] The fusion module 6554 is used to fuse the first reconstructed signal and the second reconstructed signal to obtain the reconstructed audio signal corresponding to the audio bitstream.

[0326] In some embodiments, the backbone network includes multiple cascaded first neural network layers; the first reconstruction module 6552 is further configured to perform feature reconstruction on the input of the nth first neural network layer through the nth first neural network layer of the backbone network to obtain the feature reconstruction result of the nth first neural network layer, where n is a positive integer that increases sequentially, 0 < n ≤ N, N is the number of the first neural network layers in the backbone network, and the input of the first first neural network layer is the audio feature estimate; the feature reconstruction result of the Nth first neural network layer is determined as the first reconstructed signal.

[0327] In some embodiments, the first reconstruction module 6552 is further configured to perform one of the following processes through the nth first neural network layer of the backbone network: perform convolution processing on the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer; decode the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer; and perform post-processing on the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer.

[0328] In some embodiments, the first reconstruction module 6552 is further configured to perform deconvolution processing on the input of the nth first neural network layer through the deconvolution layer included in the nth first neural network layer to obtain deconvolution features; perform upsampling on the deconvolution features through the upsampling layer included in the nth first neural network layer to obtain upsampled features; and perform feature residual processing on the upsampled features through the residual block included in the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer.

[0329] In some embodiments, the residual block includes multiple cascaded residual units; the first reconstruction module 6552 is further configured to perform feature residual processing on the upsampled features through the first residual unit included in the nth first neural network layer; output the residual result output by the first residual unit to the subsequent cascaded residual units, and continue to perform feature residual processing and output residual results through the subsequent cascaded residual units; and use the residual result output by the last residual unit as the feature reconstruction result of the nth first neural network layer.

[0330] In some embodiments, the first reconstruction module 6552 is further configured to perform the following processing through the first residual unit of the plurality of cascaded residual units: convolving the upsampled features to obtain the convolution result of the first residual unit; adding the convolution result of the first residual unit to the residual features to obtain the residual result output by the first residual unit; and performing the following processing through the j-th residual unit of the plurality of cascaded residual units: convolving the residual result output by the (j-1)-th residual unit to obtain the convolution result of the j-th residual unit; adding the convolution result of the j-th residual units to the residual result output by the (j-1)-th residual unit to obtain the residual result output by the j-th residual unit; and outputting the residual result output by the j-th residual unit to the (j+1)-th residual unit; wherein j is a positive integer that increases sequentially, 1 < j < J, and J is the number of residual units.

[0331] In some embodiments, the secondary network includes multiple cascaded second neural network layers; the second reconstruction module 6553 is further configured to obtain the feature reconstruction result of the g-th first neural network layer in the backbone network, wherein the feature reconstruction result of the g-th first neural network layer is obtained based on the audio feature estimation value, 0 < g < N, and g is a positive integer; through the k-th second neural network layer of the secondary network, feature reconstruction is performed on the input of the k-th second neural network layer to obtain the feature reconstruction result of the k-th second neural network layer, wherein k is a positive integer that increases sequentially, 0 < k ≤ K, K is the number of layers of the second neural network layer in the secondary network, and the input of the first second neural network layer is the feature reconstruction result of the g-th first neural network layer; the feature reconstruction result of the k-th second neural network layer is determined as the second reconstructed signal.

[0332] In some embodiments, the second reconstruction module 6553 is further configured to perform one of the following processes through the k-th second neural network layer of the secondary network: perform deconvolution processing on the input of the k-th second neural network layer to obtain the feature reconstruction result of the k-th second neural network layer; or decode the input of the k-th second neural network layer to obtain the feature reconstruction result of the k-th second neural network layer.

[0333] In some embodiments, the second reconstruction module 6553 is further configured to perform at least one deconvolution process on the input of the kth second neural network layer through at least one deconvolution layer included in the kth second neural network layer to obtain deconvolution features; and to upsample the convolution features through an upsampling layer included in the kth second neural network layer to obtain the feature reconstruction result of the kth second neural network layer.

[0334] In some embodiments, the secondary network includes multiple cascaded third neural network layers; the second reconstruction module 6553 is further configured to perform feature reconstruction on the input of the q-th third neural network layer through the q-th third neural network layer of the secondary network to obtain the feature reconstruction result of the q-th third neural network layer, where q is a positive integer that increases sequentially, 0 < q ≤ Q, Q is the number of the third neural network layers in the secondary network, and the input of the first third neural network layer is the audio feature estimate; the feature reconstruction result of the q-th third neural network layer is determined as the second reconstructed signal.

[0335] In some embodiments, the backbone network includes a first subnetwork and a second subnetwork, the first subnetwork being used to perform the first reconstruction; the fusion module 6554 is further used to add the first reconstructed signal and the second reconstructed signal to obtain a third reconstructed signal; and the third reconstructed signal is then reconstructed through the second subnetwork to obtain the reconstructed audio signal.

[0336] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the audio encoding or audio decoding method described in this application.

[0337] This application provides a computer-readable storage medium storing a bitstream formed by computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the audio encoding method or audio decoding method provided in this application is implemented, for example, the audio encoding method shown in FIG4A or the audio decoding method shown in FIG5.

[0338] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions thereon. When the computer-executable instructions or computer program are executed by a processor, the audio encoding method or audio decoding method provided in this application is implemented, for example, the audio encoding method shown in FIG4A or the audio decoding method shown in FIG5.

[0339] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0340] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0341] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0342] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0343] In summary, through the embodiments of this application, since the backbone network contains more convolutional layers, it can reconstruct a deeper first reconstructed signal. Since the secondary network has relatively fewer convolutional layers compared to the backbone network, it can reconstruct a shallower second reconstructed signal, which is convenient for capturing shallow details in the audio signal. By fusing the two signals, both deep and shallow information can be utilized simultaneously, so that the reconstructed audio signal contains richer audio information, so as to better recover the audio signal and thus improve the audio decoding quality.

[0344] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An audio decoding method, applied to an electronic device, the method comprising: The audio bitstream is decoded to obtain the estimated audio features corresponding to the audio bitstream; The first reconstructed signal corresponding to the audio bitstream is obtained by performing a first reconstruction on the estimated audio feature values ​​through the backbone network. The estimated audio features are reconstructed using a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream. Wherein, the number of convolutional layers in the backbone network is greater than the number of convolutional layers in the secondary network, and the dimension of the first reconstructed signal is the same as the dimension of the second reconstructed signal; By fusing the first reconstructed signal and the second reconstructed signal, the reconstructed audio signal corresponding to the audio bitstream is obtained.

2. The method according to claim 1, wherein, The backbone network includes multiple cascaded first neural network layers; The first reconstruction of the audio feature estimates through the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream includes: The input of the nth first neural network layer is used to reconstruct features through the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer, where n is a positive integer that increases sequentially, 0 < n ≤ N, N is the number of the first neural network layers in the backbone network, and the input of the first first neural network layer is the audio feature estimate. The feature reconstruction result of the Nth first neural network layer is determined as the first reconstructed signal.

3. The method according to claim 2, wherein, The step of performing feature reconstruction on the input of the nth first neural network layer through the backbone network to obtain the feature reconstruction result of the nth first neural network layer includes: One of the following processes is performed through the nth first neural network layer of the backbone network: The input of the nth first neural network layer is convolved to obtain the feature reconstruction result of the nth first neural network layer; The input of the nth first neural network layer is decoded to obtain the feature reconstruction result of the nth first neural network layer; The input of the nth first neural network layer is post-processed to obtain the feature reconstruction result of the nth first neural network layer.

4. The method according to claim 3, wherein, Decoding the input of the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer includes: By using the deconvolution layer included in the nth first neural network layer, the input of the nth first neural network layer is deconvolutionally processed to obtain deconvolution features; The deconvolutional features are upsampled through the upsampling layer included in the nth first neural network layer to obtain upsampled features; By using the residual blocks included in the nth first neural network layer, feature residual processing is performed on the upsampled features to obtain the feature reconstruction result of the nth first neural network layer.

5. The method according to claim 4, wherein, The residual block comprises multiple cascaded residual units; The step of performing feature residual processing on the upsampled features through the residual blocks included in the nth first neural network layer to obtain the feature reconstruction result of the nth first neural network layer includes: The upsampled features are processed using the first residual unit included in the nth first neural network layer; The residual result output by the first residual unit is output to the subsequent cascaded residual units, and the feature residual processing and residual result output are continued through the subsequent cascaded residual units; The residual result output by the last residual unit is used as the feature reconstruction result of the nth first neural network layer.

6. The method according to claim 5, wherein, The step of performing feature residual processing on the upsampled features through the first residual unit included in the nth first neural network layer includes: The following processing is performed through the first residual unit of the plurality of cascaded residual units: The upsampled features are convolved to obtain the convolution result of the first residual unit; The convolution result of the first residual unit is added to the residual feature to obtain the residual result output by the first residual unit; The process of further processing feature residuals and outputting residual results through the subsequent cascaded residual units includes: The following processing is performed through the j-th residual unit of the plurality of cascaded residual units: The residual result output by the (j-1)th residual unit is convolved to obtain the convolution result of the j-th residual unit; The convolution results of the j residual units are added to the residual results output by the (j-1)th residual unit to obtain the residual results output by the jth residual unit. The residual result output by the j-th residual unit is output to the (j+1)-th residual unit; Where j is a positive integer that increases sequentially, 1 < j < J, and J is the number of residual units.

7. The method according to any one of claims 2 to 6, wherein, The secondary network includes multiple cascaded second neural network layers; The step of performing a second reconstruction on the estimated audio features through a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream includes: Obtain the feature reconstruction result of the g-th first neural network layer in the backbone network, wherein the feature reconstruction result of the g-th first neural network layer is obtained based on the audio feature estimation value, 0 < g < N, and g is a positive integer; The input of the kth second neural network layer is used to reconstruct features through the kth second neural network layer to obtain the feature reconstruction result of the kth second neural network layer. Here, k is a positive integer that increases sequentially, 0 < k ≤ K, and K is the number of the second neural network layers in the secondary network. The input of the first second neural network layer is the feature reconstruction result of the gth first neural network layer. The feature reconstruction result of the Kth second neural network layer is determined as the second reconstruction signal.

8. The method according to claim 7, wherein, The step of performing feature reconstruction on the input of the kth second neural network layer through the secondary network to obtain the feature reconstruction result of the kth second neural network layer includes: One of the following processes is performed through the k-th second neural network layer of the secondary network: The input of the kth second neural network layer is deconvolutionally processed to obtain the feature reconstruction result of the kth second neural network layer; The input of the kth second neural network layer is decoded to obtain the feature reconstruction result of the kth second neural network layer.

9. The method according to claim 8, wherein, Decoding the input of the k-th second neural network layer to obtain the feature reconstruction result of the k-th second neural network layer includes: By using at least one deconvolution layer included in the kth second neural network layer, the input of the kth second neural network layer is subjected to at least one deconvolution process to obtain deconvolution features; The convolutional features are upsampled using the upsampling layer included in the k-th second neural network layer to obtain the feature reconstruction result of the k-th second neural network layer.

10. The method according to any one of claims 1 to 9, wherein, The secondary network includes multiple cascaded third neural network layers; The step of performing a second reconstruction on the estimated audio features through a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream includes: The input of the qth third neural network layer in the secondary network is used to reconstruct features to obtain the feature reconstruction result of the qth third neural network layer. Here, q is a positive integer that increases sequentially, 0 < q ≤ Q, Q is the number of the third neural network layer in the secondary network, and the input of the first third neural network layer is the audio feature estimate. The feature reconstruction result of the Qth third neural network layer is determined as the second reconstruction signal.

11. The method according to any one of claims 1 to 10, wherein, The backbone network includes a first subnetwork and a second subnetwork, wherein the first subnetwork is used to perform the first reconstruction. The process of fusing the first reconstructed signal and the second reconstructed signal to obtain the reconstructed audio signal corresponding to the audio bitstream includes: The first reconstructed signal and the second reconstructed signal are added together to obtain the third reconstructed signal; The reconstructed audio signal is obtained by performing a third reconstruction on the third reconstructed signal through the second sub-network.

12. An audio encoding method applied to an electronic device, the method comprising: Feature extraction is performed on the audio signal to obtain the audio features of the audio signal; The audio features are encoded to obtain an audio bitstream of the audio signal, wherein the audio bitstream is used for decoding based on the audio decoding method according to any one of claims 1 to 11.

13. The method according to claim 12, wherein, The feature extraction is achieved through a feature extraction network, which includes multiple cascaded fourth neural network layers. The step of extracting features from the audio signal to obtain the audio features of the audio signal includes: By using the t-th fourth neural network layer of the feature extraction network, feature extraction is performed on the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer, where t is a positive integer that increases sequentially, 0 < t ≤ T, T is the number of the fourth neural network layer in the feature extraction network, and the input of the first fourth neural network layer is the audio signal; The feature extraction result of the Tth fourth neural network layer is determined as the audio feature.

14. The method according to claim 13, wherein, The step of extracting features from the input of the t-th fourth neural network layer through the feature extraction network to obtain the feature extraction result of the t-th fourth neural network layer includes: One of the following processes is performed through the t-th fourth neural network layer of the feature extraction network: The input of the t-th fourth neural network layer is convolved to obtain the feature extraction result of the t-th fourth neural network layer; The input of the t-th fourth neural network layer is preprocessed to obtain the feature extraction result of the t-th fourth neural network layer; The input of the t-th fourth neural network layer is encoded to obtain the feature extraction result of the t-th fourth neural network layer.

15. The method according to claim 14, wherein, The process of encoding the input of the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer includes: By using the residual blocks included in the t-th fourth neural network layer, residual processing is performed on the input of the t-th fourth neural network layer to obtain residual features; The residual features are processed by convolution through the convolutional layers included in the t-th fourth neural network layer to obtain convolutional features; The convolutional features are downsampled using the downsampling layer included in the t-th fourth neural network layer to obtain the feature extraction result of the t-th fourth neural network layer.

16. A method for processing an audio bitstream, said audio bitstream being decoded based on the audio decoding method according to any one of claims 1 to 11, or generated by the audio encoding method according to any one of claims 12 to 15.

17. An audio decoding apparatus, the apparatus comprising: The signal decoding module is configured to perform signal decoding on the audio bitstream to obtain the audio feature estimation value corresponding to the audio bitstream; The first reconstruction module is configured to perform a first reconstruction on the audio feature estimate through the backbone network to obtain the first reconstructed signal corresponding to the audio bitstream. The second reconstruction module is configured to perform a second reconstruction on the audio feature estimate through a secondary network to obtain the second reconstructed signal corresponding to the audio bitstream; Wherein, the number of convolutional layers in the backbone network is greater than the number of convolutional layers in the secondary network, and the dimension of the first reconstructed signal is the same as the dimension of the second reconstructed signal; The fusion module is configured to fuse the first reconstructed signal and the second reconstructed signal to obtain the reconstructed audio signal corresponding to the audio bitstream.

18. An electronic device, the electronic device comprising: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the audio decoding method according to any one of claims 1 to 11, or the audio encoding method according to any one of claims 12 to 15.

19. A computer-readable storage medium storing a bitstream formed by computer-executable instructions or a computer program, wherein when the computer-executable instructions or the computer program are executed by a processor, they implement the audio decoding method according to any one of claims 1 to 11, or the audio encoding method according to any one of claims 12 to 15.

20. A computer-readable storage medium storing a computer program or computer-executable instructions, wherein when executed by a processor, the computer program or computer-executable instructions implement the audio decoding method according to any one of claims 1 to 11, or the audio encoding method according to any one of claims 12 to 15.

21. A computer program product comprising computer-executable instructions, which, when executed by a processor, implement the audio decoding method according to any one of claims 1 to 11, or the audio encoding method according to any one of claims 12 to 15.

Images