Audio encoding and decoding system, audio signal processing method and related device

Abstract

Provided in the present application are an audio encoding and decoding system, an audio signal processing method and a related device. The audio encoding and decoding system comprises a first encoding and decoding system and a compensation system. The first encoding and decoding system comprises a first encoding end and a first decoding end. The first encoding end is used for generating, on the basis of an input audio signal, a plurality of bitstreams having different bitrates, and sending the generated bitstreams to the first decoding end. The first decoding end may determine an output audio signal corresponding to a bitstream having one bitrate, and output audio signals corresponding to bitstreams having the other bitrates may be determined by the compensation system and a feature reconstruction unit in the first decoding end. By using the solution, the encoding and decoding of bitstreams having multiple bitrates can be implemented by means of an audio encoding and decoding system.

Description

Audio encoding and decoding systems, audio signal processing methods and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202411550681.9, filed with the China National Intellectual Property Administration on October 31, 2024, entitled "Audio Coding and Decoding System, Audio Signal Processing Method and Related Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of signal processing technology, and more specifically, to an audio encoding / decoding system, an audio signal processing method, and related equipment. Background Technology

[0003] Uncompressed audio generates a large amount of data, which puts enormous pressure on storage and bandwidth. To reduce the amount of audio data, audio signals need to be encoded, allowing more audio to be stored in limited space or enabling efficient transmission under various network conditions.

[0004] Traditional audio codecs primarily rely on digital signal processing techniques and an understanding of the human auditory system to compress and decompress audio. The design of these codecs typically requires in-depth expertise and carefully crafted algorithms to achieve efficient compression while maintaining sound quality.

[0005] In recent years, with the rapid development of artificial intelligence (AI) technology, AI has been widely applied to audio codecs. Compared with traditional codecs, AI-based codecs, through advanced neural network architectures and training methods, can provide higher compression efficiency and better sound quality. However, current AI-based codecs still have certain shortcomings in multi-bitrate encoding. Summary of the Invention

[0006] This application provides an audio encoding / decoding system, an audio signal processing method, and related equipment, which can realize the encoding and decoding of multi-bitrate streams.

[0007] In a first aspect, this application provides an audio encoding and decoding system, including a first encoding and decoding system and a compensation system. The first encoding and decoding system includes a first encoding end and a first decoding end. The first decoding end includes a feature reconstruction unit. The first encoding end is configured to: generate a target bitstream based on an input audio signal and send the target bitstream to the first decoding end. The target bitstream includes a first bitstream or a second bitstream. The first bitstream is the base bitstream of the first encoding and decoding system, and the first bitstream and the second bitstream have different bitrates. When the target bitstream is the first bitstream, the first decoding end is configured to: perform decoding and dequantization operations on the first bitstream to obtain first dequantization feature data, and generate a first output audio signal corresponding to the first bitstream based on the first dequantization feature data. When the target bitstream is the second bitstream, the compensation system is configured to: acquire second dequantization feature data corresponding to the second bitstream and determine decoding feature adjustment data based on the second dequantization feature data. The feature reconstruction unit is configured to obtain a second output audio signal corresponding to the second bitstream based on the decoding feature adjustment data. The feature reconstruction unit is trained based on the transmission process of the first bitstream.

[0008] The first codec system can be a common AI codec system. In other words, the first codec system can be an existing codec system. That is, the above technical solution adds a compensation system to an existing first codec system, and uses the added compensation system to decode the generated second bitstream to obtain the corresponding second output audio signal. Compared with a technical solution that uses two codec systems to generate and decode bitstreams at two different bitrates, the above technical solution reduces the number of neural networks that need to be trained; that is, only an additional compensation system needs to be trained, instead of training a complete codec. Furthermore, compared to deploying two codec systems, this technical solution also reduces the number of models required. This reduces the storage pressure on electronic devices.

[0009] In conjunction with the first aspect, in one possible implementation of the first aspect, the first encoding end includes a feature extraction unit and a first quantization encoding unit; the first decoding end includes a first decoding dequantization unit and the feature reconstruction unit; the feature extraction unit is used to acquire encoded feature data of the input audio signal; the first quantization encoding unit is used to perform quantization and encoding operations on the encoded feature data to obtain the first bitstream; the first decoding dequantization unit is used to perform decoding and dequantization operations on the first bitstream to obtain the first dequantized feature data; the feature reconstruction unit is used to generate the first output audio signal based on the first dequantized feature data, or to generate the second output audio signal based on the decoded feature adjustment data.

[0010] In conjunction with the first aspect, in one possible implementation of the first aspect, the compensation system includes a decoding feature adjustment unit, which is used to acquire the second inverse quantization feature data corresponding to the second bitstream, and determine the decoding feature adjustment data based on the second inverse quantization feature data.

[0011] This technical solution, by introducing a decoding feature adjustment unit to compensate for the quantization result, makes the audio signal output from the decoder closer to the input audio signal. Therefore, compared with technical solutions that use a single codec system to decode two bitrate streams, this solution can obtain a better output audio signal.

[0012] In conjunction with the first aspect, in one possible implementation of the first aspect, the first quantization encoding unit is specifically configured to: perform quantization and encoding operations on the encoded feature data according to the complete codebook of the first quantization encoding unit to obtain the first bitstream, and perform quantization and encoding operations on the encoded feature data according to a portion of the codebook of the first quantization encoding unit to obtain the second bitstream; the first decoding and dequantization unit is specifically configured to: perform decoding and dequantization operations on the first bitstream according to the complete codebook to obtain the first dequantized feature data, and perform decoding and dequantization operations on the second bitstream according to the portion of the codebook to obtain the second dequantized feature data.

[0013] The above technical solution can reuse the first quantization encoding unit and the first decoding and dequantization unit to generate the second bitstream and decode and dequantize the second bitstream. Therefore, the above technical solution can reduce the number of units that need to be deployed, thereby reducing the storage pressure on electronic devices.

[0014] In conjunction with the first aspect, in one possible implementation of the first aspect, the compensation system further includes an encoding feature adjustment unit, which is used to adjust the encoding feature data to obtain encoding feature adjustment data. The first quantization encoding unit is specifically used to: perform quantization and encoding operations on the encoding feature data according to the entire codebook of the first quantization encoding unit to obtain the first bitstream, and perform quantization and encoding operations on the encoding feature adjustment data according to a portion of the codebook of the first quantization encoding unit to obtain the second bitstream. The first decoding and dequantization unit is specifically used to: perform decoding and dequantization operations on the first bitstream according to the entire codebook to obtain the first dequantized feature data, and perform decoding and dequantization operations on the second bitstream according to the portion of the codebook to obtain the second dequantized feature data.

[0015] The above technical solution further adds a coding feature adjustment unit to adjust the feature data extracted by the feature extraction unit. This results in a higher quality output audio signal.

[0016] In conjunction with the first aspect, in one possible implementation of the first aspect, the compensation system further includes a second quantization encoding unit and a second decoding dequantization unit. The second quantization encoding unit is used to perform quantization and encoding operations on the encoded feature data to obtain the second bitstream. The second decoding dequantization unit is used to perform decoding and dequantization operations on the second bitstream to obtain the second dequantized feature data. The decoding feature adjustment unit is specifically used to obtain the second dequantized feature data from the second decoding dequantization unit and determine the decoding feature adjustment data based on the second dequantized feature data.

[0017] The above technical solution also adds a quantization encoding unit and a corresponding decoding and dequantization unit for generating the second bitstream. In other words, the above technical solution uses a dedicated quantization encoding unit to determine the second bitstream and a dedicated decoding and dequantization unit to decode and dequantize the second bitstream. Therefore, compared with the solution that reuses the first quantization encoding unit and the first decoding and dequantization unit, the above technical solution can obtain a second output audio signal of better quality.

[0018] In conjunction with the first aspect, in one possible implementation of the first aspect, the compensation system further includes an encoding feature adjustment unit, a second quantization encoding unit, and a second decoding dequantization unit. The encoding feature adjustment unit is used to adjust the encoding feature data to obtain encoding feature adjustment data, and send the encoding feature adjustment data to the second quantization encoding unit. The second quantization encoding unit is used to perform quantization and encoding operations on the encoding feature adjustment data to obtain the second bitstream. The second decoding dequantization unit is used to perform decoding and dequantization operations on the second bitstream to obtain the second dequantized feature data. Specifically, the decoding feature adjustment unit is used to obtain the second dequantized feature data from the second decoding dequantization unit, and determine the decoding feature adjustment data based on the second dequantized feature data.

[0019] The above technical solution further adds an encoding feature adjustment unit to adjust the feature data extracted by the feature extraction unit, and adds a quantization encoding unit and a corresponding decoding and dequantization unit for generating the second bitstream. In other words, the above technical solution selects the encoding feature data used to generate the second bitstream, uses a dedicated quantization encoding unit to determine the second bitstream, and uses a dedicated decoding and dequantization unit to decode and dequantize the second bitstream. Therefore, the above technical solution can obtain a second output audio signal of better quality.

[0020] In conjunction with the first aspect, in one possible implementation of the first aspect, the decoding feature adjustment unit is a first neural network, which is trained in the following manner: based on the first training samples, a first loss function and a second loss function are determined; the parameters of the first neural network are determined according to the first loss function and the second loss function; wherein the first loss function is a loss function between the first reference feature data and the first reference dequantized feature data, the second loss function is a loss function between the first reference feature data and the first reference decoding feature adjustment data, the first reference feature data is the feature data extracted by the feature extraction unit from the first training samples, the first reference dequantized feature data is determined by the second quantization encoding unit and the second decoding dequantization unit based on the first reference feature data, and the first reference decoding feature adjustment data is determined by the first neural network under training based on the first reference dequantized feature data.

[0021] In conjunction with the first aspect, in one possible implementation of the first aspect, the decoding feature adjustment unit is a second neural network, which is trained in the following manner: based on the second training samples, a third loss function and a fourth loss function are determined; the parameters of the second neural network are determined according to the third loss function and the fourth loss function; wherein, the third loss function is a loss function between the second reference feature data and the second reference dequantized feature data, the fourth loss function is a loss function between the initial dequantized feature data and the second reference decoding feature adjustment data, the second reference feature data is the feature data extracted by the feature extraction unit from the second training samples, the second reference dequantized feature data is determined by the second decoding dequantization unit based on the second reference feature data, the initial dequantized feature data is determined by the first quantization encoding unit and the first decoding dequantization unit based on the second reference feature data, and the second reference decoding feature adjustment data is determined by the second quantization encoding unit, the second decoding dequantization unit, and the second neural network under training based on the second reference feature data.

[0022] In conjunction with the first aspect, in one possible implementation of the first aspect, the decoding feature adjustment unit is a third neural network, which is trained in the following manner: based on the third training samples, a fifth loss function is determined, and the parameters of the third neural network are determined according to the fifth loss function. The fifth loss function is a loss function between the third reference inverse quantization feature data and the third reference decoding feature adjustment data. The third reference inverse quantization feature data is determined by the feature extraction unit, the first quantization encoding unit, and the first decoding inverse quantization unit based on the complete codebook of the first quantization encoding unit and the third training samples. The third reference decoding feature adjustment data is determined by the feature extraction unit, the first quantization encoding unit, the first decoding inverse quantization unit, and the trained third neural network based on a portion of the codebook of the first quantization encoding unit and the third training samples.

[0023] In conjunction with the first aspect, in one possible implementation of the first aspect, the decoding feature adjustment unit is a fourth neural network, which is trained in the following manner: based on the fourth training samples, a sixth loss function is determined, and the parameters of the fourth neural network are determined according to the sixth loss function, wherein the sixth loss function is a loss function between the fourth reference feature data and the fourth reference decoding feature adjustment data, the fourth reference feature data is the feature data extracted by the feature extraction unit from the fourth training samples, and the fourth reference decoding feature adjustment data is determined by the first quantization encoding unit, the first decoding dequantization unit, and the fourth neural network under training based on a partial codebook of the first quantization encoding unit and the fourth reference feature data.

[0024] Optionally, any two of the above training samples can be the same. In other words, all neural networks are trained using the same training samples.

[0025] Optionally, any two of the above training samples can be different. In other words, different neural networks can be trained using different training samples.

[0026] Optionally, some training samples in the above-mentioned multiple training sample distributions can be the same, while other training samples can be different. For example, the first and second training samples are the same; the third and fourth training samples are the same.

[0027] Secondly, this application provides an audio signal processing method, comprising: receiving a target bitstream, the target bitstream being a first bitstream or a second bitstream, the first bitstream being the base bitstream of a first codec system, the first bitstream and the second bitstream having different bitrates, the first codec system including a first encoding end and a first decoding end; when the target bitstream is the first bitstream, performing decoding and dequantization operations on the first bitstream using the first decoding end to obtain first dequantization feature data, and generating a first output audio signal corresponding to the first bitstream based on the first dequantization feature data; when the target bitstream is the second bitstream, using a compensation system to obtain second dequantization feature data corresponding to the second bitstream, and determining decoding feature adjustment data based on the second dequantization feature data; and using a feature reconstruction unit in the first decoding end to obtain a second output audio signal corresponding to the second bitstream based on the decoding feature adjustment data, the feature reconstruction unit being trained based on the transmission process of the first bitstream.

[0028] The first codec system can be a common AI codec system. In other words, the first codec system can be an existing codec system. That is, the above technical solution adds a compensation system to an existing first codec system, and uses the added compensation system to decode the generated second bitstream to obtain the corresponding second output audio signal. Compared with a technical solution that uses two codec systems to generate and decode bitstreams at two different bitrates, the above technical solution reduces the number of neural networks that need to be trained; that is, only an additional compensation system needs to be trained, instead of training a complete codec. Furthermore, compared to deploying two codec systems, this technical solution also reduces the number of models required. This reduces the storage pressure on electronic devices.

[0029] In conjunction with the second aspect, in one possible implementation of the second aspect, the compensation system includes a decoding feature adjustment unit. The method of using the compensation system to obtain the second inverse quantization feature data corresponding to the second bitstream and determining the decoding feature adjustment data based on the second inverse quantization feature data includes: using the decoding feature adjustment unit to obtain the second inverse quantization feature data corresponding to the second bitstream and determining the decoding feature adjustment data based on the second inverse quantization feature data.

[0030] This technical solution, by introducing a decoding feature adjustment unit to compensate for the quantization result, makes the audio signal output from the decoder closer to the input audio signal. Therefore, compared with technical solutions that use a single codec system to decode two bitrate streams, this solution can obtain a better output audio signal.

[0031] In conjunction with the second aspect, in one possible implementation of the second aspect, the first dequantization feature data is obtained by the first decoding and dequantization unit in the first decoding end performing decoding and dequantization operations on the first bitstream based on the entire codebook, and the second dequantization feature data is obtained by the first decoding and dequantization unit performing decoding and dequantization operations on the second bitstream based on a portion of the codebook.

[0032] The above technical solution can reuse the first decoding and dequantization unit in the first decoding end to perform decoding and dequantization operations on the second bitstream. Therefore, the above technical solution can reduce the number of units that need to be deployed, thereby reducing the storage pressure on electronic devices.

[0033] In conjunction with the second aspect, in one possible implementation of the second aspect, the first decoding end includes a first decoding and dequantization unit, the first dequantization feature data being obtained by the first decoding and dequantization unit performing decoding and dequantization operations on the first bitstream; the compensation system further includes a second decoding and dequantization unit, the second dequantization feature data being obtained by the second decoding and dequantization unit performing decoding and dequantization operations on the second bitstream.

[0034] The above technical solution adds a decoding and dequantization unit to the compensation system outside the first encoding / decoding system for decoding and dequantizing the second bitstream. In other words, the above technical solution uses a dedicated decoding and dequantization unit to decode and dequantize the second bitstream. Therefore, compared with the solution that multiplexes the first quantization encoding unit and the first decoding and dequantization unit, the above technical solution can obtain a second output audio signal of better quality.

[0035] Thirdly, this application provides an electronic device that includes units for implementing the first aspect or any possible implementation of the first aspect.

[0036] Fourthly, this application provides an electronic device including a processor configured to be coupled to a memory, read and execute instructions and / or program code in the memory to perform various steps performed by a first encoding / decoding system and a compensation system in the first aspect or any possible implementation of the first aspect, or to perform an audio signal processing method in the second aspect or any possible implementation of the second aspect.

[0037] Fifthly, this application provides a chip system including logic circuitry for coupling with an input / output interface to transmit data through the input / output interface, in order to perform the steps of the first aspect or any possible implementation thereof executed by the first encoding / decoding system and the compensation system, or to perform the audio signal processing method of the second aspect or any possible implementation thereof.

[0038] In a sixth aspect, this application provides a computer-readable storage medium storing program code that, when the program code stored in the computer storage medium is run on an electronic device, causes the electronic device to perform the steps performed by the first encoding / decoding system and the compensation system as in the first aspect or any possible implementation of the first aspect, or to perform the audio signal processing method in the second aspect or any possible implementation of the second aspect.

[0039] In a seventh aspect, this application provides a computer program product comprising: computer program code, which, when executed on an electronic device, causes the electronic device to perform the steps performed by the first encoding / decoding system and the compensation system as in the first aspect or any possible implementation thereof, or to perform the audio signal processing method in the second aspect or any possible implementation thereof.

[0040] Eighthly, this application provides an electronic device including units for implementing the various steps performed by a transmitting end in the first aspect or any possible implementation of the first aspect.

[0041] For example, the electronic device may include a feature extraction unit and a first quantization encoding unit.

[0042] For example, the electronic device may include a feature extraction unit and a first quantization coding unit, and the electronic device may also include at least one of a coding feature adjustment unit and a second quantization coding unit.

[0043] Ninthly, this application provides an electronic device including units for implementing the various steps performed by a receiving end in the first aspect or any possible implementation of the first aspect.

[0044] For example, the electronic device may include a feature reconstruction unit, a decoding feature adjustment unit, and a first decoding inverse quantization unit. The electronic device may further include a second decoding inverse quantization unit.

[0045] In a tenth aspect, this application provides an electronic device including a processor configured to be coupled to a memory, read and execute instructions and / or program code in the memory to perform various steps performed by a transmitter in a first encoding / decoding system and a compensation system in the first aspect or any possible implementation of the first aspect.

[0046] Eleventhly, this application provides an electronic device including a processor for coupling with a memory to read and execute instructions and / or program code in the memory to perform various steps executed by a receiver in a first encoding / decoding system and a compensation system in the first aspect or any possible implementation of the first aspect.

[0047] In a twelfth aspect, this application provides a chip system including logic circuitry for coupling with an input / output interface to transmit data through the input / output interface, thereby performing various steps executed by a transmitter in a first encoding / decoding system and a compensation system in either the first aspect or any possible implementation of the first aspect.

[0048] In a thirteenth aspect, this application provides a chip system including logic circuitry for coupling with an input / output interface to transmit data through the input / output interface, in order to perform various steps performed by a receiver in a first encoding / decoding system and a compensation system in either the first aspect or any possible implementation of the first aspect.

[0049] In a fourteenth aspect, this application provides a computer-readable storage medium storing program code that, when executed on an electronic device, causes the electronic device to perform various steps performed by the transmitting end in the first encoding / decoding system and the compensation system, as in the first aspect or any possible implementation of the first aspect.

[0050] In a fifteenth aspect, this application provides a computer-readable storage medium storing program code that, when executed on an electronic device, causes the electronic device to perform various steps performed by a receiving end in a first encoding / decoding system and a compensation system, as described in the first aspect or any possible implementation of the first aspect.

[0051] In a sixteenth aspect, this application provides a computer program product comprising: computer program code that, when executed on an electronic device, causes the electronic device to perform various steps performed by a transmitting end in a first encoding / decoding system and a compensation system as described in the first aspect or any possible implementation thereof.

[0052] In a seventeenth aspect, this application provides a computer program product comprising: computer program code that, when executed on an electronic device, causes the electronic device to perform various steps performed by a receiving end in a first encoding / decoding system and a compensation system as described in the first aspect or any possible implementation thereof. Attached Figure Description

[0053] Figure 1 is a schematic diagram of the structure of an electronic device;

[0054] Figure 2 is a schematic diagram of the application scenario provided in the embodiments of this application;

[0055] Figure 3 is a schematic diagram of the encoding and decoding process provided in an embodiment of this application;

[0056] Figure 4 is a schematic block diagram of an AI-based audio codec;

[0057] Figure 5 is a schematic structural block diagram of an audio codec system provided according to an embodiment of this application;

[0058] Figure 6 is a schematic structural block diagram of another audio codec system provided according to an embodiment of this application;

[0059] Figure 7 is a schematic structural block diagram of another audio codec system provided according to an embodiment of this application;

[0060] Figure 8 is a schematic structural block diagram of another audio codec system provided according to an embodiment of this application;

[0061] Figure 9 is a schematic diagram of a system architecture for neural network training according to an embodiment of this application;

[0062] Figure 10 is a schematic flowchart of a decoding method provided according to an embodiment of this application;

[0063] Figure 11 is a schematic flowchart of an encoding method provided according to an embodiment of this application;

[0064] Figure 12 is a schematic structural block diagram of an electronic device provided according to an embodiment of this application;

[0065] Figure 13 is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0066] This application provides an audio encoding / decoding system, an audio signal processing method, and related equipment. The audio encoding / decoding system includes a first encoding / decoding system and a compensation system. The first encoding / decoding system includes a first encoding end and a first decoding end. The first encoding end is used to generate multiple bitstreams with different bitrates based on the input audio signal and send the generated bitstreams to the first decoding end. The first decoding end can determine the output audio signal corresponding to a bitstream with one bitrate, while the output audio signals corresponding to other bitstreams can be determined by the compensation system and the feature reconstruction unit in the first decoding end. The above scheme can achieve encoding and decoding of multiple bitstreams with a single audio encoding / decoding system.

[0067] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0068] Figure 1 is a schematic diagram of an electronic device. The electronic device 100 shown in Figure 1 may include a processor 110, and at least one of an external memory interface 120 and an internal memory 121.

[0069] Electronic device 100 may also include a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, etc.

[0070] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0071] Processor 110 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0072] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0073] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and other electronic devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices.

[0074] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0075] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0076] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, internal memory 121, external memory, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0077] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0078] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0079] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0080] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0081] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0082] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0083] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0084] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0085] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0086] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0087] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0088] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0089] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0090] Electronic device 100 can be a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) device, virtual reality (VR) device, vehicle terminal, media server, cloud server, etc.

[0091] The technical solutions provided in this application can be applied to network-based voice services (such as Voice over Internet Protocol (VoIP)), video conferencing, online video streaming, and other scenarios. Figure 2 illustrates the applicable scenarios for this application embodiment using VoIP as an example.

[0092] As shown in Figure 2, user A can use terminal device 210 to communicate with user B via voice. Terminal device 210 can acquire the voice signal from user A, encode the voice signal to obtain a bitstream. The bitstream is sent to terminal device 220 via server 230. Terminal device 220 decodes the received bitstream to obtain an audio signal, and then plays the audio signal through a receiver or headset. Similarly, terminal device 220 can acquire the voice signal from user B, encode the voice signal to obtain a bitstream. The bitstream is sent to terminal device 210 via server 230. Terminal device 210 decodes the received bitstream to obtain an audio signal, and then plays the audio signal through a receiver or headset.

[0093] Terminal device 210 and terminal device 220 can be electronic device 110 as shown in Figure 1.

[0094] Figure 3 is a schematic diagram of an encoding and decoding process. As shown in Figure 3, the audio codec of the transmitting device can encode the input audio signal to obtain a bitstream. The receiving device can receive the bitstream through the transmission link, and then use the audio codec to decode the bitstream to obtain the output audio signal.

[0095] Referring to Figure 2, when user A speaks, terminal device 210 is the transmitting device, and terminal device 220 is the receiving device. The input audio signal is the voice signal of user A collected by terminal device 210 (which may also include background noise). The output audio signal is the sound signal played by terminal device 220. When user B speaks, terminal device 220 is the transmitting device, and terminal device 210 is the receiving device. The input audio signal is the voice signal of user B collected by terminal device 220 (which may also include background noise). The output audio signal is the sound signal played by terminal device 210. That is, in some scenarios, terminal device 210 can act as the transmitting device, while terminal device 220 can act as the receiving device. In other scenarios, terminal device 210 can act as the receiving device, while terminal device 220 can act as the transmitting device. In other words, in a call scenario, either terminal device can act as either a transmitting or receiving device.

[0096] Figure 4 is a schematic block diagram of an AI-based audio codec. As shown in Figure 4, the audio codec 400 includes a feature extraction unit 410, a quantization encoding unit 420, a decoding and dequantization unit 430, and a feature reconstruction unit 440. Among them, at least the feature extraction unit 410 and the feature reconstruction unit 440 are implemented based on neural networks.

[0097] The feature extraction unit 410 can extract features from the input audio signal to obtain input feature data.

[0098] The feature extraction unit 410 can be implemented using a neural network. For example, the feature extraction unit 410 can be a convolutional neural network (CNN), a recurrent neural network (RNN), a transformer neural network, or various activation functions.

[0099] The feature data can be input into the quantization encoding unit 420. The quantization encoding unit 420 performs quantization and encoding operations on the feature data to obtain a bitstream.

[0100] The feature data generated by the feature extraction unit 410 can be in vector form. Vectors can have an infinite number of values. To transmit feature data to the receiver using a finite number of bits, they can be replaced with approximate vectors from a codebook. This process is called quantization. A codebook is a predefined set of vectors used in vector quantization, and each vector is called a codeword. The codebook's role is to map the input signal to a vector in this predefined set, thereby achieving data compression and representation.

[0101] After quantizing the feature data, the quantized feature data can be entropy encoded to obtain the bitstream.

[0102] The decoding and dequantization unit 430 decodes and dequantizes the received bitstream to obtain dequantized feature data. Decoding and dequantization are the inverse processes of quantization and encoding.

[0103] The feature reconstruction unit 440 may include a feature reconstruction subunit 441.

[0104] Feature reconstruction subunit 441 reconstructs the inverse-quantized feature data to obtain the output audio signal. Feature reconstruction subunit 441 can also be implemented using a neural network. For example, feature reconstruction unit 440 can be a CNN, RNN, transformer neural network, various activation functions, etc. Feature reconstruction subunit 441 can adopt a structure that is completely dual to feature extraction unit 410.

[0105] In some embodiments, the feature reconstruction unit may further include a decoding feature adjustment subunit 442. The decoding feature adjustment subunit 442 is used to adjust the inverse quantization feature data. The adjusted inverse quantization feature data is input to the feature reconstruction subunit 441. The feature reconstruction subunit 441 reconstructs the adjusted inverse quantization feature data to obtain an output audio signal.

[0106] The feature extraction unit 410, quantization encoding unit 420, decoding and dequantization unit 430, and feature reconstruction unit 440 can be trained end-to-end. The parameters of the feature extraction unit 410, quantization encoding unit 420, decoding and dequantization unit 430, and feature reconstruction unit 440 can be optimized using a combination of reconstruction and adversarial losses, and are computed by the discriminator. The discriminator can be pre-trained. The discriminator is used to distinguish between the input audio signal and the output audio signal.

[0107] In some embodiments, the feature extraction unit 410 and the quantization encoding unit 420 may also be collectively referred to as an encoder. In other words, the encoder may include the feature extraction unit 410 and the quantization encoding unit 420. Similarly, the decoder may include the decoding and inverse quantization unit 430 and the feature reconstruction unit 440.

[0108] As mentioned above, a terminal device can function as both a transmitter and a receiver. Therefore, a terminal device needs to deploy not only an encoder (i.e., a feature extraction unit and a quantization encoding unit) but also a decoder (i.e., a decoding and dequantization unit and a feature reconstruction unit) so that when the terminal device functions as a receiver, it can decode quantized data from another terminal device.

[0109] Taking terminal devices 210 and 220 as shown in Figure 2 as examples, terminal devices 210 and 220 can each deploy an audio codec. The audio codec deployed in terminal device 210 can be called audio codec A, which includes encoder A and decoder A. Encoder A includes feature extraction unit A, quantization encoding unit A, decoding and dequantization unit A, and feature reconstruction unit A. The audio codec deployed in terminal device 220 can be called audio codec B, which includes encoder B and decoder B. Encoder B includes feature extraction unit B and quantization encoding unit B, and decoder B includes decoding and dequantization unit B and feature reconstruction unit B. Audio codec A and audio codec B are identical. In other words, audio codec A / audio codec B can include feature extraction unit 410, quantization encoding unit 420, decoding and dequantization unit 430, and feature reconstruction unit 440 as shown in Figure 4.

[0110] The audio codec shown in Figure 4 can convert an input audio signal into a fixed-bitrate bitstream. Based on the audio codec shown in Figure 4, this application also provides an audio codec system to realize the transmission of audio signals at multiple bitrates. This audio codec system includes a first codec system and a compensation system. The first codec system includes the audio codec shown in Figure 4, and the compensation system includes at least a decoding feature adjustment unit. The compensation system may also include an encoding feature adjustment unit, and / or a second quantization encoding unit and a second decoding dequantization unit, depending on different implementation methods. The audio codec system provided by the embodiments of this application will be described below with reference to Figures 5-8.

[0111] Figure 5 is a schematic structural block diagram of an audio codec system according to an embodiment of this application. As shown in Figure 5, the transmitting end of the audio codec system includes a feature extraction module 510 and a quantization encoding module 520, and the receiving end of the audio codec system includes a decoding and dequantization module 530 and a feature reconstruction module 540. The feature extraction module 510 includes a feature extraction unit 511, the quantization encoding module 520 includes a first quantization encoding unit 521 and a second quantization encoding unit 522, and the decoding and dequantization module 530 includes a first decoding and dequantization unit 531 and a second decoding and dequantization unit 532. The feature reconstruction module 540 includes a decoding feature adjustment unit 542 and a feature reconstruction unit 541. The feature extraction unit 511 can extract features from the input audio signal to obtain first feature data (which can be called encoded feature data). The first quantization encoding unit 521 can perform quantization and encoding operations on the first feature data to obtain a first bitstream and send it. The bitrate of the first bitstream is a first bitrate. The first decoding and dequantization unit 531 acquires the first bitstream and decodes and dequantizes the first quantized data to obtain first dequantized feature data. The feature reconstruction unit 541 can determine the first output audio signal based on the first inverse quantization feature data.

[0112] The second quantization and encoding unit 522 can perform quantization and encoding operations on the first feature data determined by the feature extraction unit 511 to obtain a second bitstream. The second decoding and dequantization unit 532 can acquire the second bitstream, decode and dequantize it to obtain second dequantized feature data. The decoding feature adjustment unit 542 can determine decoding feature adjustment data based on the second dequantized feature data. The feature reconstruction unit 541 can determine the second output audio signal based on the decoding feature adjustment data.

[0113] The second bit rate is different from the first bit rate. For example, in some embodiments, the second bit rate may be less than the first bit rate.

[0114] In a specific implementation, feature extraction unit 511 can be feature extraction unit 410, first quantization encoding unit 521 can be quantization encoding unit 420, first decoding and dequantization unit 531 can be decoding and dequantization unit 430, and feature reconstruction unit 541 can be feature reconstruction unit 440. For ease of description, in this application, the set of the above-mentioned feature extraction unit 511, first quantization encoding unit 521, first decoding and dequantization unit 531, and feature reconstruction unit 541 (as shown by the dotted line in the figure) is referred to as the initial encoding / decoding system, the first encoding / decoding system, or the first codec. Since the first bitstream is processed by the first encoding / decoding system, the first bitstream can be referred to as the base bitstream of the first encoding / decoding system. The above-mentioned initial encoding / decoding system can be divided into an initial encoding end (also called the first encoding end) and an initial decoding end (also called the first decoding end), wherein the initial encoding end may include feature extraction unit 511 and first quantization encoding unit 521, and the initial decoding end may include first decoding and dequantization unit 531 and feature reconstruction unit 541. The initial codec system described above can be obtained by training a neural network, and the training method of the initial codec system is not limited in the embodiments of this application. It can be seen that the audio codec system shown in Figure 5 adds a second quantization encoding unit 522, a second decoding dequantization unit 532, and a decoding feature adjustment unit 542 to the above initial codec system. This allows for the generation of a second bitstream with a different bitrate when acting as a transmitter, and the decoding of this second bitstream to obtain a second output audio signal when acting as a receiver. That is, the compensation system of the audio codec system shown in Figure 5 includes a second quantization encoding unit 522, a second decoding dequantization unit 532, and a decoding feature adjustment unit 542. Compared with the technical solution of using two codecs to generate two bitstreams with different bitrates, the above technical solution can reduce the number of neural networks that need to be trained, i.e., only one additional decoding feature adjustment unit needs to be trained without training a complete codec. Furthermore, compared with deploying two codecs, the technical solution shown in Figure 5 requires fewer models to be deployed. This can reduce the storage pressure on electronic devices. Because the technical solution shown in Figure 5 introduces a decoding feature adjustment unit to compensate for the second dequantization feature data, the audio signal output by the decoder can be closer to the input audio signal. Therefore, compared with the technical solution of using a single codec to generate two different bitrates, the technical solution shown in Figure 5 can produce a better output audio signal.

[0115] The second quantization coding unit 522 can perform quantization and encoding operations on the first feature data to obtain the second bitstream. Quantization can be implemented by the quantization subunit in the second quantization coding unit 522, and encoding can be implemented by the encoding subunit in the second quantization coding unit 522.

[0116] The second decoding and dequantization unit 532 can decode and dequantize the second bitstream to obtain the second dequantized feature data. The decoding can be implemented by the decoding subunit in the second decoding and dequantization unit 532, and the dequantization can be implemented by the dequantization subunit in the second decoding and dequantization unit 532.

[0117] The quantization and dequantization subunits can be implemented in various ways. For example, they can be scalar-based quantizers / dequantizers, vector quantizers / dequantizers, residual vector quantizers / dequantizers, etc. This application does not limit the types of quantization and dequantization subunits. The codebook in the vector quantizer / dequantizer and residual vector quantizer / dequantizer can be trained using a training dataset. The codebook can be trained by initializing the vectors in the codebook using clustering algorithms (such as K-means or hierarchical clustering), and then updating the vectors in the codebook using the training dataset.

[0118] Encoding subunits / decoding subunits can be implemented using entropy coding.

[0119] The decoding feature adjustment unit 542 can be a neural network. For example, the decoding feature adjustment unit 542 can be a CNN, RNN, transformer neural network, various activation functions, etc.

[0120] The decoding feature adjustment unit 542 can be trained based on a pre-trained initial encoding / decoding system. The training dataset may include multiple training samples. The training process of the decoding feature adjustment unit 542 is described below using training sample a as an example. In some embodiments, if the second quantization encoding unit 522 / second decoding dequantization unit 532 includes quantization / dequantization sub-units that need to be trained (e.g., vector quantizer / dequantizer, residual vector quantizer / dequantizer), then the second quantization encoding unit 522 / second decoding dequantization unit 532 can be trained first, and then the decoding feature adjustment unit 542 can be trained based on the trained second quantization encoding unit 522 / second decoding dequantization unit 532. In some embodiments, if the second quantization encoding unit 522 / second decoding dequantization unit 532 includes quantization / dequantization sub-units that need to be trained, these quantization / dequantization sub-units can also be trained together with the decoding feature adjustment unit 542.

[0121] The training sample a is input into the feature extraction unit 511.

[0122] The feature extraction unit 511 extracts the features of the training sample a to obtain the first feature data a1.

[0123] The second quantization and encoding unit 522 performs quantization and encoding operations on the first feature data a1 to obtain the code stream a2.

[0124] The second decoding and dequantization unit 532 decodes and dequantizes the bit stream a2 to obtain dequantized feature data a3.

[0125] The decoding feature adjustment unit 542 determines the decoding feature adjustment data a4 based on the dequantized feature data a3. It can be understood that the decoding feature adjustment unit 542 here is an initial neural network or a neural network during the training process.

[0126] A loss function L51 can be determined based on the decoded feature adjustment data a4 and the first feature data a1. The loss function L51 can be used to optimize the decoded feature adjustment unit 542, therefore the loss function L51 can also be called the decoded feature adjustment loss function.

[0127] A loss function L52 can be determined based on the inverse quantization feature data a3 and the first feature data a1. The loss function L52 can be used to optimize the second quantization coding unit 522 and the second decoding inverse quantization unit 532; therefore, the loss function L52 can also be called the quantization loss function.

[0128] In some embodiments, the parameters of the neural network can be determined based on loss function L51 and loss function L52 to obtain the decoding feature adjustment unit.

[0129] In other embodiments, in addition to loss functions L51 and L52, a loss function L53 may also be determined. Loss function L53 is the loss function between the output sample a' and the training sample a. The output sample a' is determined by the feature reconstruction unit 541 based on the decoded feature adjustment data a4. In this case, the parameters of the neural network can be determined based on loss functions L51, L52, and L53 to obtain the decoded feature adjustment unit.

[0130] In the above embodiments, the loss function L51 is determined based on the output of the decoding feature adjustment unit and the output of the feature extraction unit 511. In other embodiments, a loss function L51' can also be determined based on the output of the decoding feature adjustment unit and the output of the decoding dequantization unit. In this case, the parameters of the neural network can be determined based on the loss function L51' and the loss function L52 to obtain the decoding feature adjustment unit. Alternatively, the parameters of the neural network can be determined based on the loss function L51', the loss function L52, and the loss function L53 to obtain the decoding feature adjustment unit.

[0131] The following describes how the loss function L51' is determined. The first quantization encoding unit 521 can quantize the first feature data a1 using its entire codebook, and then encode the quantization result to obtain the bitstream a2'. The first decoding and dequantization unit 531 decodes the bitstream a2', and then dequantizes the decoding result using its entire codebook to obtain the dequantized feature bitstream a3'. The loss function L51' is the loss function between the decoded feature adjustment data a4 and the dequantized feature data a3'.

[0132] Figure 6 is a schematic structural block diagram of another audio codec system provided according to an embodiment of this application. As shown in Figure 6, the transmitting end of the audio codec system includes a feature extraction module 610 and a quantization encoding module 620, and the receiving end of the audio codec system includes a decoding and dequantization module 630 and a feature reconstruction module 640. The feature extraction module 610 includes a feature extraction unit 611, the quantization encoding module 620 includes a first quantization encoding unit 621, the decoding and dequantization module 630 includes a first decoding and dequantization unit 631, and the feature reconstruction module 640 includes a decoding feature adjustment unit 642 and a feature reconstruction unit 641.

[0133] As shown in Figure 6, at the transmitting end, the feature extraction unit 611 extracts features from the input audio signal to obtain first feature data. The first quantization encoding unit 621 can quantize the first feature data using the entire codebook, then encodes the quantization result to obtain a first bitstream and sends it. The bit rate of the first bitstream is a first bit rate. At the receiving end, the first decoding and dequantization unit 631 acquires the first bitstream, decodes the first bitstream, and dequantizes the decoding result using the entire codebook to obtain first dequantized feature data. The feature reconstruction unit 641 can determine the first output audio signal based on the first dequantized feature data.

[0134] As shown in Figure 6, the first quantization encoding unit 621 can also use a portion of the codebook to quantize the first feature data determined by the feature extraction unit 611, and then encode the quantization result to obtain a second bitstream. The bitrate of the second bitstream is a second bitrate. The first quantization encoding unit 621 can include multiple codebooks. For example, suppose the first quantization encoding unit 621 can include N codebooks, where N is a positive integer greater than or equal to 2. The first quantization encoding unit 621 can use these N codebooks to quantize the first feature data determined by the feature extraction unit 611. The quantization result using all codebooks can be encoded into a first bitstream with a first bitrate. The first quantization encoding unit 621 can also use a portion of the codebooks from the N codebooks to quantize the first feature data determined by the feature extraction unit 611. For example, the first quantization encoding unit 621 can use M codebooks from the N codebooks to quantize the first feature data. M is a positive integer greater than or equal to 1 and less than N. For example, the M codebooks are the first M codebooks from the N codebooks. The quantization result using a portion of the codebooks can be encoded into a second bitstream with a second bitrate. The first decoding and dequantization unit 631 can decode the second bitstream, and then dequantize the decoding result using M codebooks to obtain second dequantized feature data. The decoding feature adjustment unit 642 determines decoding feature adjustment data based on the second dequantized feature data. The feature reconstruction unit 641 can determine the second output audio signal based on the decoding feature adjustment data.

[0135] Compared to Figure 5, Figure 6 shows a single quantization encoding unit, namely the first quantization encoding unit 621, which implements different quantization functions using different codebooks. Correspondingly, the first decoding and dequantization unit 631 implements the dequantization function using the codebook corresponding to the first quantization encoding unit 621.

[0136] In Figure 6, the feature extraction unit 611 can be the feature extraction unit 410, the first quantization encoding unit 621 can be the quantization encoding unit 420, the first decoding and dequantization unit 631 can be the decoding and dequantization unit 430, and the feature reconstruction unit 641 can be the feature reconstruction unit 440. The set of the aforementioned feature extraction unit 611, first quantization encoding unit 621, first decoding and dequantization unit 631, and feature reconstruction unit 641 (as shown by the dotted line in the figure) can be called the initial encoding / decoding system, the first encoding / decoding system, or the first codec. The aforementioned initial encoding / decoding system can be divided into an initial encoding end (also called the first encoding end) and an initial decoding end (also called the first decoding end). The initial encoding end can include the feature extraction unit 611 and the first quantization encoding unit 621, and the initial decoding end can include the first decoding and dequantization unit 631 and the feature reconstruction unit 641. The initial encoding / decoding system can be obtained by training a neural network. The audio encoding / decoding system shown in Figure 6 adds a decoding feature adjustment unit 642 to the aforementioned initial encoding / decoding system. That is, the compensation system of the audio encoding / decoding system shown in Figure 6 includes the feature adjustment unit 642. The first quantization encoding unit 621 at the transmitting end can also generate a bitstream with a different bitrate using a portion of the codebook. The first decoding and dequantization unit 631 at the receiving end, along with the added decoding feature adjustment unit 642, decodes this bitstream to obtain the output audio signal. Compared to the technical solution of using two codecs to generate two bitstreams with different bitrates, the above technical solution reduces the number of neural networks that need to be trained; that is, only one additional decoding feature adjustment unit needs to be trained, instead of training a complete codec. Furthermore, compared to deploying two codecs, the technical solution shown in Figure 6 requires fewer models. This reduces the storage pressure on electronic devices. Because the technical solution shown in Figure 6 introduces the decoding feature adjustment unit 642 to compensate for the quantization results, the audio signal output by the decoder is closer to the input audio signal. Therefore, compared to the technical solution of using one codec to generate two bitrates, the technical solution shown in Figure 6 yields a better output audio signal.

[0137] Referring to the description in Figure 5, the decoding feature adjustment unit 642 can be trained based on a pre-trained initial codec system. The training dataset can include multiple training samples. The training process of the decoding feature adjustment unit 642 will be described below using training sample b as an example.

[0138] The training sample b is input into the feature extraction unit 611.

[0139] The feature extraction unit 611 extracts the features of the training sample b to obtain the first feature data b1.

[0140] The first quantization encoding unit 621 uses a portion of the codebook to perform quantization and encoding operations on the first feature data b1, obtaining bitstream b2. The portion of the codebook used by the first quantization encoding unit 621 to determine bitstream b2 during the training phase is the same as the codebook used during the inference phase. For example, if the first quantization encoding unit 621 uses the first M codebooks from N codebooks to perform quantization and encoding operations on the first feature data b1 during the training phase, obtaining bitstream b2, then the first quantization encoding unit 621 will also use the first M codebooks from N codebooks to perform quantization and encoding operations on the first feature data of the input audio signal during the inference phase, obtaining bitstream.

[0141] The first decoding and dequantization unit 631 uses a portion of the codebook to decode and dequantize the bitstream b2, obtaining dequantized feature data b3. It can be understood that the portion of the codebook used for dequantization is the same as the portion used for quantization. That is, if the first quantization and encoding unit 621 uses the first M codebooks from the N codebooks to perform quantization and encoding operations on the first feature data b1, obtaining bitstream b2, then the first decoding and dequantization unit 631 also uses the first M codebooks from the N codebooks to dequantize bitstream b2, obtaining dequantized feature data b3.

[0142] The decoding feature adjustment unit 642 determines the decoding feature adjustment data b2 based on the dequantized feature data b3. It can be understood that the decoding feature adjustment unit 642 mentioned here is an initial neural network or a neural network during the training process.

[0143] A loss function L61 can be determined based on the decoded feature adjustment data b4 and the first feature data b1. The loss function L61 can be used to optimize the decoded feature adjustment unit 642, therefore the loss function L61 can also be called the decoded feature adjustment loss function.

[0144] In some embodiments, the parameters of the neural network can be determined based on the loss function L61 to obtain the decoding feature adjustment unit 642.

[0145] In other embodiments, in addition to loss function L61, loss function L62 may also be determined. Loss function L62 is the loss function between output sample b' and training sample b. Output sample b' is determined by feature reconstruction unit 641 based on decoded feature adjustment data b4. In this case, the parameters of the neural network can be determined based on loss function L61 and loss function L62 to obtain the decoded feature adjustment unit 642.

[0146] In the above embodiments, the loss function L61 is determined based on the output of the decoding feature adjustment unit and the output of the feature extraction unit 611. In other embodiments, a loss function L61' can also be determined based on the output of the decoding feature adjustment unit 642 and the output of the first decoding dequantization unit 631. In this case, the parameters of the neural network can be determined based on the loss function L61' to obtain the decoding feature adjustment unit 642. Alternatively, the parameters of the neural network can be determined based on the loss function L61' and the loss function L62 to obtain the decoding feature adjustment unit 642.

[0147] The following describes how the loss function L61' is determined. The first quantization and encoding unit 621 can use its entire codebook to perform quantization and encoding operations on the first feature data b1, obtaining the bitstream b2'. The first decoding and dequantization unit 631 uses its entire codebook to decode and dequantize the quantized data b2', obtaining the dequantized feature data b3'. The loss function L61' is the loss function between the decoded feature adjustment data b4 and the dequantized feature data b3'.

[0148] Figure 7 is a schematic structural block diagram of another audio codec system provided according to an embodiment of this application. Figure 7 is an improvement on the audio codec system shown in Figure 5. As shown in Figure 7, the transmitting end of the audio codec system includes a feature extraction module 710 and a quantization encoding module 720, and the receiving end of the audio codec system includes a decoding and dequantization module 730 and a feature reconstruction module 740. The feature extraction module 710 includes a feature extraction unit 711 and an encoding feature adjustment unit 712. The quantization encoding module 720 includes a first quantization encoding unit 721 and a second quantization encoding unit 722. The decoding and dequantization module 730 includes a first decoding and dequantization unit 731 and a second decoding and dequantization unit 732. The feature reconstruction module 740 includes a decoding feature adjustment unit 742 and a feature reconstruction unit 741. Compared with Figure 5, the feature extraction module 710 in Figure 7 further includes an encoding feature adjustment unit 712, which can determine encoding feature adjustment data based on the first feature data determined by the feature extraction unit 711. The second quantization encoding unit 722 can perform quantization and encoding operations on the encoding feature adjustment data determined by the encoding feature adjustment unit 712 to obtain a second bitstream. The bit rate of the second bitstream is a second bit rate. The second decoding and dequantization unit 732 can acquire the second bit rate, decode and dequantize the second bit rate to obtain second dequantized feature data. The decoding feature adjustment unit 742 can determine the decoding feature adjustment data based on the second dequantized feature data. The feature reconstruction unit 741 can determine the second output audio signal based on the decoded feature adjustment data.

[0149] As mentioned earlier, Figure 7 adds an encoding feature adjustment unit 712 to the feature extraction module at the transmitting end, based on Figure 5. The encoding feature adjustment unit 712 compensates for the output of the feature extraction unit 711, making the audio signal output by the feature reconstruction unit 741 closer to the input audio signal. Therefore, the compensation system of the audio codec system shown in Figure 7 includes the encoding feature adjustment unit 712, the second quantization encoding unit 722, the second decoding dequantization unit 732, and the decoding feature adjustment unit 742. Compared to a technical solution that uses a single codec to generate two different bitrates, the technical solution shown in Figure 7 yields a better output audio signal.

[0150] The implementation of the first quantization encoding unit 721, the second quantization encoding unit 722, the first decoding and dequantization unit 731, and the second decoding and dequantization unit 732 in Figure 7 can be referred to the description of the first quantization encoding unit 521, the second quantization encoding unit 522, the first decoding and dequantization unit 531, and the second decoding and dequantization unit 532 in Figure 5.

[0151] The encoding feature adjustment unit 712 and the decoding feature adjustment unit 742 can be trained based on an already trained initial encoding / decoding system. The training dataset can include multiple training samples.

[0152] The following section uses training sample c as an example to introduce how to train the encoding feature adjustment unit 712 and the decoding feature adjustment unit 742.

[0153] The training sample c is input into the feature extraction unit 711.

[0154] The feature extraction unit 711 extracts the features of the training sample c to obtain the first feature data c1.

[0155] The encoding feature adjustment unit 712 determines the encoding feature adjustment data c1' based on the first feature data c1. It can be understood that the encoding feature adjustment unit 712 mentioned here is an initial neural network or a neural network during the training process.

[0156] The second quantization and encoding unit 722 performs quantization and encoding operations on the encoding feature adjustment data c1' to obtain the bit stream c2, and sends the bit stream c2 to the second decoding and dequantization unit 732 in the decoder.

[0157] The second decoding and dequantization unit 732 decodes and dequantizes the code stream c2 to obtain dequantized feature data c3.

[0158] The decoding feature adjustment unit 742 determines the decoding feature adjustment data c4 based on the dequantized feature data c3. It can be understood that the decoding feature adjustment unit 742 mentioned here is an initial neural network or a neural network during the training process.

[0159] A loss function L71 can be determined based on the decoded feature adjustment data c4 and the first feature data c1. The loss function L71 can be used to optimize the decoded feature adjustment unit 742 and the encoded feature adjustment unit 712; therefore, the loss function L71 can also be called the feature adjustment loss function.

[0160] A loss function L72 can be determined based on the inverse quantization feature data c3 and the encoding feature adjustment data c1'. The loss function L72 can be used to optimize the first decoding inverse quantization unit 731 and the second decoding inverse quantization unit 732, so the loss function L72 can also be called the quantization loss function.

[0161] In some embodiments, the parameters of the neural network can be determined based on loss function L71 and loss function L72 to obtain decoding feature adjustment unit 742 or encoding feature adjustment unit 712.

[0162] In other embodiments, in addition to loss functions L71 and L72, a loss function L73 can also be determined. Loss function L73 is the loss function between the output sample c' and the training sample c. The output sample c' is determined by the feature reconstruction unit 741 based on the decoded feature adjustment data c4. In this case, the parameters of the neural network can be determined based on loss functions L71, L72, and L73 to obtain the encoding feature adjustment unit and the decoding feature adjustment unit.

[0163] In the above embodiments, the loss function L71 is determined based on the output of the decoding feature adjustment unit 742 and the output of the feature extraction unit 711. In other embodiments, a loss function L71' can also be determined based on the output of the decoding feature adjustment unit 742 and the output of the first decoding dequantization unit 731. In this case, the parameters of the neural network can be determined based on the loss function L71' and the loss function L72 to obtain the decoding feature adjustment unit. Alternatively, the parameters of the neural network can be determined based on the loss function L71', the loss function L72, and the loss function L73 to obtain the decoding feature adjustment unit 741 and the encoding feature adjustment unit 712.

[0164] The following describes how the loss function L71' is determined. The first quantization encoding unit 721 can quantize the first feature data c1 using its entire codebook, and then encode the quantization result to obtain the bitstream c2'. The first decoding and dequantization unit 731 decodes the bitstream c2', and then dequantizes the decoding result using its entire codebook to obtain the dequantized feature data c3'. The loss function L71' is the loss function between the decoded feature adjustment data c4 and the dequantized feature data c3'.

[0165] Although Figures 5 and 7 show two quantization coding units and two decoding and dequantization units respectively, the audio and video encoding system provided in this application embodiment does not limit the number of quantization coding units and decoding and dequantization units. That is, an audio and video encoding and decoding system may also include three or more quantization coding units and corresponding decoding and dequantization units to support the generation of bitstreams with three or more bitrates.

[0166] Figure 8 is a schematic structural block diagram of another audio codec system provided according to an embodiment of this application. Figure 8 is an improvement on the audio codec system shown in Figure 6. As shown in Figure 8, the transmitting end of the audio codec system includes a feature extraction module 810 and a quantization encoding module 820, and the transmitting end of the audio codec system includes a decoding inverse quantization module 830 and a feature reconstruction module 840. The feature extraction module 810 includes a feature extraction unit 811 and an encoding feature adjustment unit 812, the quantization encoding module 820 includes a first quantization encoding unit 821, the decoding inverse quantization module 830 includes a first decoding inverse quantization unit 831, and the feature reconstruction module 840 includes a decoding feature adjustment unit 842 and a feature reconstruction unit 841.

[0167] The feature extraction unit 811 at the transmitting end can extract features of the first input audio signal to obtain first feature data. The first quantization encoding unit 821 can quantize the first feature data using the entire codebook, then encode the quantization result to obtain a first bitstream and send it to the receiving end. The bit rate of the first bitstream is a first bit rate. The first decoding and dequantization unit 831 at the decoding end acquires the first bitstream, decodes the first bitstream, and dequantizes the decoding result using the entire codebook to obtain first dequantized feature data. The feature reconstruction unit 841 can determine the first output audio signal based on the first dequantized feature data.

[0168] Further, the encoding feature adjustment unit 812 can determine encoding feature adjustment data based on the first feature data determined by the feature extraction unit 811. The first quantization encoding unit 821 can use a portion of the codebooks to perform quantization and encoding operations on the encoding feature adjustment data determined by the encoding feature adjustment unit 812 to obtain a second bitstream. The first quantization encoding unit 821 can include multiple codebooks. For example, assume that the first quantization encoding unit 821 includes N codebooks, where N is a positive integer greater than or equal to 2. The first quantization encoding unit 821 can use these N codebooks to quantize the first feature data determined by the feature extraction unit 811, and then encode the quantization result to obtain a first bitstream. The first quantization encoding unit 821 can also use a portion of the codebooks in the N codebooks to quantize the encoding feature adjustment data determined by the encoding feature adjustment unit 812. For example, the first quantization encoding unit 821 can use M codebooks from the N codebooks to quantize the encoding feature adjustment data, and then encode the quantized result to obtain a second bitstream. M is a positive integer greater than or equal to 1 and less than N. For example, the M codebooks are the first M codebooks from the N codebooks. The first decoding and dequantization unit 831 at the receiving end can decode the second bitstream, and then dequantize the decoding result using M codebooks to obtain second dequantized feature data. The decoding feature adjustment unit 842 determines decoding feature adjustment data based on the second dequantized feature data. The feature reconstruction unit 841 determines the second output audio signal based on the decoding feature adjustment data.

[0169] As mentioned earlier, Figure 8 adds an encoding feature adjustment unit 812 to the feature extraction module at the transmitting end, based on Figure 6. The encoding feature adjustment unit 812 compensates for the output of the feature extraction unit 811, making the audio signal output by the feature reconstruction unit 841 closer to the input audio signal. Therefore, the compensation system of the audio codec system in Figure 8 includes the encoding feature adjustment unit 812 and the decoding feature adjustment unit 842. Compared to a technique that uses a single codec to generate two different bitrates, the technique shown in Figure 8 yields a better output audio signal.

[0170] Compared to techniques that use two codecs to generate two different bitrates, the above-described technique reduces the training and storage pressure on the model. As shown in Figure 8, the technique introduces an encoding feature adjustment unit 812 and a decoding feature adjustment unit 842 to compensate for the outputs of the feature extraction unit 811 and the first decoding inverse quantization unit 831, respectively. This allows the audio signal output by the feature reconstruction unit 841 to more closely approximate the input audio signal. Therefore, compared to techniques that use a single codec to generate two different bitrates, the technique shown in Figure 8 produces a better output audio signal.

[0171] Both the encoding feature adjustment unit 812 and the decoding feature adjustment unit 842 can be a neural network. This neural network can be a CNN, RNN, transformer neural network, or various activation functions.

[0172] The encoding feature adjustment unit 812 and the decoding feature adjustment unit 842 can be trained based on an already trained initial encoding / decoding system. The training dataset can include multiple training samples. The training process of the encoding feature adjustment unit 812 and the decoding feature adjustment unit 842 is described below using training sample d as an example.

[0173] The training sample d is input into the feature extraction unit 811.

[0174] The feature extraction unit 811 extracts the features of the training sample d to obtain the first feature data d1.

[0175] The encoding feature adjustment unit 812 determines the encoding feature adjustment data d1' based on the first feature data d1. It can be understood that the encoding feature adjustment unit 812 mentioned here is an initial neural network or a neural network during the training process.

[0176] The first quantization and encoding unit 821 uses a portion of the codebook to perform quantization and encoding operations on the coded feature adjustment data d1' to obtain the bitstream d2. The portion of the codebook used by the first quantization and encoding unit 821 to determine the bitstream d2 during the training phase is the same as the codebook used during the inference phase. For example, if the first quantization and encoding unit 821 uses the first M codebooks from N codebooks to perform quantization and encoding operations on the coded feature adjustment data d1' during the training phase to obtain the bitstream d2, then the first quantization and encoding unit 821 will also use the first M codebooks from N codebooks to perform quantization and encoding operations on the coded feature adjustment data during the inference phase to obtain the bitstream.

[0177] The first decoding and dequantization unit 831 uses a portion of the codebook to decode and dequantize the bitstream d2, obtaining dequantized feature data d3. It can be understood that the portion of the codebook used for dequantization is the same as the portion of the codebook used for quantization. That is, if the first quantization and encoding unit 821 uses the first M codebooks from the N codebooks to perform quantization and encoding operations on the encoded feature adjustment data d1', obtaining bitstream d2, then the first decoding and dequantization unit 831 also uses the first M codebooks from the N codebooks to decode and dequantize bitstream d2, obtaining dequantized feature data d3.

[0178] The decoding feature adjustment unit 842 determines the decoding feature adjustment data d4 based on the dequantized feature data d3. It can be understood that the decoding feature adjustment unit 842 mentioned here is an initial neural network or a neural network during the training process.

[0179] A loss function L81 can be determined based on the decoded feature adjustment data d4 and the first feature data d1. The loss function L81 can be used to optimize the decoded feature adjustment unit 842 and the encoded feature adjustment unit 812, so the loss function L81 can also be called the feature adjustment loss function.

[0180] In some embodiments, the parameters of the neural network can be determined based on the loss function L81 to obtain the decoding feature adjustment unit and the encoding feature adjustment unit.

[0181] In other embodiments, in addition to loss function L81, loss function L82 may also be determined. Loss function L82 is the loss function between output sample b' and training sample b. Output sample b' is determined by feature reconstruction unit 841 based on decoded feature adjustment data d4. In this case, the parameters of the neural network can be determined based on loss function L81 and loss function L82 to obtain the decoded feature adjustment unit and the encoded feature adjustment unit.

[0182] In the above embodiments, the loss function L81 is determined based on the output of the decoding feature adjustment unit 842 and the output of the feature extraction unit 811. In other embodiments, a loss function L81' can also be determined based on the output of the decoding feature adjustment unit 842 and the output of the first decoding dequantization unit 831. In this case, the parameters of the neural network can be determined based on the loss function L81' to obtain the decoding feature adjustment unit and the encoding feature adjustment unit. Alternatively, the parameters of the neural network can be determined based on the loss function L81' and the loss function L82 to obtain the decoding feature adjustment unit.

[0183] The following describes how the loss function L81' is determined. The first quantization and encoding unit 821 can use its entire codebook to perform quantization and encoding operations on the first feature data d1', obtaining the bitstream d2'. The first decoding and dequantization unit 831 uses its entire codebook to decode and dequantize the quantized data d2', obtaining the dequantized feature data d3'. The loss function L81' is the loss function between the decoded feature adjustment data d4 and the dequantized feature data d3'.

[0184] The partial codebooks described in Figures 6 and 8 can have multiple design methods. In addition to the first M codebooks in the N codebooks mentioned above, the partial codebooks can also be the last K codebooks in the N codebooks, the X codebooks in the middle of the latter, etc. By selecting different codebooks for the quantization coding unit, the audio codec system in Figures 6 and 8 can support the generation of three or more bitrate streams.

[0185] The multi-bitrate codec model shown in Figures 5 to 8 includes an initial codec system and a compensation system (also referred to as a compensation model). In some embodiments, the multi-bitrate codec model may include an initial codec system and multiple (i.e., two or more) compensation models. These multiple compensation models are used to determine the output signals at multiple bitrates. For example, a compensation model can be added to the multi-bitrate codec model shown in Figure 5. This compensation model may include a third quantization coding unit, a third decoding dequantization unit, and a second decoding feature adjustment unit. For distinction, the newly added compensation model can be called the second compensation model, and the compensation model shown in Figure 5 can be called the first compensation model. Using this second compensation model, an output audio signal with a bitrate of the third bitrate can be output. The third bitrate is different from the first and second bitrates, and the third bitrate can be less than the first bitrate. The process of determining the output audio signal with the third bitrate can refer to the process of determining the output audio signal with the second bitrate, and will not be repeated here for simplicity. The training method of each compensation model in these multiple compensation models is the same as the training method of each compensation model in Figures 5 to 8. For simplicity, will not be repeated here.

[0186] In some embodiments, if the multi-rate codec model includes multiple compensation models, then any two of the multiple compensation models have the same structure. For example, if multiple compensation models are added to the multi-rate codec model shown in Figure 5, then each of the multiple compensation models includes a quantization coding unit, a decoding dequantization unit, and a decoding feature adjustment unit.

[0187] In other embodiments, if the multi-rate codec model includes multiple compensation models, the structures of these multiple compensation models may not be entirely identical. For example, if multiple compensation models are added to the multi-rate codec model shown in Figure 5, one of these compensation models may be the second compensation model described above, and the third compensation model may only include coding feature adjustment units and decoding feature adjustment units, but not quantization coding units and decoding dequantization units. In other words, the structure of this compensation model is not entirely the same as that of the compensation model shown in Figure 5. The training process of this compensation model can be referred to the embodiment shown in Figure 8, which will not be elaborated here for the sake of simplicity.

[0188] Referring to Figure 9, this application embodiment provides a system architecture 900 for neural network training. As shown in the system architecture 900, a data acquisition device 960 is used to collect training data and can store the training data in a database 930. In this application embodiment, the training data can be audio signals. The training device 920 can train a first target model 901 and / or a second target model 902 based on the training data maintained in the database 930. The first target model 901 can be the decoding feature adjustment unit mentioned in the above embodiment. The second target model 902 can be the encoding feature adjustment unit mentioned in the above embodiment. It should be noted that in practical applications, the training data maintained in the database 930 may not all come from the data acquisition device 960, but may also be received from other devices. In addition, it should be noted that the training device 920 may not necessarily train the target models 901 and / or 902 entirely based on the training data maintained in the database 930, but may also obtain training data from the cloud or other places for model training. The above description should not be construed as limiting the embodiments of this application.

[0189] The first target model 901 and / or the second target model 902 obtained by training according to the training device 920 can be applied to the execution device 910 shown in FIG9. The execution device 910 can be the terminal device mentioned above or the electronic device 100 shown in FIG1.

[0190] For example, the execution device 910 can encode and quantize the acquired input data based on the second target model 902, and send the quantized data out through the communication module 914. As shown in FIG9, the execution device 910 is configured with an I / O interface 912 for data interaction with external devices. Users can input data into the I / O interface 912. The input data in this embodiment may include audio signals collected by a microphone. The microphone may be a built-in microphone of the execution device 910, or a microphone of a wired headset connected to the execution device 910 through a wired interface (such as a headphone jack, a universal serial bus (USB) interface, etc.), or a microphone of a wireless headset connected to the execution device 910 through wireless communication technology (such as Wi-Fi).

[0191] The preprocessing module 913 is used to preprocess the input data (such as the audio signal) received by the I / O interface 912, such as noise reduction, etc., which are not limited in this application.

[0192] During the preprocessing of input data by the execution device 910, or during the calculation and related processing of the calculation module 911 of the execution device 910, the modules in the execution device 910 (such as the calculation module 911 and the preprocessing module 913) can call the data, code, etc. in the storage module 915 to achieve the corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the storage module 915.

[0193] For example, the execution module 910 can dequantize the quantized data received through the communication module 914, and then decode the dequantized data based on the first target model 901 to obtain the output audio signal, which is then played through headphones or a receiver.

[0194] As shown in Figure 9, the execution device 910 may include a chip. This chip is used to perform the calculations of the calculation module 911. This chip can also be included in the training device 920 shown in Figure 9 to perform the training work of the training device 920 and output the first target model 901 and / or the second target model 902. All modules and units (e.g., encoder, decoder, encoding feature adjustment unit, decoding feature adjustment unit, etc.) in the embodiments of this application can be implemented in this chip. This chip may include a convolutional neural network processor.

[0195] The convolutional neural network processor can be a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU), or any processor suitable for large-scale XOR operations. Taking an NPU as an example: the NPU can be attached as a coprocessor to the central processing unit (CPU), i.e., the host CPU, which allocates tasks to it. The core of the NPU is the arithmetic circuitry, which, controlled by a controller, retrieves matrix data from memory and performs multiplication and addition operations.

[0196] In some implementations, the arithmetic circuitry includes multiple process engines (PEs). In some implementations, the arithmetic circuitry is a two-dimensional pulsating array. The arithmetic circuitry can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry is a general-purpose matrix processor.

[0197] It is worth noting that Figure 9 is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc. shown in Figure 9 do not constitute any limitation. For example, in Figure 9, the storage module 915 is located within the execution device 910. In other cases, the storage module 915 may also be an external memory relative to the execution device 910.

[0198] As described above, the units included in the initial codec system can be further divided into a first encoding end and a first decoding end. The first encoding end may include a feature extraction unit and a quantization encoding unit (or a first quantization encoding unit), and the first decoding end may include a feature reconstruction unit and a decoding inverse quantization unit (or a first decoding inverse quantization unit). Similarly, the units included in the compensation system are further divided into a compensation encoding end and a compensation decoding end. In some embodiments, the compensation system may only include a compensation decoding end. For example, the compensation system in the audio codec system shown in FIG6 only includes a compensation decoding end, which includes a decoding feature adjustment unit 642. The compensation system in the audio codec systems shown in FIG5, FIG7, and FIG8 includes a compensation encoding end and a compensation decoding end. For example, in the audio codec system shown in FIG5, the compensation encoding end includes a second quantization encoding unit 522, and the compensation decoding end includes a second decoding inverse quantization unit 532 and a decoding feature adjustment unit 542. As another example, in the audio codec system shown in FIG7, the compensation encoding end includes an encoding feature adjustment unit 712, a second quantization encoding unit 722, a second decoding inverse quantization unit 732, and a decoding feature adjustment unit 742. For example, in the audio codec system shown in Figure 8, the compensation encoding end includes an encoding feature adjustment unit 812, and the compensation decoding end includes a decoding feature adjustment unit 842.

[0199] Based on the above embodiments, this application also provides an electronic device that may include the aforementioned audio codec system. For example, the electronic device may include the audio codec system shown in any one of Figures 5 to 8.

[0200] This application also provides an electronic device that may include the units included in the transmitting end of the aforementioned audio codec system. In other words, the electronic device may include a first encoding end. If the compensation system also includes a compensation encoding end, then the electronic device may also include a compensation encoding end. For example, in some embodiments, the electronic device may include a feature extraction unit 511, a first quantization encoding unit 521, and a second quantization encoding unit 522. In some embodiments, the electronic device may include a feature extraction unit 611 and a first quantization encoding unit 621. In some embodiments, the electronic device may include a feature extraction unit 711, an encoding feature adjustment unit 712, a first quantization encoding unit 721, and a second quantization encoding unit 722. In some embodiments, the electronic device may include a feature extraction unit 811, an encoding adjustment unit 812, and a first quantization encoding unit 821. For ease of description, the electronic device including the various units of the transmitting end may be referred to as a transmitting end device.

[0201] This application also provides an electronic device that may include the units included in the receiving end of the aforementioned audio codec system. In other words, the electronic device may include a first decoding end and a compensation decoding end. For example, in some embodiments, the electronic device may include a first decoding inversion unit 531, a second decoding inversion unit 532, a feature reconstruction unit 541, and a decoding feature adjustment unit 542. As another example, in some embodiments, the electronic device may include a first decoding inversion unit 631, a feature reconstruction unit 641, and a decoding feature adjustment unit 642. As yet another example, in some embodiments, the electronic device may include a first decoding inversion unit 731, a second decoding inversion unit 732, a feature reconstruction unit 741, and a decoding feature adjustment unit 742. As yet another example, in some embodiments, the electronic device may include a first decoding inversion unit 831, a feature reconstruction unit 841, and a decoding feature adjustment unit 841. For ease of description, the electronic device including the various units of the transmitting end can be referred to as a receiving end device.

[0202] Figure 10 is a schematic flowchart of an audio signal processing method according to an embodiment of this application. The method shown in Figure 10 can be executed by a receiving device. The method includes steps 1001-1003.

[0203] 1001, Receive target bitstream.

[0204] The target bitstream can be either a first bitstream or a second bitstream. The first bitstream is the base bitstream in the first codec system. The first bitstream and the second bitstream have different bitrates. The first codec system includes a first encoder and a first decoder.

[0205] 1002. When the target bitstream is the first bitstream, the first decoding end is used to perform decoding and dequantization operations on the first bitstream to obtain the first dequantization feature data, and the first output audio signal corresponding to the first bitstream is generated based on the first dequantization feature data.

[0206] 1003, when the target bitstream is the second bitstream, the compensation system is used to obtain the second inverse quantization feature data corresponding to the second bitstream, and the decoding feature adjustment data is determined based on the second inverse quantization feature data; the feature reconstruction unit in the first decoding end is used to obtain the second output audio signal corresponding to the second bitstream based on the decoding feature adjustment data, and the feature reconstruction unit is trained based on the transmission process of the first bitstream.

[0207] Optionally, in some embodiments, the compensation system may include a decoding feature adjustment unit. This decoding feature adjustment unit can acquire second inverse quantization feature data corresponding to the second bitstream, and determine the decoding feature adjustment data based on the second inverse quantization feature data.

[0208] Optionally, in some embodiments, the first decoding and dequantization unit in the first decoding end can decode and dequantize the second bitstream according to a portion of the codebook to obtain the second dequantized feature data. The decoding feature adjustment unit can obtain the second dequantized feature data from the first decoding and dequantization unit. The first decoding and dequantization unit also decodes and dequantizes the second bitstream according to the entire codebook to obtain the second dequantized feature data.

[0209] Optionally, in some embodiments, the compensation system may further include a second decoding and dequantization unit. This second decoding and dequantization unit can decode and dequantize the second bitstream to obtain the second dequantized feature data. The decoding feature adjustment unit can obtain the second dequantized feature data from the second decoding and dequantization unit.

[0210] Figure 11 is a schematic flowchart of another audio signal processing method provided according to an embodiment of this application. The method shown in Figure 11 can be performed by a transmitting device. The method includes steps 1101 and 1102.

[0211] 1101, Obtain the input audio signal.

[0212] 1102, Generate the target bitstream based on the input audio signal.

[0213] The target bitstream can be a first bitstream or a second bitstream, the first bitstream being the base bitstream of a first codec system, and the first bitstream and the second bitstream having different bitrates.

[0214] The first encoding / decoding system may include a first encoding end and a first decoding end.

[0215] In some embodiments, the first encoding end includes a feature extraction unit and a first quantization encoding unit. The first decoding end includes a feature reconstruction unit and a first decoding inverse quantization unit.

[0216] In some embodiments, the first encoding end can generate a first bitstream based on the input audio signal. For example, the feature extraction unit can acquire the encoded feature data of the input audio signal. The first quantization encoding unit can perform quantization and encoding operations on the encoded feature data according to the complete codebook of the first encoding unit to obtain the first bitstream.

[0217] In other embodiments, the first encoding end can generate a second bitstream based on the input audio signal. For example, the feature extraction unit can acquire the encoded feature data of the input audio signal. The first quantization encoding unit can perform quantization and encoding operations on the encoded feature data based on a portion of the codebook of the first encoding unit to obtain the second bitstream.

[0218] In some embodiments, the second bitstream may be generated by the first encoding end and the compensation system.

[0219] For example, in some embodiments, the compensation system may include a second quantization encoding unit. The feature extraction unit can acquire coded feature data of the input audio signal. The second quantization encoding unit can perform quantization and encoding operations on the coded feature data to obtain the second bitstream.

[0220] For example, in some other embodiments, the compensation system may include a coding feature adjustment unit. The feature extraction unit can acquire coding feature data of the input audio signal. The coding feature adjustment unit can adjust the coding feature data to obtain coding feature adjustment data. The first quantization coding unit can perform quantization and encoding operations on the coding feature adjustment data according to a portion of its codebook to obtain the second bitstream.

[0221] For example, in some other embodiments, the compensation system may include a coding feature adjustment unit and a second quantization coding unit. In this case, the feature extraction unit can acquire the coding feature data of the input audio signal. The coding feature adjustment unit can adjust the coding feature data to obtain coding feature adjustment data. The second quantization coding unit can perform quantization and encoding operations on the coding feature adjustment data to obtain the second bitstream.

[0222] Each unit in the compensation system can be implemented by a neural network. Optionally, in some embodiments, the decoding feature adjustment unit is a first neural network, which is trained in the following manner: based on the first training samples, a first loss function and a second loss function are determined; the parameters of the first neural network are determined according to the first loss function and the second loss function, wherein the first loss function is a loss function between the first reference feature data and the first reference inverse quantization feature data, the second loss function is a loss function between the first reference feature data and the first reference decoding feature adjustment data, the first reference feature data is the feature data extracted by the feature extraction unit from the first training samples, the first reference inverse quantization feature data is determined by the second quantization encoding unit and the second decoding inverse quantization unit based on the first reference feature data, and the first reference decoding feature adjustment data is determined by the first neural network under training based on the first reference inverse quantization feature data.

[0223] Optionally, in some embodiments, the decoding feature adjustment unit is a second neural network trained in the following manner: based on the second training samples, a third loss function and a fourth loss function are determined; based on the third loss function and the fourth loss function, the parameters of the second neural network are determined; wherein the third loss function is a loss function between the second reference feature data and the second reference dequantization feature data, the fourth loss function is a loss function between the second target dequantization feature data and the second reference decoding feature adjustment data, the second reference feature data is feature data extracted by the feature extraction unit from the second training samples, the second reference dequantization feature data is determined by the second quantization encoding unit and the second decoding dequantization unit based on the second reference feature data, the second target dequantization feature data is determined by the first quantization encoding unit and the first decoding dequantization unit based on the second reference feature data, and the second reference decoding feature adjustment data is determined by the second quantization encoding unit, the second decoding dequantization unit, and the second neural network under training based on the second reference feature data.

[0224] Optionally, in some embodiments, the second bitstream is obtained by the first quantization encoding unit performing quantization and encoding operations on the encoded feature data according to a portion of the codebook of the first quantization encoding unit; the first decoding and dequantization unit decodes and dequantizes the second bitstream according to a portion of the codebook of the first quantization encoding unit to obtain the second dequantized feature data.

[0225] Optionally, in some embodiments, the decoding feature adjustment unit is a third neural network, which is trained in the following manner: a fifth loss function is determined based on a third training sample, and the parameters of the third neural network are determined according to the fifth loss function. The fifth loss function is a loss function between the third reference inverse quantization feature data and the third reference decoding feature adjustment data. The third reference inverse quantization feature data is determined by the feature extraction unit, the first quantization encoding unit, and the first decoding inverse quantization unit based on the complete codebook of the first quantization encoding unit and the third training sample. The third reference decoding feature adjustment data is determined by the feature extraction unit, the first quantization encoding unit, the first decoding inverse quantization unit, and the trained third neural network based on a portion of the codebook of the first quantization encoding unit and the third training sample.

[0226] Optionally, in some embodiments, the decoding feature adjustment unit is a fourth neural network, which is trained based on the following: determining a sixth loss function based on a fourth training sample, and determining the parameters of the fourth neural network according to the sixth loss function, wherein the sixth loss function is a loss function between the fourth reference feature data and the fourth reference decoding feature adjustment data, the fourth reference feature data is feature data extracted by the feature extraction unit from the fourth training sample, and the fourth reference decoding feature adjustment data is determined by the first quantization encoding unit, the first decoding dequantization unit, and the fourth neural network under training based on a partial codebook of the first quantization encoding unit and the fourth reference feature data.

[0227] Optionally, in some embodiments, the encoded feature data is determined by the encoded feature adjustment unit based on the input feature data, which is the feature data extracted by the feature extraction unit from the input audio signal.

[0228] Optionally, in some embodiments, the encoding feature adjustment unit is a fifth neural network, and the decoding feature adjustment unit is a sixth neural network. The fifth neural network and the sixth neural network are trained in the following manner: based on the fifth training samples, a seventh loss function is determined; based on the seventh loss function, the parameters of the fifth neural network and the parameters of the sixth neural network are determined. The seventh loss function is a loss function between the fifth reference feature data and the fifth decoding feature adjustment data. The fifth reference feature data is the feature data extracted by the feature extraction unit from the fifth training samples. The fifth decoding feature adjustment data is determined by the fifth neural network, the quantization encoding module, the decoding dequantization module, and the sixth neural network in training based on the fifth reference feature data.

[0229] Optionally, in some embodiments, the encoding feature adjustment unit is a seventh neural network, and the decoding feature adjustment unit is an eighth neural network. The seventh neural network and the eighth neural network are trained in the following manner: based on the sixth training samples, an eighth loss function is determined; based on the eighth loss function, the parameters of the seventh neural network and the parameters of the eighth neural network are determined. The eighth loss function is a loss function between the sixth reference inverse quantization feature data and the sixth reference decoding feature adjustment data. The sixth reference inverse quantization feature data is determined by the feature extraction unit, the trained seventh neural network, the quantization encoding module, and the decoding inverse quantization module based on the sixth training samples. The sixth reference decoding feature adjustment data is determined by the trained eighth neural network based on the sixth reference inverse quantization feature data.

[0230] Optionally, in some embodiments, the encoded feature data is a feature signal extracted from the input audio signal by a feature extraction unit.

[0231] Optionally, in some embodiments, a ninth loss function is also determined during the training process. This ninth loss function is used to train the encoding feature adjustment unit and / or the decoding feature adjustment unit. The ninth loss function is a loss function between the seventh training sample and the seventh output sample. The seventh output sample is determined by the feature extraction module, the quantization encoding module, the decoding dequantization module, the decoding feature adjustment unit under training, and the first feature reconstruction unit based on the seventh training sample.

[0232] The first loss function and the third loss function can be the quantization loss functions in the above embodiments.

[0233] The second loss function, the fourth loss function, the fifth loss function, and the sixth loss function can be the decoding feature adjustment loss function in the above embodiments.

[0234] The seventh loss function and the eighth loss function can be the feature adjustment loss functions in the above embodiments.

[0235] The ninth loss function can be the loss function between the output sample a' and the training sample a in the above embodiment.

[0236] The reference feature data (e.g., first reference feature data, second reference feature data, etc.) can be the first feature data a1 or the first feature data b1 in the above embodiments.

[0237] The reference dequantization feature data (e.g., first reference dequantization feature data, second reference dequantization feature data, etc.) can be dequantization feature data a3, dequantization feature data a3', dequantization feature data b3', or dequantization feature data b3 in the above embodiments.

[0238] The reference decoding feature adjustment data (e.g., first reference decoding feature adjustment data, second reference decoding feature adjustment data, etc.) can be the decoding feature adjustment data a4 or decoding feature adjustment data b4 in the above embodiments.

[0239] The reference dequantization data (e.g., the second reference dequantization data) can be the dequantization feature data a3' in the above embodiments.

[0240] Figure 12 is a schematic structural block diagram of an electronic device according to an embodiment of this application. As shown in Figure 12, the electronic device 1200 includes a processor 1201, which executes computer programs or instructions stored in a memory 1202, or reads data / signaling stored in the memory 1202, to perform the methods in the above-described method embodiments. Optionally, there may be one or more processors 1201.

[0241] The memory 1202 is used to store computer programs or instructions and / or data. The memory 1202 may be integrated with the processor 1201 or may be disposed separately. Optionally, there may be one or more memories 1202.

[0242] Optionally, as shown in FIG12, the electronic device 1200 further includes a transceiver 1203 for receiving and / or transmitting signals. For example, the processor 1201 is used to control the transceiver 1203 to receive and / or transmit signals.

[0243] As one approach, electronic device 1200 is used to implement the operations performed by the aforementioned audio codec system. In other words, electronic device 1200 can perform the operations performed by the transmitting end and the operations performed by the receiving end in the various method embodiments described above.

[0244] As an alternative, electronic device 1200 is used to implement the operations performed by the sending end in the various method embodiments described above.

[0245] As an alternative, electronic device 1200 is used to implement the operations performed by the receiving end in the various method embodiments described above.

[0246] It should be understood that the processor mentioned in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0247] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM). For example, RAM can be used as an external cache. By way of example and not limitation, RAM includes the following forms: static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0248] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) can be integrated into the processor.

[0249] It should also be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0250] Figure 13 is a schematic diagram of a chip system 1300 provided in an embodiment of this application. The chip system 1300 (or may also be called a processing system) includes logic circuits 1301 and input / output interface 1302.

[0251] The logic circuit 1301 can be a processing circuit in the chip system 1300. The logic circuit 1301 can be coupled to a memory unit, calling instructions from the memory unit, enabling the chip system 1300 to implement the methods and functions of the embodiments of this application. The input / output interface 1302 can be an input / output circuit in the chip system 1300, outputting processed information from the chip system 1300, or inputting data or signaling information to be processed into the chip system 1300 for processing.

[0252] As one option, the chip system 1300 is used to implement the operations performed by the transmitting end and / or receiving end in the various method embodiments described above.

[0253] This application also provides a computer-readable storage medium storing computer instructions for implementing the methods executed by the sending end and / or receiving end in the above-described method embodiments.

[0254] For example, when the computer program is executed by an electronic device, the electronic device can implement the methods executed by the sending end and / or receiving end in the various embodiments of the above methods.

[0255] This application also provides a computer program product comprising instructions which, when executed by a computer, implement the methods performed by the sending end and / or receiving end in the above-described method embodiments.

[0256] This application also provides a communication system, including the aforementioned transmitting device and receiving device.

[0257] The explanations and beneficial effects of the relevant contents in any of the devices provided above can be found in the corresponding method embodiments provided above, and will not be repeated here.

[0258] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of apparatus or units may be electrical, mechanical, or other forms.

[0259] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An audio encoding and decoding system, characterized in that, It includes a first encoding / decoding system and a compensation system. The first encoding / decoding system includes a first encoding end and a first decoding end. The first decoding end includes a feature reconstruction unit. The first encoding end is used to: generate a target bitstream based on the input audio signal and send the target bitstream to the first decoding end. The target bitstream includes a first bitstream or a second bitstream. The first bitstream is the base bitstream of the first encoding and decoding system. The first bitstream and the second bitstream have different bitrates. When the target bitstream is the first bitstream, the first decoding end is used to: perform decoding and dequantization operations on the first bitstream to obtain first dequantization feature data, and generate a first output audio signal corresponding to the first bitstream based on the first dequantization feature data; When the target bitstream is the second bitstream, the compensation system is used to: obtain the second inverse quantization feature data corresponding to the second bitstream, and determine the decoding feature adjustment data based on the second inverse quantization feature data; The feature reconstruction unit is used to adjust the data according to the decoding features to obtain the second output audio signal corresponding to the second bitstream. The feature reconstruction unit is trained according to the transmission process of the first bitstream.

2. The audio encoding and decoding system according to claim 1, characterized in that, The first encoding end includes a feature extraction unit and a first quantization encoding unit; The first decoding end includes a first decoding inverse quantization unit and the feature reconstruction unit; The feature extraction unit is used to obtain the encoded feature data of the input audio signal; The first quantization and encoding unit is used to perform quantization and encoding operations on the encoded feature data to obtain the first bitstream; The first decoding and dequantization unit is used to perform decoding and dequantization operations on the first bitstream to obtain the first dequantization feature data; The feature reconstruction unit is used to generate the first output audio signal based on the first inverse quantization feature data, or to generate the second output audio signal based on the decoding feature adjustment data.

3. The audio encoding and decoding system according to claim 2, characterized in that, The compensation system includes a decoding feature adjustment unit, which is used to acquire the second inverse quantization feature data corresponding to the second bitstream, and determine the decoding feature adjustment data based on the second inverse quantization feature data.

4. The audio encoding / decoding system according to claim 2 or 3, characterized in that, The first quantization encoding unit is specifically used for: The first bitstream is obtained by performing quantization and encoding operations on the encoded feature data according to the complete codebook of the first quantization encoding unit, and the second bitstream is obtained by performing quantization and encoding operations on the encoded feature data according to a portion of the codebook of the first quantization encoding unit. The first decoding and dequantization unit is specifically used to: perform decoding and dequantization operations on the first bitstream according to the complete codebook to obtain the first dequantization feature data, and perform decoding and dequantization operations on the second bitstream according to the partial codebook to obtain the second dequantization feature data.

5. The audio encoding / decoding system according to claim 2 or 3, characterized in that, The compensation system further includes a coding feature adjustment unit, which is used to adjust the coding feature data to obtain coding feature adjustment data. The first quantization encoding unit is specifically used for: The first bitstream is obtained by performing quantization and encoding operations on the encoded feature data according to the complete codebook of the first quantization encoding unit, and the second bitstream is obtained by performing quantization and encoding operations on the encoded feature adjustment data according to a portion of the codebook of the first quantization encoding unit. The first decoding and dequantization unit is specifically used to: perform decoding and dequantization operations on the first bitstream according to the complete codebook to obtain the first dequantization feature data, and perform decoding and dequantization operations on the second bitstream according to the partial codebook to obtain the second dequantization feature data.

6. The audio encoding and decoding system according to claim 3, characterized in that, The compensation system further includes a second quantization encoding unit and a second decoding and dequantization unit. The second quantization and encoding unit is used to perform quantization and encoding operations on the encoded feature data to obtain the second bitstream; The second decoding and dequantization unit is used to perform decoding and dequantization operations on the second bitstream to obtain the second dequantized feature data; The decoding feature adjustment unit is specifically used to obtain the second dequantization feature data from the second decoding dequantization unit, and determine the decoding feature adjustment data based on the second dequantization feature data.

7. The audio encoding / decoding system according to claim 3, characterized in that, The compensation system further includes an encoding feature adjustment unit, a second quantization encoding unit, and a second decoding and dequantization unit. The coding feature adjustment unit is used to adjust the coding feature data to obtain coding feature adjustment data, and send the coding feature adjustment data to the second quantization coding unit; The second quantization and encoding unit is used to perform quantization and encoding operations on the encoded feature adjustment data to obtain the second bitstream; The second decoding and dequantization unit is used to perform decoding and dequantization operations on the second bitstream to obtain the second dequantized feature data; The decoding feature adjustment unit is specifically used to obtain the second dequantization feature data from the second decoding dequantization unit, and determine the decoding feature adjustment data based on the second dequantization feature data.

8. The audio encoding / decoding system according to claim 6 or 7, characterized in that, The decoding feature adjustment unit is a first neural network, which is trained in the following way: Based on the first training sample, determine the first loss function and the second loss function; The parameters of the first neural network are determined based on the first loss function and the second loss function; in The first loss function is the loss function between the first reference feature data and the first reference dequantized feature data. The second loss function is the loss function between the first reference feature data and the first reference decoded feature adjustment data. The first reference feature data is the feature data extracted by the feature extraction unit from the first training sample. The first reference dequantization feature data is determined by the second quantization encoding unit and the second decoding dequantization unit based on the first reference feature data. The first reference decoding feature adjustment data is determined by the first neural network during training based on the first reference dequantization feature data.

9. The audio encoding / decoding system according to claim 6 or 7, characterized in that, The decoding feature adjustment unit is a second neural network, which is trained in the following way: Based on the second training samples, determine the third and fourth loss functions; The parameters of the second neural network are determined based on the third loss function and the fourth loss function; wherein, The third loss function is the loss function between the second reference feature data and the second reference dequantized feature data. The fourth loss function is the loss function between the initial dequantized feature data and the second reference decoded feature adjusted data. The second reference feature data is the feature data extracted by the feature extraction unit from the second training sample. The second reference dequantization feature data is determined by the second decoding dequantization unit based on the second reference feature data. The initial dequantization feature data is determined by the first quantization encoding unit and the first decoding dequantization unit based on the second reference feature data. The second reference decoding feature adjustment data is determined by the second quantization encoding unit, the second decoding dequantization unit, and the second neural network under training based on the second reference feature data.

10. The audio encoding and decoding system according to claim 4, characterized in that, The decoding feature adjustment unit is a third neural network, which is trained in the following way: Based on the third training sample, determine the fifth loss function. Based on the fifth loss function, the parameters of the third neural network are determined, wherein, The fifth loss function is the loss function between the third reference inverse quantization feature data and the third reference decoded feature adjustment data. The third reference inverse quantization feature data is determined by the feature extraction unit, the first quantization encoding unit, and the first decoding inverse quantization unit based on the complete codebook of the first quantization encoding unit and the third training samples. The third reference decoding feature adjustment data is determined by the feature extraction unit, the first quantization encoding unit, the first decoding dequantization unit, and the third neural network under training, based on a partial codebook of the first quantization encoding unit and the third training samples.

11. The audio encoding and decoding system according to claim 4, characterized in that, The decoding feature adjustment unit is a fourth neural network, which is trained in the following way: Based on the fourth training sample, determine the sixth loss function. Based on the sixth loss function, the parameters of the fourth neural network are determined, wherein, The sixth loss function is the loss function between the fourth reference feature data and the fourth reference decoded feature adjustment data. The fourth reference feature data is the feature data extracted by the feature extraction unit from the fourth training sample. The fourth reference decoding feature adjustment data is determined by the first quantization encoding unit, the first decoding dequantization unit, and the fourth neural network under training based on a partial codebook of the first quantization encoding unit and the fourth reference feature data.

12. An audio signal processing method, characterized in that, include: Receive a target bitstream, the target bitstream being a first bitstream or a second bitstream, the first bitstream being the base bitstream of a first codec system, the first bitstream and the second bitstream having different bitrates, the first codec system including a first encoding end and a first decoding end; When the target bitstream is the first bitstream, the first decoding end is used to perform decoding and dequantization operations on the first bitstream to obtain the first dequantization feature data, and the first output audio signal corresponding to the first bitstream is generated based on the first dequantization feature data. When the target bitstream is the second bitstream, the compensation system is used to obtain the second inverse quantization feature data corresponding to the second bitstream, and the decoding feature adjustment data is determined based on the second inverse quantization feature data; the feature reconstruction unit in the first decoding end is used to obtain the second output audio signal corresponding to the second bitstream based on the decoding feature adjustment data, and the feature reconstruction unit is trained based on the transmission process of the first bitstream.

13. The audio signal processing method according to claim 12, characterized in that, The compensation system includes a decoding feature adjustment unit. The step of using a compensation system to obtain the second inverse quantization feature data corresponding to the second bitstream, and determining the decoding feature adjustment data based on the second inverse quantization feature data, includes: The decoding feature adjustment unit is used to obtain the second inverse quantization feature data corresponding to the second bitstream, and the decoding feature adjustment data is determined based on the second inverse quantization feature data.

14. The audio signal processing method according to claim 13, characterized in that, The first dequantization feature data is obtained by the first decoding and dequantization unit in the first decoding end performing decoding and dequantization operations on the first bitstream based on the entire codebook, and the second dequantization feature data is obtained by the first decoding and dequantization unit performing decoding and dequantization operations on the second bitstream based on a portion of the codebook.

15. The audio signal processing method according to claim 13, characterized in that: The first decoding end includes a first decoding and dequantization unit, and the first dequantization feature data is obtained by the first decoding and dequantization unit performing decoding and dequantization operations on the first bitstream; The compensation system further includes a second decoding and dequantization unit, wherein the second dequantization feature data is obtained by the second decoding and dequantization unit performing decoding and dequantization operations on the second bitstream.

16. An electronic device, characterized in that, The electronic device includes the audio encoding / decoding system according to any one of claims 1 to 11.

17. An electronic device, characterized in that, include: A processor, configured to be coupled to a memory, read and execute instructions and / or program code in the memory to perform various steps performed by a first codec system and a compensation system in an audio codec system as claimed in any one of claims 1 to 11, or to perform an audio signal processing method as claimed in any one of claims 12 to 15.

18. A chip system, characterized in that, include: A logic circuit, the logic circuit being coupled to an input / output interface, through which data is transmitted to perform various steps performed by a first codec system and a compensation system in an audio codec system as described in any one of claims 1 to 11, or to perform an audio signal processing method as described in any one of claims 12 to 15.

19. A computer-readable medium, characterized in that, The computer-readable medium stores program code that, when executed on an electronic device, performs the steps performed by the first encoding / decoding system and the compensation system in the audio encoding / decoding system as described in any one of claims 1 to 11, or performs the audio signal processing method as described in any one of claims 12 to 15.

Images