Coding method and apparatus, and decoding method and apparatus

Abstract

The present application relates to the technical field of coding and decoding, and provides a coding method and apparatus, and a decoding method and apparatus. The coding method comprises: acquiring redundant audio data having a first data length and corresponding to an audio frame to be coded; and using an artificial intelligence model to code the redundant audio data, so as to obtain a bitstream, wherein the bitstream comprises first coded data of the redundant audio data and second coded data of said audio frame, and the compression ratio of the artificial intelligence model is higher than the compression ratio corresponding to said audio frame. The method can improve the resistance to packet loss during audio data transmission, without remarkably increasing the coding bit rate of audio and reducing sound quality.

Description

Encoding and decoding methods and devices

[0001] This application claims priority to Chinese Patent Application No. 202411653686.4, filed on November 18, 2024, entitled “Encoding / Decoding Method and Apparatus”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of encoding and decoding technology, and in particular to an encoding and decoding method and apparatus. Background Technology

[0003] In the field of audio, whether it's speech or music, audio encoding and decoding, as well as audio transmission, are involved. During the transmission of audio data, interference from the transmission channel can cause the encoded audio data to be lost, making it impossible for the decoding end to recover continuous audio and reducing the listening experience.

[0004] Therefore, how to improve the anti-interference capability of audio data during transmission so that the decoding end can recover continuous audio is an urgent problem to be solved in this field. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides an encoding / decoding method and apparatus. This application can improve the packet loss resistance during audio data transmission without significantly increasing the audio encoding bitrate or degrading the sound quality.

[0006] Firstly, this application provides an encoding method. The method includes: acquiring redundant audio data of a first data length corresponding to an audio frame to be encoded; and encoding the redundant audio data using an artificial intelligence (AI) model to obtain a bitstream, wherein the bitstream includes first encoded data of the redundant audio data and second encoded data of the audio frame to be encoded, and the compression ratio of the AI ​​model is higher than the compression ratio corresponding to the audio frame to be encoded.

[0007] This method can be applied to the encoding end.

[0008] In this method, an AI model is used to perform redundant encoding on the redundant audio data corresponding to the audio frame to be encoded in order to obtain a bitstream. In this way, the bitstream includes not only the encoded data of the audio frame to be encoded, but also the encoded data of the redundant audio data of a first data length. Moreover, the data length of the redundant audio data is not limited to a single audio frame, thus enabling strong resistance to packet loss (e.g., in continuous packet loss scenarios).

[0009] Because the compression ratio of the AI ​​model is higher than that of the current frame (such as the compression ratio used for traditional encoding of the current frame), even if the length of the redundant audio data is long, the encoded data of the redundant audio data compressed by the AI ​​model only occupies a small amount of bandwidth. This can resist packet loss while avoiding the problem of excessive bandwidth occupation of the bitstream due to encoding redundant audio data.

[0010] This application uses an AI model to encode redundant audio data separately at a higher compression rate, instead of using half the encoding bitrate of the current frame in a traditional encoder. This reduces the proportion of the redundant audio data's bitrate in the total audio frame bitrate while still providing robustness against packet loss. Consequently, it improves packet loss resistance without significantly degrading sound quality. The decoder can then decode this redundant audio data to obtain continuous, complete, and high-quality audio data without significantly reducing the listening experience.

[0011] In one implementation, the redundant audio data is extracted from previous audio frames and / or subsequent audio frames of the audio frame to be encoded, and the first data length includes any number of frames, or any number of frames plus any number of sampling points.

[0012] In related technologies, when encoding redundant audio data, the redundant audio data can only be a complete audio frame. However, in this application, the length of the redundant audio data encoded by the encoder can include any number of frames (e.g., one or more frames). This allows the method provided in this application to support encoding multiple audio frames as redundant audio data, thereby carrying multiple audio frames in the redundant audio data corresponding to the audio frame to be encoded. This allows for resistance to packet loss of multiple frames while occupying a smaller bitrate. In this way, more redundant audio data can be encoded at a lower bitrate, thereby improving the anti-interference capability during audio transmission in scenarios where the total bitrate of audio data encoding is limited (transmission bandwidth is limited).

[0013] Furthermore, the method provided in this application also supports encoding audio data with any number of sampling points in an audio frame of a given length. This means that the length of redundant audio data is not limited to a complete frame, but can also be audio data with any number of sampling points, such as half a frame or 1 / 3 of a frame. The length of the redundant audio data encoded by the method provided in this application is not limited by the frame granularity and can be flexibly encoded with the number of sampling points as the granularity. Thus, based on application requirements such as the latency of the encoder and decoder, redundant audio data of different lengths can be flexibly encoded, making the length of redundant audio data more flexible and variable, thereby improving the flexibility of the encoding method. Moreover, when there is latency in the traditional encoding of the audio frame to be encoded or the AI ​​encoding of redundant audio data, it is difficult to reconstruct the entire redundant audio frame by only encoding the redundant audio frame. Therefore, by encoding redundant audio data with a variable number of sampling points, the requirement for accurate reconstruction of lost audio frames can be met, ensuring the complete and accurate reconstruction of lost audio frames.

[0014] The method provided in this application can flexibly select redundant encoded data from previous and / or subsequent audio frames for encoding based on the latency requirements in audio wireless transmission scenarios. For example, if a lower latency is required, redundant audio data can be selected from previous audio frames; if a longer latency is required, redundant audio data can be selected from subsequent audio frames.

[0015] In one implementation, the redundant audio data also includes data of arbitrary length from the audio frame to be encoded.

[0016] In short-range wireless transmission scenarios, the encoder can package the encoded data of the audio frame to be encoded and the encoded data of the corresponding redundant audio data of the audio frame into a single data packet to obtain the bitstream. Thus, during the short-range wireless transmission of the bitstream from the encoder to the decoder, if a data packet is lost, not only is the audio frame to be encoded lost, but also the redundant audio data belonging to the same data packet is lost. Therefore, the redundant audio data corresponding to the audio frame to be encoded may not include the complete audio frame to be encoded, to avoid the decoder being unable to reconstruct the audio frame. Based on this, the redundant audio data corresponding to the audio frame to be encoded can be extracted from the previous and / or subsequent audio frames of the audio frame to be encoded. Of course, in scenarios where traditional encoders have latency, the redundant audio data corresponding to the audio frame to be encoded can also include audio data with any number of sampling points in the audio frame to be encoded. For example, if the audio frame to be encoded has 160 sampling points, then the audio data with any number of sampling points can be audio data with a number greater than 0 and less than 160 sampling points in the audio frame to be encoded.

[0017] In long-distance wireless transmission scenarios, the encoding end can package the encoded data of the audio frame to be encoded into a single data packet, and package the encoded data of the corresponding redundant audio data of the audio frame to be encoded into another data packet to obtain a bitstream. That is, the loss of the encoded data of the audio frame to be encoded will not directly lead to the loss of its redundant audio data set. Therefore, in this scenario, the redundant audio data corresponding to the audio frame to be encoded can include not only audio data extracted from previous and / or subsequent audio frames, but also the complete audio frame to be encoded (or audio data from audio frames with any number of sampling points (not the complete audio frame)). Thus, even if the encoded data of the audio frame to be encoded is lost during transmission, the decoding end can still use the redundant audio data to reconstruct the audio data of some or all of the sampling points of the audio frame to be encoded. Two bitstreams are obtained and transmitted separately to the decoding end. When packet loss is detected at the decoding end, the lost audio frame can be recovered based on the redundant audio data.

[0018] The method provided in this application can flexibly select data of any length in the audio frame to be encoded as part of the redundant audio data corresponding to the audio frame under different scenarios of short-range and long-range wireless transmission, as well as under different conditions of whether the traditional encoder has latency and the latency magnitude, so as to meet the audio encoding and decoding requirements under various scenarios, resist packet loss, and improve anti-interference capability.

[0019] In one implementation, the method further includes: encoding a first data length of the redundant audio data into a bitstream, the first data length including the number of complete audio frames and the number of sampling points of incomplete audio frames.

[0020] In this way, the encoding end can encode the first data length of the redundant data into the bitstream, so that the decoding end can obtain the first data length of the redundant audio data encoded in the bitstream. Since the first data length includes the number of complete audio frames and the number of sampling points of incomplete audio frames, the decoding end can determine which audio frames the redundant audio data belongs to based on the first data length, and thus use it to reconstruct which lost audio frames.

[0021] In one implementation, before encoding redundant audio data using an AI model to obtain the bitstream, the method further includes: acquiring channel state information, which indicates the packet loss rate of the transmission channel of the bitstream; and acquiring a first data length based on the channel state information.

[0022] In other words, the implementation method of this application can flexibly and dynamically determine the first data length of redundant audio data according to the interference of the channel.

[0023] When the channel state information indicates a high packet loss rate, a longer first data length can be obtained; when the channel state information indicates a low packet loss rate, a shorter first data length can be obtained. The Nth frame is the audio frame to be encoded. When the channel state information indicates a high packet loss rate, the first data length can be multiple frames or multiple sample points, such as two frames, three frames, three frames plus half the sample points of one frame, etc. When the channel state information indicates a low packet loss rate, the first data length can be one frame, one frame plus half the sample points of one frame, etc.

[0024] The encoding method implemented in this application can dynamically determine the first data length of the redundant audio data to be encoded based on channel state information, thereby dynamically adjusting the data length of the encoded redundant audio data. In this way, when the transmission channel is severely interfered with, more redundant audio data can be encoded, allowing the bitstream to include more redundant information, thus improving the anti-interference capability of the audio data during transmission. Therefore, even under severe interference conditions, the decoding end can still receive sufficient redundant information to recover lost data.

[0025] In one implementation, the method further includes: obtaining a first bitrate based on the encoding bitrate of the audio frame to be encoded, wherein the first bitrate is the encoding bitrate of the AI ​​model; and using the AI ​​model to encode redundant audio data according to the first bitrate to obtain first encoded data.

[0026] In other words, the implementation method of this application can flexibly and dynamically determine the bitrate used by the AI ​​model for encoding based on the total bitrate of the audio frames to be encoded.

[0027] Without affecting the audio, if the total bit rate of the audio frame to be encoded is high, a higher first bit rate can be obtained, thereby encoding more redundant information into the bitstream and further improving the anti-interference ability of the audio data during transmission.

[0028] Thus, in the implementation of this application, when using an AI encoding model to encode redundant audio data, the encoding bitrate can be flexibly adjusted according to the change in the total bitrate of the audio frames, thereby enabling the bitstream to carry more redundant information and improving the anti-interference capability of the audio data during transmission.

[0029] In one implementation, the code rate of the second encoded data is a second code rate, and the sum of the first code rate and the second code rate is the encoded code rate.

[0030] In other words, the implementation method of this application can flexibly and dynamically allocate the first bitrate and the second bitrate according to the total bitrate of the audio frame to be encoded.

[0031] Thus, in the implementation of this application, when performing traditional encoding on audio frames and AI encoding on redundant audio data, the bitrate of traditional encoding and the bitrate of AI encoding can be flexibly adjusted within the total bitrate of the audio frames. This allows the bitstream to carry more redundant information and improve the anti-interference capability of audio data during transmission without significantly affecting the audio quality of the traditional encoding bitrate.

[0032] Secondly, this application also provides a decoding method. The method includes: acquiring first information indicating that a first audio frame in a bitstream is lost; acquiring first reconstructed data of redundant audio data based on the bitstream, the redundant audio data including the first audio frame, the first reconstructed data being obtained by decoding the first encoded data of the redundant audio data in the bitstream using an AI model, the bitstream also including second encoded data of a second audio frame corresponding to the redundant audio data; and obtaining second reconstructed data of the first audio frame based on the first reconstructed data.

[0033] For ease of explanation, the following implementation will also refer to the first audio frame as a "lost frame" and the second audio frame as a "related frame". The method will be illustrated using an example applied to the decoding end.

[0034] Thus, in the event that the first audio frame in the bitstream is lost (represented by the lost frame), the decoding method of this application can obtain the first reconstructed data containing the redundant audio data of the lost frame based on the bitstream. The first reconstructed data is obtained by decoding the first encoded data of the redundant audio data in the bitstream using an AI model (e.g., an AI decoding model). This enables the rapid recovery of lost audio frames in the event of channel interference, allowing the decoding end to obtain continuous audio frames and improve anti-interference capability.

[0035] In one implementation, the second audio frame includes at least one of the following: a previous audio frame of the first audio frame, and a subsequent audio frame of the first audio frame.

[0036] In this implementation, to reconstruct the lost audio frame, the decoding method can obtain the reconstructed data of the lost audio frame from the first reconstructed data of the redundant audio data (containing the lost audio frame) corresponding to the previous and / or subsequent frames of the lost audio frame. This allows for flexible selection of the reconstructed data of the lost audio frame from the redundant audio data corresponding to the previous and / or subsequent frames, based on the latency requirements of wireless audio transmission scenarios. This enables the decoding end to utilize the reconstructed data of the redundant audio data obtained from AI model decoding to restore the lost audio data in packet loss scenarios, thereby improving the channel's anti-interference and packet loss capabilities, providing users with a continuous audio experience, and reducing audio stuttering.

[0037] In one implementation, the redundant audio data also includes data of arbitrary length from the first audio frame.

[0038] In the implementation of this application, the redundant audio data corresponding to a single audio frame can contain a first audio frame of arbitrary length (i.e., a lost frame). Thus, the redundant audio data corresponding to a single audio frame can contain complete audio frames or incomplete audio frames, making the granularity of the encoded redundant audio data in the bitstream not limited to frame granularity but also usable at the sampling point level. Therefore, in different scenarios of short-range and long-range wireless transmission, and considering the presence and magnitude of latency in traditional encoders, the decoding end can flexibly decode the first encoded data of the redundant audio data corresponding to a single audio frame to obtain data of arbitrary length in the current frame, meeting the audio encoding and decoding requirements of various scenarios, resisting packet loss, and improving anti-interference capabilities.

[0039] In one implementation, obtaining first reconstructed data of redundant audio data based on the bitstream includes: determining a second audio frame corresponding to a first audio frame; and using an AI model to decode the first encoded data of the redundant audio data corresponding to the second audio frame in the bitstream to obtain the first reconstructed data.

[0040] In the implementation of this application, by first determining the second audio frame (i.e., the "associated frame") corresponding to the first audio frame, and then performing AI decoding on the first encoded data of the redundant audio data corresponding to the second audio frame, the first reconstructed data of the redundant audio data corresponding to the second audio frame is obtained, and then the first reconstructed data of the first audio frame (i.e., the "lost frame") is obtained from it. This allows only the encoded data that helps to reconstruct the first audio to be decoded, avoiding useless decoding operations and improving the overall decoding efficiency.

[0041] In one implementation, the second audio frame includes the previous audio frame; based on the bitstream, first reconstructed data of redundant audio data is obtained, including: obtaining first reconstructed data of redundant audio data corresponding to the previous audio frame that has been decoded and saved by the bitstream.

[0042] In the implementation of this application, when an audio frame is lost (hereinafter referred to as "lost frame"), the decoding end can directly extract the first reconstructed data containing the "lost frame" from the reconstructed data of the redundant audio data obtained by the cached decoding, without having to decode the currently received or future received or already received bitstream to obtain the reconstructed data of the "lost frame", thereby improving the overall decoding efficiency.

[0043] In one implementation, the method further includes: saving the first reconstruction data.

[0044] In the implementation of this application, when an audio frame is lost (hereinafter referred to as a "lost frame"), the first reconstructed data containing the redundant audio data of the lost frame can be saved. Thus, if subsequent frames after the first audio frame are lost at the decoding end, the decoding end can directly extract part or all of the reconstructed data of the lost subsequent frames from the saved reconstructed data of the decoded redundant audio data, without needing to decode the currently received, future, or already received bitstream to obtain the reconstructed data of the lost subsequent frames, thereby improving decoding efficiency.

[0045] In one implementation, the method further includes: decoding the target data length of the first audio frame in the redundant audio data from the bitstream, wherein the target data length indicates the amount of data of the first audio frame included in the redundant audio data.

[0046] In this way, the specific amount of data of the lost first audio frame contained in the redundant audio data can be accurately determined, so as to achieve accurate reconstruction of the first audio frame.

[0047] In one implementation, obtaining second reconstructed data of a first audio frame based on the first reconstructed data includes: obtaining second reconstructed data of the first audio frame based on the target data length and the first reconstructed data.

[0048] In this way, the decoding end can decode the bitstream to obtain the target data length of the "lost frame" in the redundant audio data of the "associated frame". This target data length can be, for example, information such as the number of sampling points or frames of the "lost frame" in the redundant audio data. Then, the decoding end can combine the first reconstructed data of the redundant audio data with the target data length to obtain the second reconstructed data of the "lost frame", so as to ensure the accurate and complete reconstruction of the "lost frame" at the decoding end.

[0049] In one implementation, the method further includes: acquiring third reconstructed data of a third audio frame adjacent to the first audio frame; and smoothing the third reconstructed data and the second reconstructed data to obtain audio sequence data.

[0050] In the implementation of this application, by smoothing the reconstructed data of the lost first audio frame and the reconstructed data of the adjacent audio frames of the lost first audio frame, problems such as unnatural sound, uneven connection of adjacent one or more frames, and unnatural transition in the reconstructed data of each audio frame obtained by decoding can be avoided, making the sound of the obtained audio sequence more natural and smooth.

[0051] Thirdly, this application also provides an encoding apparatus, comprising: a data acquisition module for acquiring redundant audio data of a first data length corresponding to an audio frame to be encoded; and a data encoding module for encoding the redundant audio data using an AI model to obtain a bitstream, wherein the bitstream includes first encoded data of the redundant audio data and second encoded data of the audio frame to be encoded, and the compression ratio of the AI ​​model is higher than the compression ratio corresponding to the audio frame to be encoded.

[0052] In one implementation, the redundant audio data is extracted from previous audio frames and / or subsequent audio frames of the audio frame to be encoded, and the first data length includes any number of frames, or any number of frames plus any number of sampling points.

[0053] In one implementation, the redundant audio data also includes data of arbitrary length from the audio frame to be encoded.

[0054] In one implementation, the data encoding module is further configured to: encode a first data length of the redundant audio data into the bitstream, the first data length including the number of complete audio frames and the number of sampling points of incomplete audio frames.

[0055] In one implementation, the data acquisition module is further configured to: acquire channel state information, which indicates the packet loss rate of the transmission channel of the code stream; and acquire a first data length based on the channel state information.

[0056] In one implementation, the data acquisition module is further configured to: obtain a first bitrate based on the encoding bitrate of the audio frame to be encoded, wherein the first bitrate is the encoding bitrate of the AI ​​model; the data encoding module is further configured to: use the AI ​​model to encode redundant audio data according to the first bitrate to obtain first encoded data.

[0057] In one implementation, the bitrate of the second encoded data is the second bitrate, and the sum of the first bitrate and the second bitrate is the encoding bitrate.

[0058] It should be understood that the encoding device may also include other modules such as a communication module.

[0059] It should be understood that the encoding device can also be used to execute the encoding methods in the first aspect or any possible implementation of the first aspect, which will not be elaborated here.

[0060] Fourthly, this application also provides a decoding apparatus, comprising: an information acquisition module for acquiring first information indicating that a first audio frame in a bitstream is lost; a decoding module for acquiring first reconstructed data of redundant audio data based on the bitstream, the redundant audio data including the first audio frame, the first reconstructed data being obtained by decoding the first encoded data of the redundant audio data in the bitstream using an AI model, the bitstream also including second encoded data of a second audio frame corresponding to the redundant audio data; and the decoding module further for obtaining second reconstructed data of the first audio frame based on the first reconstructed data.

[0061] In one implementation, the second audio frame includes at least one of the following: a previous audio frame of the first audio frame, and a subsequent audio frame of the first audio frame.

[0062] In one implementation, the redundant audio data also includes data of arbitrary length from the first audio frame.

[0063] In one implementation, the decoding module is specifically used to: determine the second audio frame corresponding to the first audio frame; and use an AI model to decode the first encoded data of the redundant audio data corresponding to the second audio frame in the bitstream to obtain the first reconstructed data.

[0064] In one implementation, the second audio frame includes the previous audio frame, and the decoding module is specifically used to: obtain first reconstructed data of the redundant audio data corresponding to the previous audio frame that has been decoded and saved by the bitstream.

[0065] In one implementation, the decoding module is also used to: save the first reconstructed data.

[0066] In one implementation, the decoding module is further configured to: decode from the bitstream to obtain the target data length of the first audio frame in the redundant audio data, wherein the target data length indicates the amount of data of the first audio frame included in the redundant audio data.

[0067] In one implementation, the decoding module is specifically used to: obtain the second reconstructed data of the first audio frame based on the target data length and the first reconstructed data.

[0068] In one implementation, the decoding module is further configured to: acquire third reconstructed data of a third audio frame adjacent to the first audio frame; and perform smoothing processing on the third reconstructed data and the second reconstructed data to obtain audio sequence data.

[0069] It should be understood that the decoding device may also include other modules such as a communication module.

[0070] It should be understood that the decoding device can also be used to execute the decoding method in the second aspect or any possible implementation of the second aspect, which will not be elaborated here.

[0071] Fifthly, this application provides an encoder, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, cause the one or more processors to implement the method as described in the first aspect or any implementation thereof.

[0072] In a sixth aspect, this application provides a decoder, comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions that, when executed by the processor, cause the one or more processors to implement the method as described in the second aspect or any implementation thereof.

[0073] Seventhly, this application provides a bitstream generated according to the method in the first aspect or any implementation thereof.

[0074] Eighthly, this application provides a computer-readable storage medium including a computer program that, when executed on a device, causes the device to perform the method described in the first aspect or any implementation thereof.

[0075] Ninthly, this application provides a computer-readable storage medium including a computer program that, when executed on a device, causes the device to perform the method described in the second aspect or any implementation thereof.

[0076] In a tenth aspect, this application provides a computer program that, when executed by a device, performs the method described in the first aspect or any implementation thereof.

[0077] In one aspect, this application provides a computer program that, when executed by a device, performs the method in the second aspect or any implementation thereof.

[0078] In a twelfth aspect, this application provides a computer program product comprising computer program code, which, when executed on a device, causes the device to perform the method described in the first aspect or any implementation thereof.

[0079] In a thirteenth aspect, this application provides a computer program product comprising computer program code, which, when run on a device, is used to execute the method in the second aspect or any implementation thereof described above.

[0080] In a fourteenth aspect, this application provides a computer-readable storage medium storing a bitstream generated according to the method in any implementation of the first aspect described above.

[0081] In a fifteenth aspect, an apparatus for storing a bitstream is provided, the apparatus comprising: a transceiver unit and a storage unit, the transceiver unit being configured to receive a bitstream generated according to the method described in accordance with the first aspect or any implementation thereof, and the storage unit being configured to store the bitstream.

[0082] In a sixteenth aspect, an apparatus for transmitting a bitstream is provided, the apparatus comprising: a storage unit and a transceiver unit, the storage unit being configured to store a bitstream generated according to the method described in accordance with the first aspect or any implementation thereof, and the transceiver unit being configured to transmit the bitstream.

[0083] In a seventeenth aspect, a system for distributing bitstreams is provided, the system comprising: at least one storage medium for storing a bitstream generated according to the method described in accordance with the first aspect or any implementation thereof; and a streaming media device for obtaining the bitstream from the at least one storage medium and transmitting the bitstream, wherein the streaming media device includes a content server or a content distribution server.

[0084] Eighteenthly, a transcoding system is provided, the transcoding system comprising: at least one storage medium for storing a bitstream generated by the method according to the first aspect or any implementation thereof; and a transcoding device for acquiring the bitstream from the at least one storage medium and transcoding the bitstream.

[0085] In one implementation, the transcoding device can convert the bitstream into Moving Picture Experts Group-4 (MPEG-4) audio, MPEG audio layer 3 (MP3) audio, Audio Video Interleave (AVI) audio, Digital Audio Video (DAV) audio, etc. Attached Figure Description

[0086] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of this application;

[0087] Figure 2 is a flowchart of an encoding method provided in an embodiment of this application;

[0088] Figure 3a is a schematic diagram of eleven types of redundant audio data provided in the embodiments of this application;

[0089] Figure 3b is a schematic diagram of three types of redundant audio data provided in the embodiments of this application;

[0090] Figure 4 is a schematic diagram of a data structure provided in an embodiment of this application;

[0091] Figure 5a is a flowchart of an L2HC encoding method provided in an embodiment of this application;

[0092] Figure 5b is a flowchart of an L2HC decoding method provided in an embodiment of this application;

[0093] Figure 6 is a schematic diagram of an L2HC encoding / decoding process provided in an embodiment of this application;

[0094] Figure 7a is a schematic diagram of the structure of an AI coding model provided in an embodiment of this application;

[0095] Figure 7b is a schematic diagram of the structure of an AI decoding model provided in an embodiment of this application;

[0096] Figure 8a is a schematic diagram of the encoding process of an AI encoding model provided in an embodiment of this application;

[0097] Figure 8b is a schematic diagram of the decoding process of an AI decoding model provided in an embodiment of this application;

[0098] Figure 9 is a flowchart of another encoding method provided in an embodiment of this application;

[0099] Figure 10 is a flowchart of another encoding method provided in an embodiment of this application;

[0100] Figure 11 is a flowchart of a decoding method provided in an embodiment of this application;

[0101] Figure 12 is a flowchart of a decoding method provided in an embodiment of this application;

[0102] Figure 13 is a flowchart of a decoding method provided in an embodiment of this application;

[0103] Figure 14 is a schematic diagram of encoded data and decoded data provided in an embodiment of this application;

[0104] Figure 15 is a schematic diagram of another type of encoded and decoded data provided in an embodiment of this application;

[0105] Figure 16 is a schematic diagram of an encoding device provided in an embodiment of this application;

[0106] Figure 17 is a schematic diagram of the structure of a decoding device provided in an embodiment of this application. Detailed Implementation

[0107] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0108] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0109] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0110] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0111] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0112] Before introducing the technical solution of this application, some technical terms involved in this application will be explained and clarified first:

[0113] 1) Audio Frame: An audio frame is a unit of audio data processing. It is divided along a time axis, and an audio frame contains audio data within a certain time period. For example, an audio frame may contain 10 milliseconds (ms) of audio data.

[0114] 2) Current frame: On the encoding side, "current frame" refers to the audio frame currently being encoded; on the decoding side, "current frame" refers to the audio frame currently being decoded.

[0115] 3) Previous frame: On the encoding side, "previous frame" refers to the encoded audio frame that is encoded before the current frame and whose encoding order is adjacent to the current frame; on the decoding side, "previous frame" refers to the decoded audio frame that is decoded before the current frame and whose decoding order is adjacent to the current frame.

[0116] 4) Next frame: On the encoding side, "next frame" refers to the audio frame that is encoded after the current frame and whose encoding order is adjacent to the current frame; on the decoding side, "next frame" refers to the audio frame that is decoded after the current frame and whose decoding order is adjacent to the current frame.

[0117] 5) Previous frame (also known as previous audio frame): At the encoding end, "previous frame" refers to the audio frame that has been encoded before the current frame; at the decoding end, "previous frame" refers to the audio frame that has been decoded before the current frame.

[0118] 6) Subsequent frames (also known as subsequent audio frames): At the encoding end, a "subsequent frame" refers to an audio frame that is encoded after the current frame and has not been encoded; at the decoding end, a "subsequent frame" refers to an audio frame that is decoded after the current frame and has not been decoded.

[0119] 7) Total bitrate: refers to the sum of the bitrate used to encode the current frame and the bitrate used to encode the redundant audio data corresponding to the current frame.

[0120] 8) Pulse code modulation (PCM): a technique for converting analog signals into digital signals. 9) PCM data: in this application, refers to digital audio signals generated based on PCM technology.

[0121] The process of generating PCM data may include the following steps:

[0122] Sampling: Measuring the amplitude (voltage value) of an analog signal over consecutive time intervals converts the analog signal into a discrete time signal. The value at each time point is called a sample.

[0123] Quantization: Mapping the sampled amplitude values ​​to a finite digital range. This process introduces a certain error (quantization error), usually expressed as bit depth.

[0124] Encoding: Converting the quantized values ​​into binary format to form PCM data.

[0125] 9) An artificial intelligence (AI) model is a mathematical or computational model that learns patterns and rules from data to make predictions or decisions to perform specific tasks. AI models can be implemented using various technologies, including but not limited to machine learning, deep learning, and reinforcement learning.

[0126] 10) Analysis window of time T: refers to a time window of length T used to process audio signals.

[0127] 11) Packet loss rate: This refers to the proportion of data packets lost during transmission out of the total number of data packets sent. A high packet loss rate can lead to audio interruptions, distortion, or delays.

[0128] 12) Bandwidth: This refers to the maximum data transmission rate that a transmission channel can support. Higher bandwidth means more data can be transmitted, thus supporting higher quality audio formats (such as lossless audio).

[0129] 13) Jitter: This refers to the variation in the time it takes for a data packet to travel from the encoding end to the decoding end. Significant jitter can affect the stability of the audio stream, causing stuttering or skipping during playback.

[0130] In the audio field, whether it's voice or music audio, audio encoding / decoding and transmission are involved. Audio transmission scenarios can include long-range and short-range wireless transmission. For example, long-range wireless transmission can be based on technologies such as Voice over Internet Protocol (VoIP) and cellular communication, while short-range wireless transmission can be based on technologies such as Bluetooth, infrared, Wireless Fidelity (Wi-Fi), and Wi-Fi.

[0131] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of this application.

[0132] As shown in Figure 1, the encoding end can encode audio data to obtain a bitstream, and then transmit the bitstream to the decoding end 1 via long-distance wireless transmission, and / or, the encoding end can transmit the encoded bitstream to the decoding end 2 via short-distance wireless transmission. After receiving the bitstream, the decoding end (e.g., decoding end 1 or decoding end 2) can decode the bitstream to recover the audio data. In some embodiments, the decoding end (e.g., decoding end 1 or decoding end 2) can also play the decoded audio data.

[0133] In some embodiments, the encoding end and decoding end shown in FIG1 can be implemented as either a client or a server.

[0134] The client can be software (such as an encoder or decoder), an application, a browser, an in-vehicle infotainment system, a terminal device, etc. When the client is implemented as a terminal device, the client can include, but is not limited to: mobile phone, personal computer (PC), virtual reality (VR) device, augmented reality (AR) device, tablet computer, laptop computer, wireless music headphones, wireless headphones, wireless neckband headphones, true wireless stereo (TWS) headphones, etc.

[0135] The client can also be a smart TV, smart speaker, mobile internet device (MID), wearable device (such as a smartwatch, smart glasses, or smart helmet), smart car, wireless terminal device in industrial control, wireless terminal device in self-driving, wireless terminal device in remote medical surgery, wireless terminal device in smart grid, wireless terminal device in transportation safety, wireless terminal device in smart city, wireless terminal device in smart home, etc. The following embodiments do not impose any special limitations on the specific form of the client.

[0136] Server-side functionality can be implemented using software or hardware.

[0137] In the first example, when the functionality of the server is implemented through software, the server may be, for example, an application running on a computing instance (such as an encoder or decoder, or an application that implements the encoding and decoding methods of this application), which may be, for example, a virtual machine, a container, or a host.

[0138] In the second example, when the functionality of the server is implemented through hardware, the server can be implemented through at least one physical device including a processor, such as a server.

[0139] The processor can be a central processing unit (CPU) or a graphics processing unit (GPU), or it can be any type of processor or any combination thereof, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-chip (SoC), a software-defined infrastructure (SDI) chip, an AI chip, or a data processing unit (DPU).

[0140] Furthermore, the number of processors included in the server can be arbitrary, and the types of processors included can be one or more. The specific number and types of processors can be set according to the actual business needs of the application, and this application does not impose any restrictions on this.

[0141] The physical device can be a server (such as a central server or an edge server), a base station, a relay device, a satellite device, etc., and there are no restrictions here.

[0142] In the third example, when the server-side functionality is implemented through hardware, the server can also be a computing cluster comprising multiple computing nodes. Furthermore, these multiple computing nodes can communicate through the at least one switching node. Exemplarily, a computing node can be a computing server including an accelerator card. This accelerator card can be, for example, a deep-learning processing unit (DPU), a GPU, a neural-network processing unit (NPU), or a tensor processing unit (TPU), or other types of accelerator cards. Alternatively, a computing node can be a computing server including a general-purpose processor (such as a CPU).

[0143] The following describes the interaction process between the encoder and decoder shown in Figure 1, using different application scenarios.

[0144] For example, as shown in Figure 1, both the encoding and decoding ends can be implemented as clients. Taking a mobile phone as an example, when mobile phone 1 and mobile phone 2 conduct a VoIP voice call, mobile phone 1 can collect the voice audio signal of user 1 and encode it to obtain a bitstream. Then, mobile phone 1 can send the bitstream to mobile phone 2 via VoIP. Mobile phone 2 can receive the bitstream sent by mobile phone 1 and decode it to obtain the voice audio signal of user 1. Optionally, mobile phone 2 can play the voice audio signal through a speaker. In this process, mobile phone 1 is the encoding end and mobile phone 2 is the decoding end. Similarly, mobile phone 2 can also collect the voice audio signal of user 2 and encode it to send to mobile phone 1. The implementation process is the same. In this way, mobile phone 1 is implemented as the decoding end and mobile phone 2 is implemented as the encoding end.

[0145] For example, as shown in Figure 1, both the encoding end and the decoding end can be implemented as clients. For instance, the encoding end can be implemented as mobile phone 1, and the decoding end 2 can be implemented as earphone 1. Mobile phone 1 connects to earphone 1 via Bluetooth to play music audio. Specifically, mobile phone 1 can encode the music audio to be played to obtain a bitstream; then, mobile phone 1 can transmit the bitstream to earphone 1 via Bluetooth. Earphone 1 can receive the bitstream sent by mobile phone 1 and decode it to obtain the music audio to be played; then, earphone 1 can play the obtained music audio. In this process, mobile phone 1 is the encoding end, and earphone 1 is the decoding end.

[0146] For example, the encoding end shown in Figure 1 can be implemented as a client, and the decoding end can be implemented as a server. For instance, the encoding end shown in Figure 1 can be implemented as a mobile phone 1, and the decoding end can be implemented as a media server 1 of an application 1 (e.g., a music recording application). The mobile phone 1 and the media server 1 can establish a long-range wireless connection (e.g., Wi-Fi) to allow the mobile phone 1 to send audio data 1 to the media server 1 through the application 1. Specifically, the mobile phone 1 can encode the audio data 1 uploaded by the user through the application 1 to obtain a bitstream; then, the mobile phone 1 can send the bitstream to the media server 1 via long-range wireless transmission; the media server 1 can receive the bitstream sent by the mobile phone 1 and decode it to obtain the audio data 1, thereby enabling the mobile phone 1 to upload the user's recorded audio. In this process, the mobile phone 1 is the encoding end, and the media server 1 is the decoding end.

[0147] For example, the encoding end shown in Figure 1 can also be implemented as a server, and the decoding end can be implemented as a client. For instance, the encoding end shown in Figure 1 can be implemented as a media server 1 for an application 1, and the decoding end can be implemented as a mobile phone 2. Continuing with the example from the previous paragraph, after decoding the audio data 1, the media server 1 can enhance the audio quality of the audio data 1. The media server 1 can then send the enhanced audio data 1 to the mobile phone 1. Specifically, when a user wants to download or play the audio data 1 through the mobile phone 2, the media server 1 can encode the enhanced audio data 1 to obtain a bitstream; then, it can send the bitstream to the mobile phone 2 via long-distance wireless transmission; after receiving the bitstream, the mobile phone 2 can decode it to obtain the enhanced audio data 1, thereby enabling the mobile phone 2 to play and download the audio data uploaded by the mobile phone 1. In this process, the media server 1 is the encoding end, and the mobile phone 2 is the decoding end.

[0148] For example, as shown in Figure 1, both the encoding and decoding ends can be implemented as servers. The explanation will focus on a cloud server as the server. For instance, in a cloud computing scenario, cloud server 1 and cloud server 2 can establish a long-range wireless connection. Cloud server 1 can encode the audio uploaded by client 1 to obtain a bitstream; then, it sends the bitstream to cloud server 2 via long-range wireless transmission. Cloud server 2 can receive the bitstream and decode it to obtain the audio data uploaded by client 1, thereby enabling shared storage of the same audio data across different cloud servers, ensuring remote access and backup of the audio data. In this process, cloud server 1 is the encoding end, and cloud server 2 is the decoding end. Similarly, cloud server 2 can also encode the audio uploaded by client 2 and send it to cloud server 1; the implementation process is similar and will not be elaborated here.

[0149] In this embodiment, Figure 1 illustrates the interaction between a single encoder and two decoders. In other embodiments, an encoder can interact with one or more decoders via long-range wireless transmission (or short-range wireless transmission); and multiple encoders can interact with one or more decoders via long-range wireless transmission (or short-range wireless transmission). This application does not limit the number of encoders and decoders in long-range or short-range wireless transmission scenarios, and the number can be flexibly varied according to the application scenario.

[0150] In the application scenario shown in Figure 1, during the transmission of audio data from the encoder to the decoder, the audio frames sent by the encoder may be lost during transmission due to the influence of the channel environment. Moreover, the lost audio frames are random and unpredictable, which causes the decoder to be unable to receive continuous audio frames correctly, resulting in audio stuttering, discontinuity and other problems, thus affecting the user's listening experience.

[0151] To address audio stuttering and discontinuity caused by audio frame loss, related technologies offer an in-band redundancy coding method that resists packet loss by repeatedly encoding audio frames. This method encodes the current frame at a certain bitrate and simultaneously encodes redundant audio frames (such as the previous frame) at the same bitrate (or a slightly lower bitrate). Thus, if the previous frame is lost during transmission, the decoder can retrieve the data from the previous frame by decoding the current frame's bitstream, thus mitigating packet loss. In other words, in the encoding schemes of related technologies, the bitrate of the current frame is the same as the bitrate of its redundant audio frames, or the bitrate of the redundant audio frames is slightly lower than the bitrate of the current frame.

[0152] For example, if the encoder is set to a total bitrate of 64 kilobits per second (Kbps) for audio encoding, and the channel capacity for transmitting this bitstream is variable (e.g., the channel capacity supports a bitstream of 128 Kbps), then the total bitstream can be adjusted to 128 Kbps. The encoder can encode the current frame at 64 Kbps and its redundant audio frames at the same rate, resulting in a total bitrate of 128 Kbps. This doubles the total bitrate to encode the redundant audio frames, increasing bandwidth usage. Consequently, the encoding end needs to transmit more data in the same amount of time, increasing the transmission burden and potentially causing transmission delays.

[0153] For example, the encoder might be set to a total bitrate of 64Kbps for audio encoding, but the maximum channel capacity for transmitting this bitstream is also 64Kbps. This means the channel can only support a maximum bitrate of 64Kbps. Therefore, with a limited total bitrate for audio data encoding, to encode redundant audio frames, these redundant frames must occupy approximately half the current frame's bitrate. For instance, the current frame's bitrate can be halved. Specifically, the encoder can use in-band redundancy coding to encode both the current frame and its redundant audio frames at 32Kbps. This maintains a total bitrate of 64Kbps, meeting the channel capacity requirement even with redundancy coding. Alternatively, the current frame can be encoded at 34Kbps, and the redundant audio frames at 30Kbps. However, encoding a bitstream by reducing the bitrate of the current frame results in redundant audio frames occupying the current frame's bitrate and transmission bandwidth. This leads to audio signal loss in the decoded data of the current frame, thus reducing sound quality and affecting the user's listening experience. This is especially true in low-bitrate audio transmission scenarios such as Bluetooth, where the audio quality of the decoded data will be significantly reduced.

[0154] Furthermore, in the encoding schemes of the aforementioned related technologies, the redundant audio data corresponding to the current frame can only be one audio frame. Therefore, to avoid poor sound quality and increased bandwidth usage, redundant encoding is generally not performed on every frame. Instead, one or several audio frames are selected from the audio sequence, and their corresponding redundant audio frames are encoded separately. Consequently, in complex scenarios (such as continuous packet loss scenarios), this scheme does not possess strong resilience against packet loss.

[0155] Based on this, in order to improve the packet loss resistance of audio data transmission without significantly increasing the audio encoding bitrate or degrading the sound quality, embodiments of this application provide an encoding method and a decoding method. In this method, an AI model can be used to redundantly encode the redundant audio data corresponding to the current frame to obtain a bitstream. Thus, the bitstream includes not only the encoded data of the current frame but also the encoded data of redundant audio data of a first data length. Furthermore, the data length of the redundant audio data is not limited to a single audio frame, thereby enabling strong packet loss resistance (e.g., in scenarios of continuous packet loss).

[0156] Because the compression ratio of the AI ​​model is higher than that of the current frame (such as the compression ratio used for traditional encoding of the current frame), even if the length of the redundant audio data is long, the encoded data of the redundant audio data compressed by the AI ​​model only occupies a small amount of bandwidth. This can resist packet loss while avoiding the problem of excessive bandwidth occupation of the bitstream due to encoding redundant audio data.

[0157] Furthermore, an AI model is used to encode redundant audio data separately at a higher compression rate, instead of using half the encoding bitrate of the current frame in a traditional encoder. This reduces the proportion of the redundant audio data's bitrate in the total audio frame bitrate while still providing robustness against packet loss. Consequently, this improves packet loss resistance without significantly degrading sound quality. The decoder can then decode this redundant audio data to obtain continuous, complete, and high-quality audio data without significantly reducing the listening experience.

[0158] The encoding method provided in the embodiments of this application is described below.

[0159] Please refer to Figure 2, which is a flowchart illustrating an encoding method provided in an embodiment of this application. This encoding method can be applied to the encoding end shown in Figure 1. As shown in Figure 2, the encoding method may include, but is not limited to, the following steps: S201, S202, and optionally S301.

[0160] The encoding end may include, but is not limited to, an AI encoder, and optionally, a traditional encoder.

[0161] The encoding methods of the various embodiments of this application can be implemented by an AI encoder. Optionally, some steps can be implemented by a conventional encoder (e.g., S301 below).

[0162] The AI ​​encoder executes steps S201 and S202 to encode the redundant audio data corresponding to the current frame, thereby obtaining a bitstream (also called a hybrid bitstream) that includes the second encoded data of the current frame and the first encoded data of the redundant audio data. This utilizes the redundant audio data to resist packet loss in audio data encoded by a conventional encoder, improving anti-interference capabilities. A conventional encoder or an AI encoder executes step S301 below to encode the current frame. That is, when encoding the current frame, the AI ​​encoder of this application can be used, or an encoder currently disclosed or to be disclosed in the future for audio encoding (hereinafter referred to as a "conventional encoder") can be used; this application makes no restriction on this.

[0163] The traditional codec can be the core audio codec, or any audio codec suitable for wireless audio compression and transmission scenarios. For example, the traditional codec can include, but is not limited to, at least one of the following: mainstream Bluetooth audio codecs used for short-range wireless transmission; audio codecs supporting sub-band coding (SBC) format; audio codecs supporting the MPEG advanced audio coding (AAC) standard; audio codecs supporting LDAC format; audio codecs of the aptX series (such as aptX, aptX high definition (HD), and aptX low latency); the next-generation Bluetooth default low-power, low-latency, low-complexity communication codec (LC3) and its derivative high-definition audio codec LC3 plus; and low-latency, low-complexity, high-resolution audio codecs (L2HC); and the Opus codec based on Voice over Internet Protocol (VoIP) used for long-range wireless transmission.

[0164] The lowest coding bit rate (or highest compression rate) of a traditional encoder can be higher than the highest coding bit rate (or lowest compression rate) of an AI encoder.

[0165] An AI encoder can be an encoder for audio encoding that includes an AI model. This application does not limit the internal structure of the AI ​​encoder. The AI ​​model is hereinafter referred to as the AI ​​encoding model, to distinguish it from the AI ​​model at the decoding end (hereinafter referred to as the AI ​​decoding model).

[0166] AI encoding models can be models built on neural networks, or other types of models that can achieve higher compression rates for audio encoding than traditional encoders.

[0167] The network structure of this AI encoding model can be a non-streaming / non-causal fully convolutional network. The term "non-streaming / non-causal" means that the compression ratio is extremely high and multiple audio frames can be compressed at once.

[0168] In some embodiments, the encoding end of this application may include a separate AI encoder and a traditional encoder. Alternatively, the AI ​​encoder may be used as a component of a traditional encoder. This application does not limit the adaptation method between the AI ​​encoder and the traditional encoder, and it should be understood that the AI ​​encoder provided in this application can be adapted to various traditional encoders.

[0169] In the aforementioned related technologies, a certain bitrate can only be allocated in a single codec (such as the traditional codec mentioned above) to encode the redundant audio frame corresponding to the current frame. This is equivalent to the encoding bitrate of the redundant audio frame occupying part of the encoding bitrate of the current frame, thereby significantly reducing the encoding quality of the current frame.

[0170] However, in this embodiment, the current frame and its corresponding redundant audio data can be encoded separately using two encoders, a traditional encoder and an AI encoder, and then mixed into a single bitstream for transmission to the decoding end. This ensures that the encoding of redundant audio data does not occupy a large proportion of the current frame's encoding bitrate, thereby resisting packet loss without significantly reducing audio quality.

[0171] The encoding process shown in Figure 2 is described below:

[0172] S201: The encoding end obtains redundant audio data of the first data length corresponding to the current frame.

[0173] The current frame can be an audio frame to be encoded in an audio sequence, which may include one or more audio frames.

[0174] The audio sequence can be media audio data or speech audio data.

[0175] Media audio data may include, but is not limited to, at least one of the following: music audio data, dubbing audio data, recording audio data, audio data carried in videos, etc.

[0176] The current frame and its corresponding redundant audio data can be digital audio signals, such as audio PCM data, or other formats of digital audio signals such as MP3, standard digital audio file (Waveform Audio File Format, WAV), etc. This application does not impose any restrictions on this.

[0177] If the current frame or its corresponding redundant audio data is not in PCM format, it can be converted to PCM format for audio encoding and decoding, such as converting MP3 to PCM data. There are no restrictions on this.

[0178] As shown in Figure 2, the audio sequence input to the encoder may include: frame 1, ..., frame (N-1), frame N, frame N+1, etc., arranged sequentially according to the encoding order. For example, if frames 1 to (N-1) have already been encoded (e.g., by S301 as shown in Figure 2 below), then the current frame to be encoded may be frame N. In other embodiments, the current frame may be any frame in the audio sequence.

[0179] As shown in Figure 2, each audio frame in the audio sequence is audio PCM data.

[0180] In some embodiments, the encoder (e.g., an AI encoder or a traditional encoder, without limitation) can acquire the audio sequence to be encoded and cache it (e.g., cache it in buffer 1). Thus, when the current frame is the Nth frame, the encoder may have already cached frames 1 to N-1 in buffer 1. Optionally, it may further cache audio frames after the current frame (e.g., the N+1th frame).

[0181] In some embodiments, when the encoding end (e.g., a mobile phone) acquires the aforementioned audio sequence, it can sample the received audio data by setting the number of channels and the audio sampling rate to obtain the audio sequence to be encoded and write it into Buffer1. Traditional encoders can read the current frame from Buffer1 for encoding, while AI encoders can read redundant audio data corresponding to the current frame from Buffer1 for encoding to resist packet loss.

[0182] The number of channels mentioned above can be mono, stereo, or multi-channel.

[0183] The sampling rate of audio can vary depending on the application scenario. For example, in short-range (Bluetooth) voice calls, the sampling rate may be 16 kilohertz (kHz) or 32 kHz; in short-range music audio, the sampling rate is usually 48 kHz or 96 kHz; and in long-range VoIP voice calls, the sampling rate is usually 16 kHz or 48 kHz.

[0184] The encoding end can sample analog audio signals from the received audio data at the appropriate sampling rate and number of channels in the corresponding application scenario, and then convert the sampled analog audio signals into digital audio signals through analog-to-digital conversion to obtain audio sequences (audio PCM data).

[0185] Taking a short-range Bluetooth voice call scenario as an example, if the audio sampling rate (i.e., bandwidth) is 16kHz and the frame length is 10ms, then the number of sampling points in each audio frame of the sampled audio sequence (audio PCM data) is 160 sampling points. This number of points is the number of sampling points in each audio frame (digital audio signal, i.e., audio PCM data) of the audio sequence.

[0186] Taking the case where each audio frame in an audio sequence has the same data length as an example, then each audio frame in the audio sequence has 160 sampling points, for example, a duration of 10ms. In other embodiments, the data lengths of different audio frames in the audio sequence can also be different, that is, the audio sequence to be encoded can be obtained by sampling using multiple sampling rates.

[0187] The redundant audio data obtained by the encoder in this embodiment of the application through the above S201 may be audio data (audio PCM data) of the first data length extracted by the encoder (e.g., AI encoder) from the above Buffer1.

[0188] In some embodiments, the first data length may be a fixed data length. For example, the fixed data length may be the maximum amount of audio data that will be lost when the bitstream encoded by a conventional encoder is transmitted to the decoding end, based on prior knowledge.

[0189] For example, the first data length can be a fixed number of frames such as 3 frames or 5 frames, and / or the first data length can be a given number of sampling points. For example, if the number of sampling points in one frame is 160, then the given number of sampling points can be 220, and there is no specific limitation.

[0190] In some embodiments, the first data length may also be the data length flexibly determined by the encoding end (e.g., an AI encoder) in the current encoding scenario. This application does not limit the method of determining and obtaining the first data length.

[0191] In some embodiments, the redundant audio data extracted by the encoding end may include one or more complete frames, such as 1 frame, 2 frames, 3 frames, etc. And / or, based on the premise that an audio frame can have a fixed number of sampling points, in some embodiments, the redundant audio data extracted by the encoding end may include audio data with any number of sampling points (the number of sampling points in a digital audio signal). Taking an audio frame with 160 sampling points as an example, the audio data of each audio frame involved in the redundant audio data may be audio data with any number of sampling points from 1 to 160. For example, when the redundant audio data includes the audio data of a certain audio frame (e.g., the (N-1)th frame shown in Figure 2), the redundant audio data may include data with any number of sampling points from 1 to 160 sampling points of the (N-1)th frame.

[0192] In this embodiment, the length of the redundant audio data encoded by the encoder can include any number of frames (e.g., one or more frames). This allows the method of this embodiment to support encoding multiple audio frames as redundant audio data, thereby carrying multiple audio frames in the redundant audio data corresponding to the current frame. This helps resist packet loss of multiple frames while using a smaller bitrate. In this way, more redundant audio data can be encoded at a lower bitrate, thus improving the anti-interference capability during audio transmission in scenarios where the total bitrate of audio data encoding is limited (transmission bandwidth is limited).

[0193] Furthermore, the method in this application embodiment also supports encoding at least one audio frame and any number of audio data points within an audio frame of a given length as redundant audio data for redundant encoding. This means the length of the redundant audio data is not limited to a complete frame; it can also be based on at least one frame, adding any number of audio data points (e.g., half a frame, 1 / 3 frame, etc., with any number of audio data points). For example, if the length of an audio frame is 10ms, the length of the redundant audio data can be 13ms, 18ms, etc., including at least one frame and any number of audio data points. The length of the redundant audio data encoded in this application embodiment is not limited by frame granularity and can be flexibly encoded using the number of sampling points as the granularity. This allows for flexible encoding of redundant audio data of different lengths based on application requirements such as latency at the encoding / decoding end, making the length of redundant audio data more flexible and variable, thus improving the flexibility of the encoding method. Furthermore, in cases where there is latency in the traditional encoding of the current frame (e.g., S301 below) or the AI ​​encoding of redundant audio data (e.g., S202 below) (which will be described in the following embodiments), it is difficult to reconstruct the entire redundant audio frame by only encoding the entire redundant audio frame. Therefore, by encoding redundant audio data with a variable number of sampling points, the requirement for accurate reconstruction of lost audio frames can be met, ensuring the complete and accurate reconstruction of lost audio frames.

[0194] In some embodiments, redundant audio data may be extracted from previous frames and / or subsequent frames of the current frame.

[0195] The AI ​​encoder can extract audio data of the first data length from Buffer1 as redundant audio data for the current frame.

[0196] Redundant audio data can be continuous audio data (e.g., at least two consecutive audio frames) or discontinuous audio data (e.g., the first frame and the third frame).

[0197] Example 1: The AI ​​encoder can extract redundant audio data from the previous frame of the current frame (the Nth frame) in Buffer1. For example, it can extract audio data of the first data length from the first frame to the (N-1)th frame as shown in Figure 2 as redundant audio data.

[0198] Please refer to Figure 3a. The audio sequence is as shown in Figure 3a: frame 1... frame N-3, frame N-2, frame N, frame N+1, frame N+2, frame N+3... Each frame of the audio sequence has 160 sampling points. The current frame is frame N. For example, if the first data length is one frame, the AI ​​encoder can extract frame N-1 as shown in Figure 3a(1) as redundant audio data. Or, for example, if the first data length is 200 sampling points, the AI ​​encoder can extract all (160 sampling points) of frame N-1 as shown in Figure 3a(2) and the last 40 sampling points of frame N-2 as redundant audio data. The implementation method is the same when the current frame is an audio frame other than frame N in the audio sequence.

[0199] Because the encoding order of the previous frame is before the current frame, when encoding the current frame (or when encoding the redundant audio data corresponding to the current frame), the previous frame of the current frame has already been cached in Buffer1. Therefore, the AI ​​encoder can improve the speed of obtaining redundant audio data by extracting one or more frames from the previous frames and / or audio data with any number of sampling points as the redundant audio data corresponding to the current frame, thereby improving the encoding efficiency.

[0200] In scenarios requiring low encoding latency, the encoder (AI encoder) can extract audio data of a first data length from the previous frame of the current frame as redundant audio data. Thus, when encoding the current frame, the AI ​​encoder redundantly encodes the audio data from the previous frame (e.g., frame N-1). This allows the decoder to reconstruct the lost previous frame (e.g., frame N) using the encoded redundant audio data from subsequent frames (e.g., frame N) in the bitstream, even if the previous frame is lost. This redundant audio data from subsequent frames helps mitigate the impact of packet loss in the previous frame.

[0201] Example 2: The AI ​​encoder can also extract redundant audio data from subsequent frames of the current frame (the Nth frame) in Buffer1. For example, it can extract audio data of the first data length from subsequent frames after the Nth frame (e.g., the N+1th frame) as redundant audio data.

[0202] As mentioned above, please refer to Figure 3a. The current frame is the Nth frame. For example, the first data length is 400 sampling points. The AI ​​encoder can extract the first 80 sampling points of the N+1th frame, the N+2th frame and the N+3th frame as shown in Figure 3a(4) as redundant audio data.

[0203] Because subsequent frames are encoded after the current frame, and the encoder can only receive subsequent frames after the current frame to write them into Buffer1, the encoder needs to wait for the subsequent frames (e.g., the Nth frame) to be received and written into the buffer while encoding the current frame (e.g., the N+1th frame). This results in a long encoding latency. In scenarios where encoding latency is not critical, audio data can be extracted from subsequent frames to obtain redundant audio data. In this way, when encoding the current frame, an AI encoder can redundantly encode the audio data in subsequent frames, thus mitigating packet loss in future audio frames after the current frame.

[0204] Example 3 is a combination of Example 1 and Example 2 above.

[0205] The AI ​​encoder can extract redundant audio data from the previous and subsequent frames of the current frame (the Nth frame) in Buffer1. For example, it can extract audio data of length 1 from the first frame to the (N-1)th frame as shown in Figure 2, and extract audio data of length 2 from the subsequent frame after the Nth frame (e.g., the N+1th frame) as shown in Figure 2. The audio data of length 1 and the audio data of length 2 can constitute redundant audio data of the first data length corresponding to the current frame.

[0206] As mentioned above, please refer to Figure 3a. The current frame is the Nth frame. For example, the first data length is three frames. The AI ​​encoder can extract the N-1th frame, the N+1th frame, and the N+2th frame shown in Figure 3a (5) as redundant audio data. That is, it extracts one frame of audio data from the previous frame and two frames of audio data from the subsequent frame. The extracted audio data together constitute the redundant audio data of the first data length (three frames).

[0207] Thus, in this embodiment of the application, the encoding end can flexibly select to extract part of the audio data from the previous frame and the subsequent frame of the current frame according to the encoding latency requirements and application scenario, so as to obtain the redundant audio data corresponding to the current frame. Compared with Example 2, the encoding latency can be further reduced. Compared with Example 1, when the number of previous frames is small, the amount of redundant audio data in a single encoding can be increased by extracting audio data from the subsequent frames as part of the redundant audio data, so as to enhance the anti-interference capability.

[0208] Based on this, the method in this application embodiment can flexibly select redundant encoded data from previous and / or subsequent frames for encoding based on the latency requirements in audio wireless transmission scenarios. For example, if low latency is required, redundant audio data can be selected from previous frames; conversely, if a longer latency is required, redundant audio data can be selected from subsequent frames. By encoding audio data selected from previous and subsequent frames to increase information redundancy, the decoding end can use the redundant audio data encoded by the AI ​​model to restore lost audio data in packet loss scenarios, thereby improving the channel's anti-interference and packet loss capabilities, providing users with a continuous audio experience, and reducing audio stuttering.

[0209] In a scenario where there is no latency in a traditional encoder, for example, a traditional encoder encodes the Nth frame in 10ms to obtain a bitstream, which includes the encoded data of the Nth frame; the traditional decoder at the decoding end decodes the bitstream to obtain the decoded data of the Nth frame, which is the complete Nth frame in 10ms.

[0210] However, in scenarios where traditional encoders have a time delay, such as a delay of 2.5ms (the specific delay is not limited), in the above process, the decoded data of the Nth frame obtained by the traditional decoder at the decoding end only includes the first 7.5ms of data of the Nth frame and the last 2.5ms of data of the N-1th frame (i.e., the previous frame), resulting in a 2.5ms time delay.

[0211] In some embodiments, in scenarios where there is no delay in the conventional encoder, redundant audio data can be a complete one or more audio frames. For example, redundant audio data includes one or more previous frames and / or one or more subsequent frames.

[0212] In some embodiments, where conventional encoders have latency, redundant audio data may not only include one or more complete audio frames, but also audio data with a target number of sample points (which can be extracted from previous and / or subsequent frames). This target number of sample points is greater than or equal to 1, but less than the number of sample points in an audio frame (e.g., 160). This ensures that, in scenarios where conventional encoders have latency, when the encoded data of a certain frame is lost, the decoding end can reconstruct the lost actual audio data, ensuring the accuracy of the reconstructed audio frame.

[0213] In any of the above embodiments, the redundant audio data may include not only audio data extracted from previous frames and / or subsequent frames of the current frame, but also data of any length in the current frame (e.g., a complete current frame, or data of any number of sampling points in the current frame (an incomplete current frame)).

[0214] As mentioned above, please refer to Figure 3a. The current frame is the Nth frame. For example, the first data length is 180 sampling points. The AI ​​encoder can extract all the data of the N-1th frame shown in Figure 3a (6) and the first 20 sampling points of the Nth frame as redundant audio data.

[0215] In short-range wireless transmission scenarios, the encoder can package the encoded data of the current frame and the encoded data of the corresponding redundant audio data of the current frame into a data packet to obtain the bitstream. In this way, if a data packet is lost during the process of transmitting the bitstream from the encoder to the decoder via short-range wireless transmission, not only will the current frame be lost, but also the redundant audio data belonging to the same data packet will be lost. Therefore, the redundant audio data corresponding to the current frame may not include the complete current frame to avoid the decoder being unable to reconstruct the current frame. Based on this, the redundant audio data corresponding to the current frame can be extracted from the previous and / or subsequent frames of the current frame. This redundant audio data can be shown as shaded in any of the figures in Figures 3a(1) to 3a(5). Of course, in scenarios where there is a time delay in the traditional encoder, the redundant audio data corresponding to the current frame can also include audio data with any number of sampling points in the current frame. For example, if the current frame has 160 sampling points, then the audio data with any number of sampling points can be audio data with a number of sampling points greater than 0 and less than 160 in the current frame, such as the redundant audio data shown as shaded in the current frame (the Nth frame) extracted in Figure 3a(6).

[0216] In long-distance wireless transmission scenarios, the encoding end can package the encoded data of the current frame into a data packet and package the encoded data of the redundant audio data corresponding to the current frame into another data packet to obtain the bitstream. That is to say, the loss of the encoded data of the current frame will not directly lead to the loss of the encoded dataset of its redundant audio data. In this scenario, the redundant audio data corresponding to the current frame can include not only the audio data extracted from the previous and / or subsequent frames of the current frame, but also the complete current frame (or the audio data in the current frame with any number of sampling points (not the complete current frame)), such as the redundant audio data shown in the shaded area in any of the figures in Figures 3a(1) to 3a(6). In addition, in long-distance wireless transmission scenarios, the redundant audio data corresponding to the current frame can be only the current frame itself, such as the redundant audio data shown in the shaded area corresponding to the current frame (the Nth frame) extracted in Figure 3a(7). In this way, even if the encoded data of the current frame is lost during transmission, the decoding end can still use the redundant audio data corresponding to the current frame to reconstruct the audio data of some or all sampling points of the current frame. Two bitstreams are obtained and transmitted separately to the decoding end. If packet loss is detected at the decoding end, the lost audio frames can be recovered based on the redundant audio data.

[0217] In the embodiments of this application, data of any length in the current frame can be flexibly selected as part of the redundant audio data corresponding to the current frame in different scenarios of short-range and long-range wireless transmission, as well as in cases where there is latency in traditional encoders and the latency magnitude is different. This satisfies the audio encoding and decoding requirements in various scenarios, resists packet loss, and improves anti-interference capability.

[0218] Optionally, as shown in Figure 2, the method may further include S301.

[0219] S301: The encoding end performs traditional encoding on the current frame to obtain the second encoded data of the current frame.

[0220] The current frame (e.g., the Nth frame) can be encoded using a traditional encoder in the encoding end to obtain the second encoded data of the current frame.

[0221] The compression rate used by traditional encoders to encode the current frame is lower than the compression rate used by the AI ​​model in S202 below to encode the redundant audio data corresponding to the current frame.

[0222] S202: The encoding end uses an AI model to encode the redundant audio data corresponding to the current frame to obtain a bitstream (also known as a mixed bitstream).

[0223] An AI encoder can include an AI model. The AI ​​encoder can use the AI ​​model to encode the redundant audio data corresponding to the current frame to obtain a bitstream.

[0224] The bitstream includes not only the first encoded data of the redundant audio data of the current frame (obtained by the AI ​​model encoding the redundant audio data corresponding to the current frame), but also the second encoded data of the current frame (obtained by the traditional encoder encoding the current frame).

[0225] The AI ​​encoder can receive second encoded data sent by traditional encoders and obtain the aforementioned bitstream based on the first and second encoded data.

[0226] The AI ​​model has a higher compression rate than the current frame (i.e., the compression rate used by the traditional encoder to encode the current frame in S301). This allows the AI ​​model to encode redundant audio data at a higher compression rate, thereby reducing the bandwidth usage of the first encoded data of the redundant audio data.

[0227] When using AI models to encode redundant audio data, the redundant audio data can be converted into multi-dimensional feature vectors. Optionally, the multi-dimensional feature vectors can also be quantized to obtain the first encoded data, thus obtaining the bitstream.

[0228] It should be noted that S202 is executed after S201. However, this application embodiment does not impose specific restrictions on the execution order between S301 and S201, or between S301 and S202.

[0229] The operation of encoding the first and second encoded data into a single bitstream can be performed by a traditional encoder or an AI encoder.

[0230] In long-distance wireless transmission scenarios, Figures 4(1) and 4(2) respectively show the data structure of the bitstream obtained by the method shown in Figure 2. The bitstream for the current frame can include two data packets. One data packet is the redundant audio data packet corresponding to the current frame, and its data structure is shown in Figure 4(1). The header portion (Header1) of this data structure contains the header information of the redundant audio data, and the data portion (Data) is the first encoded data obtained by encoding the redundant audio data corresponding to the current frame using an AI model. The other data packet is the data packet for the current frame, and its data structure is shown in Figure 4(2). The header portion (Header2) of this data structure contains the header information of the current frame, and the data portion is the second encoded data obtained by encoding the current frame using a traditional encoder.

[0231] In the short-range wireless transmission scenario, Figure 4(3) shows the data structure of the bitstream obtained by the method shown in Figure 2.

[0232] As mentioned above, in short-range wireless transmission scenarios, the encoding end (e.g., an AI encoder) can package the first encoded data of the current frame and the second encoded data of the redundant audio data corresponding to the current frame into a data packet to obtain a bitstream. As shown in Figure 4(3), the aforementioned data packet corresponding to the current frame may include a header and a data portion, wherein the data portion may include the first encoded data and the second encoded data. This application does not restrict the position of the first encoded data and the second encoded data in the aforementioned Data.

[0233] Optionally, in order to enable the decoding end to find the second encoded data that needs to be conventionally decoded and the first encoded data that needs to be AI decoded from the data packet shown in FIG4(3), the Header shown in FIG4(3) may include not only the header information of Header1 and the header information of Header2 shown in FIG4(1) and FIG4(2), but may also include position information indicating the position of the first encoded data and / or the second encoded data in the above data portion.

[0234] Of course, this application does not limit the location information in the bitstream (e.g., a data packet) to be in the header information. In other embodiments, the location information may also be carried in the data portion.

[0235] In short-range wireless transmission scenarios, the encoder may not perform AI encoding of the corresponding redundant audio data for each audio frame. In the scenario where the current frame is encoded only by a traditional encoder and the redundant audio data corresponding to the current frame is not encoded by an AI encoder, the data structure of the bitstream encoded by the encoder can be shown in Figure 4(2).

[0236] Optionally, in this scenario, the data structure shown in Figure 4(2) can carry information indicating the encoding type (e.g., a flag field). When the value of flag is 0, it indicates that the data structure contains not only the second encoded data of traditional encoding but also the first encoded data of AI encoding, that is, flag indicates that the encoding type is traditional encoding and AI encoding; when the value of flag is 1, it indicates that the data structure includes the second encoded data of traditional encoding but does not include the first encoded data of AI encoding, that is, flag indicates that the encoding type is traditional encoding. In this way, the decoding end does not need to perform AI decoding on the bitstream, but only traditional decoding.

[0237] In some embodiments, when the encoded audio frame is an audio frame in music audio data, since music can have two or more channels, the current frame can be sampled at the encoding end according to the number of channels. For example, if the number of channels is 2, namely the left channel and the right channel, the encoding end can sample 10ms of PCM data for the left channel and 10ms of PCM data for the right channel of the current frame. Then, when the encoding end executes S301, it can perform parallel encoding of the PCM data of the two channels of the current frame to obtain the second encoded data. However, the redundant audio data corresponding to the PCM data of the two channels of the current frame is the same. Therefore, when the encoding end executes S202, it can encode the redundant audio data corresponding to the PCM data of the two channels of the current frame.

[0238] The following illustrates a data structure for a bitstream according to an embodiment of this application:

[0239] 1) Data Header (Optional): Used to record metadata of the bitstream.

[0240] 2) First encoded data (required).

[0241] 3) Second encoded data (optional).

[0242] 4) Number of audio channels for audio sampling (optional).

[0243] 5) The above flag (optional).

[0244] 6) The frame length (optional) of the audio frame encoded by traditional encoding (e.g., the current frame), for example, 10ms.

[0245] 7) Location information (optional).

[0246] In other embodiments, the bitstream may include more or fewer fields.

[0247] In this embodiment, when the encoding end (e.g., an AI encoder) encodes redundant audio data corresponding to a single audio frame, it can encode one or more frames at a time, and / or redundant audio data with any number of sampling points. Compared with related technologies that can only encode a single audio frame as redundant audio data, the encoding method of this embodiment has higher flexibility in encoding redundant audio data. Furthermore, by encoding not only the second encoded data of the current frame but also the first encoded data of the redundant audio data corresponding to the current frame in the bitstream, the bitstream can have the ability to actively resist packet loss.

[0248] Furthermore, in this embodiment, the redundant audio data is encoded using the aforementioned AI model. Because the compression rate of the AI ​​model is higher than the compression rate used in traditional encoding of the current frame to obtain the second encoded data (an example of the compression rate corresponding to the current frame), the amount of data in the first encoded data obtained by encoding the redundant audio data using the AI ​​model is extremely small. Compared to related technologies that use the same compression rate as the current frame to encode redundant audio frames, the amount of data in the encoded data of redundant audio data (i.e., the first encoded data) can be significantly reduced, thereby reducing the overall bitrate of the bitstream.

[0249] For example, when the audio encoding bitrate is 64Kbps, one implementation method in related technologies is to reduce the encoding bitrate of the current frame by half to 32Kbps, and also to encode the redundant audio frames of the current frame at 32Kbps, thereby reducing the sound quality; or another implementation method in related technologies is to keep the encoding bitrate of the current frame unchanged at 64Kbps, and also to encode the redundant audio frames of the current frame at 64Kbps, so that the total bitrate of the bitstream increases from 64Kbps to 128Kbps, thereby increasing the bitrate and increasing the bandwidth usage.

[0250] However, using the method of this application embodiment, for example, if the audio encoding bitrate is 64Kbps, the encoding bitrate of redundant audio data does not need to occupy half of the bitrate. Because the compression ratio of the AI ​​model is extremely high compared to that of the traditional encoder, the amount of data in the first encoded data of the redundant audio data is very small. Therefore, the AI ​​model can use a very small bitrate, such as 4Kbps (not limited to 4Kbps), to encode the longer redundant audio data, while the current frame uses 60Kbps for traditional encoding. This allows for the encoding of more redundant audio data without significantly reducing the sound quality, thereby improving the ability to resist packet loss. For example, in scenarios where the total bitrate of audio data encoding is limited (transmission bandwidth is limited), it can improve the anti-interference ability while ensuring that the audio quality is not significantly reduced.

[0251] Alternatively, using the method of this application embodiment, if the encoding bitrate of the current frame remains unchanged at 64Kbps, an additional 4Kbps can be used to encode the redundant audio data corresponding to the current frame. In this way, although redundant audio data is encoded, the total bitrate only increases from 64Kbps to 68Kbps, and does not double. Thus, while having the ability to resist packet loss, the total bitrate and the bandwidth occupation of the bitstream will not be significantly increased. In this way, although the encoding end needs to transmit the first encoded data of the redundant audio data in the same amount of time, it will not increase the transmission burden or cause a large transmission delay, thus better adapting to low-latency scenarios.

[0252] The following sections describe the encoding / decoding process of the traditional encoder / decoder mentioned in the above embodiments, as well as the encoding process of the AI ​​encoder.

[0253] The encoding and decoding process of a traditional codec:

[0254] Taking a short-range Bluetooth wireless transmission voice call scenario as an example, the audio sampling rate is 16KHz. The traditional codec uses the L2HC voice codec method to encode each audio frame. The frame length of each audio frame is 10ms. Due to factors such as window function and number of filter points, the traditional codec has an algorithm delay of 2.5ms.

[0255] The delay of this algorithm is as follows: the traditional encoder receives 10ms of PCM data for the current frame to be encoded, but the traditional decoder only receives the first 7.5ms of PCM data for the current frame, and the remaining 2.5ms of PCM data is the last 2.5ms of PCM data from the previous frame.

[0256] The encoding and decoding process of a traditional codec is as follows:

[0257] Please refer to Figure 5a, which is a schematic diagram of the encoding process of L2HC encoding (such as S301 shown in Figure 2 above) provided in the embodiments of this application.

[0258] As shown in Figure 5a, a conventional encoder can receive 10ms of PCM data (a type of digital audio signal) of the current frame, and then add a 20ms analysis window to the 10ms PCM data to process the 10ms PCM data into 20ms PCM data.

[0259] Subsequently, a conventional encoder can perform an improved discrete cosine transform (MDCT) on the 20ms PCM data to obtain the corresponding frequency domain signal. In this example, the PCM data has a sampling rate of 16kHz, allowing the frequency domain signal to include the spectral values ​​of 160 sampling points.

[0260] Next, a conventional encoder can quantize and encode the aforementioned frequency domain signal to obtain the second encoded data of the current frame.

[0261] The obtained frequency domain signal can be quantized based on a psychoacoustic model, or it can be quantized using other methods.

[0262] Traditional encoders can perform entropy coding when encoding quantized data.

[0263] Finally, the encoding end (traditional encoder or AI encoder) can package the second encoded data and the first encoded data obtained by the AI ​​encoder into a single bitstream, which can then be transmitted to the decoding end via short-range Bluetooth.

[0264] Next, please refer to Figure 5b, which is a schematic diagram of the decoding process of an L2HC decoder provided in an embodiment of this application.

[0265] As shown in Figure 5b, after the decoding end receives the bit stream (binary data stream), the traditional decoder can obtain the second encoded data in the bit stream.

[0266] As shown in Figure 5b, a traditional decoder can extract the second coded data from the bitstream, decode the second coded data (e.g., entropy decoding, without limitation), and dequantize the decoded second coded data to obtain the frequency domain signal.

[0267] Next, the traditional encoder can perform an improved inverse discrete cosine transform (IMDCT) on the frequency domain signal to obtain a 20ms time domain signal, which is time domain aliasing data (described below).

[0268] Finally, traditional decoders can add a synthesis window to the 20ms temporal aliasing data for overlapping processing, thereby processing the 20ms temporal aliasing data into 10ms of decoded data (digital audio signal, such as PCM data) of the current frame.

[0269] The following section, in conjunction with Figure 6, details the process of adding an analysis window and MDCT as described in Figure 5a, and the overlapping process of IMDCT and the addition of a synthesis window as shown in Figure 5b.

[0270] Referring to Figure 6(1), a conventional encoder can add a 20ms analysis window to the 10ms PCM data of the current frame. The analysis window can be used to process the 20ms PCM data consisting of the 6.25ms of the previous frame, the 3.75ms of the next frame, and the 10ms of the current frame using a window function to obtain the 20ms time-domain data shown in Figure 6(1).

[0271] Since L2HC is a coding based on frequency domain MDCT transform, when the window function is a low-delay window and a symmetrical window, as shown in Figure 6(1), the leftmost and rightmost 3.75ms of the 20ms time domain obtained after processing by the window function are 0. That is to say, although the 6.25ms of PCM data in the previous frame is not 0, the window function of the analysis window can process the first 3.72ns of the 6.25ms to 0. Similarly, the 3.75ms of the next frame will also be processed to 0 to obtain the 20ms time domain data after adding the analysis window as shown in Figure 6(1).

[0272] Then, as shown in Figure 6(1), the conventional encoder can perform MDCT processing on the 20ms time-domain data after adding the analysis window to obtain the frequency-domain signal. The subsequent operations can be referred to the introduction in Figure 5a, and will not be repeated here.

[0273] Referring to Figure 5b above and continuing to Figure 6(2), the traditional decoder can decode the bitstream according to the process in Figure 5b to obtain the frequency domain signal, and then perform IMDCT processing on the frequency domain signal to obtain a 20ms time domain signal. However, due to the processing principles of MDCT and IMDCT, the 20ms time domain signal obtained here is time domain aliasing data.

[0274] Then, as shown in Figure 6(2), a synthesis window can be added to the 20ms temporal aliasing data obtained after IMDCT. This synthesis window is the inverse window function corresponding to the window function on the encoding side. By adding the inverse window function to the 20ms temporal aliasing data and overlapping it with the previous frame, the decoding data of the current frame of 10ms as shown in Figure 6(2) can be obtained.

[0275] Due to the mechanism of the window function, the data in the first 3.75ms of the 20ms temporal aliasing data after IMDCT, as shown in Figure 6(2), is 0. Therefore, the time-domain aliasing data with 0 data will overlap with any data and result in 0. So, in the following description of the process of adding a synthesis window for overlap processing, the processing of the first 3.75ms will not be described. Instead, the overlap processing of the 16.25ms temporal aliasing data obtained after IMDCT can be described.

[0276] In the traditional decoder, when processing the 20ms temporal aliasing data (here only the 16.25ms temporal aliasing data is described) using the inverse window function, as shown in Figure 6(2), the last 6.25ms of the temporal aliasing data in the 20ms temporal aliasing data after the IMDCT and synthesis window obtained by the traditional decoder when decoding the previous frame can be overlapped with the first 6.25ms of the 16.25ms temporal aliasing data after the IMDCT of the current frame (represented by a hollow "+"), thereby obtaining the decoded data of the current frame (specifically 10ms) and the last 6.25ms of the 16.25ms temporal aliasing data after the IMDCT of the current frame.

[0277] As shown in Figure 6(2), the last 6.25ms of the temporal aliasing data after IMDCT of the current frame can be used as temporal aliasing data for overlapping when the traditional decoder decodes the next frame.

[0278] As shown in Figure 6(2), the 10ms decoding data of the current frame may include 6.25ms of data obtained by the above overlapping process, and 3.75ms of data after IMDCT of the current frame as shown by the thick hollow arrow.

[0279] Referring to the decoded data of the current frame shown in Figure 6(2) and the 10ms data of the current frame shown in Figure 6(1), it can be seen that there is a delay of 2.5ms as shown in Figure 6(2).

[0280] Therefore, although the traditional decoder decodes the second encoded data of the current frame to obtain the decoded data of the current frame, the decoded data of the current frame is not the complete current frame. Instead, it specifically includes the 7.5ms of PCM data at the beginning of the current frame as shown in Figure 6(2), and the 2.5ms of PCM data at the end of the previous frame as shown in Figure 6(2), which makes the traditional codec have a delay of 2.5ms.

[0281] Of course, in other embodiments, the conventional codec may also have no latency, that is, the latency is 0, or the latency is not 2.5ms, but more or less latency, and this application does not limit this.

[0282] The AI ​​encoder of this application and its implemented encoding process are described below:

[0283] AI encoders may include AI encoding models (an example of an AI model mentioned in the encoding methods above).

[0284] Please refer to Figures 7a and 8a. Figure 7a is a structural schematic diagram of an AI encoding model provided in an embodiment of this application, and Figure 8a is a schematic diagram of the process of encoding redundant audio data by an AI encoding model provided in an embodiment of this application.

[0285] As shown in Figure 7a, the AI ​​encoding model may include, but is not limited to, encoding module 601 and quantization module 602.

[0286] As shown in Figure 7a, the encoding module 601 can be a trained convolutional neural network (CNN). For example, the encoding module 601 may include m convolutional layers, where m is a positive integer greater than 1. This application does not limit the number of convolutional layers and the network structure can be flexibly set according to the application scenario.

[0287] Furthermore, the network structure of the encoding module 601 in this application embodiment is not limited to the above-described convolutional neural network, but can also be other network structures, such as feedforward neural network (FNN), recurrent neural network (RNN), long short-term memory network (LSTM), gated recurrent unit (GRU), self-attention mechanism and Transformer, generative adversarial network (GAN), variational autoencoder (VAE), deep belief network (DBN), etc. This application does not limit the network structure of the encoding module 601.

[0288] Furthermore, the aforementioned quantization module 602 can be implemented using quantization techniques such as vector quantization (VQ), scalar quantization (SQ), or residual vector quantization (RVQ).

[0289] Depending on the quantization technique used, the quantization module 602 may optionally further include one or more codebooks. When the quantization module 602 is implemented using VQ or RVQ, it may include one or more codebooks.

[0290] Each codebook may contain one or more feature vectors. Optionally, each codebook may also include an index corresponding to the feature vector. The feature vector and index within a single codebook can have a one-to-one mapping relationship. When the quantization module 602 includes multiple codebooks, the same feature vector can uniquely correspond to an index in each of the multiple codebooks, allowing the same feature vector to be quantized into multiple indices through multiple codebooks. At the decoding end, these multiple indices can be dequantized and combined into a single feature vector.

[0291] The codebook used by the quantization module 602 and the network parameters of the encoding module 601 in the AI ​​encoding model can be obtained by pre-training the network of the encoding module 601 through the method of the embodiments of this application, thereby achieving network convergence. Of course, during the process of using the AI ​​encoding model to perform inference on the input redundant audio data, the network parameters and codebook in the AI ​​encoding model can be continuously optimized. This application does not limit the training time of the AI ​​encoding model.

[0292] When training the AI ​​encoding model, the dataset used can be various types of audio data such as music audio data and speech audio data. The audio type in the dataset can be flexibly set according to the application scenario of redundant audio encoding, and this application does not impose any restrictions on this.

[0293] Thus, in this embodiment of the application, by using a certain type of audio data (such as speech audio data or music audio data) to train the AI ​​encoding model, the generalization of the AI ​​encoding model in this embodiment of the application can be improved, so as to be applicable to redundant encoding of any audio data of this type, thereby improving the ability to resist packet loss.

[0294] The method in this application embodiment can obtain not only the codebook and network parameters of the AI ​​coding model after training, but also the bitrate information supported by the AI ​​coding model.

[0295] The bitrate information can be at least one of the following: the highest bitrate, the lowest bitrate, or the bitrate level supported by the AI ​​encoding model for AI encoding of audio data.

[0296] In some embodiments, when using an AI encoding model for inference (e.g., without model optimization during inference), the codebook and network parameters remain unchanged. However, the bitrate can be flexibly set and adjusted according to different application scenarios (e.g., the length of redundant audio data, encoding efficiency requirements, packet loss rate, etc.). For example, the encoding bitrate for AI encoding of redundant audio data can be set within the range of the highest and lowest bitrates supported by the AI ​​encoding model (an example of the first bitrate below). Alternatively, within the bitrate ranges supported by the AI ​​encoding model, the encoding bitrate for AI encoding of redundant audio data can be selected according to the application scenario. For example, bitrate ranges could be 2Kbps, 4Kbps, 5Kbps, etc.

[0297] In some embodiments, when the AI ​​encoding model performs AI encoding on redundant audio data corresponding to each audio frame in the audio sequence (e.g., each time S202 shown in Figure 2 is executed), it can use the same bitrate to improve encoding efficiency. Alternatively, the AI ​​encoding model can also use different bitrates when performing AI encoding on redundant audio data corresponding to each audio frame in the audio sequence, so that the bandwidth occupied by the encoded data of the redundant audio data corresponding to each audio frame in the audio sequence can be flexibly varied to meet the needs of various scenarios.

[0298] In some embodiments, when the AI ​​encoding model performs AI encoding on the redundant audio data corresponding to the current frame (e.g., S202 as shown in Figure 2 above), the redundant audio data can be encoded using the same bit rate.

[0299] In some embodiments, when the AI ​​encoding model performs AI encoding on the redundant audio data corresponding to the current frame (e.g., S202 as shown in Figure 2 above), it can also use different bitrates to encode different numbers of points in the redundant audio data corresponding to the same current frame or different frames of audio data.

[0300] For example, please refer to Figure 3a. The current frame is the Nth frame. The redundant audio data is the shaded data shown in Figure 3a(2). Taking 10ms per frame as an example, the redundant audio data specifically includes the N-1th frame and the 40 sampling points at the end of the N-2th frame. Then the AI ​​encoding model can use different bit rates to encode the 40 sampling points at the end of the N-2th frame and the N-1th frame respectively to obtain the first encoded data.

[0301] Furthermore, it should be understood that when redundant audio data includes more than one frame of audio data, such as the case shown in Figure 3a(5) above, the AI ​​coding model can encode each frame of audio data separately multiple times (here, 3 times), such as the case of encoding at different bit rates as exemplified above. Alternatively, the redundant audio data shown in Figure 3a(5) can also be input together into the AI ​​coding model, and the AI ​​coding model can encode it once to obtain the first encoded data.

[0302] In some embodiments, based on the differences in audio sampling rates described above, the method of this application embodiment can train multiple AI encoding models corresponding to different audio sampling rates in scenarios with different audio sampling rates. For example, an AI encoding model can correspond to a unique audio sampling rate, enabling the AI ​​encoding model to perform AI encoding inference on redundant audio data sampled based on that audio sampling rate to obtain a bitstream.

[0303] The encoding end of this application can select an AI encoding model that matches the audio sampling rate to encode the redundant audio data corresponding to the current frame (PCM data sampled according to the audio sampling rate). Once the AI ​​encoding model is determined, one or more codebooks used by that AI encoding model are also determined. During subsequent AI model inference, the set of codebooks used by the AI ​​encoding model (corresponding to one or more codebooks for different bitrates) remains unchanged.

[0304] In some embodiments, the different bitrates used by the AI ​​encoding model can affect the quantization operation of the AI ​​encoding model.

[0305] An AI encoding model has a set of codebooks, which may include multiple codebooks corresponding to different bitrates supported by the AI ​​encoding model. In this way, the AI ​​encoding model can select the corresponding codebook for quantization based on the bitrate used.

[0306] In some embodiments, an AI model may have one or more codebooks, which are shared by the AI ​​encoding model and the corresponding AI decoding model. Different codebooks may support different bitrates, and the same codebook may also support multiple bitrates.

[0307] When performing quantization, AI encoding models can also refer to the quantization step size instead of using the codebook. This quantization step size is also obtained after the AI ​​encoding model has converged during training. Therefore, the AI ​​encoding model can use the bitrate of the AI ​​encoding (e.g., the first bitrate described below) to select the corresponding quantization step size for quantization operations.

[0308] In other words, the bitrate used by the AI ​​encoding model can affect the quantization parameters of the AI ​​encoding model (such as quantization step size or codebook information).

[0309] AI encoding models and their corresponding AI decoding models can share the same quantization parameters. Therefore, after being trained together, they share a set of quantization step size or codebook information.

[0310] In some embodiments, in conjunction with the data structure of the bitstream shown in the above embodiments, the following further illustrates a data structure of the bitstream according to an embodiment of this application, which may further include:

[0311] 8) Audio sampling rate (optional): AI encoding uses the same sampling rate as traditional encoding.

[0312] 9) Bitrate (e.g., the first bitrate described below) (optional): The encoding bitrate used to record redundant audio data. It can be the encoding bitrate of each frame in the redundant audio data or the uniform encoding bitrate of the redundant audio data.

[0313] Based on this, as shown in the bitstream data structure above, the bitstream can include the audio sampling rate. Thus, the decoder can select an AI decoding model matching the sampling rate of the audio decoded from the bitstream to decode the bitstream. Furthermore, the decoder (e.g., the AI ​​decoding model corresponding to the aforementioned AI encoding model) can also select quantization parameters such as a codebook or quantization step size based on the bitrate information (first bitrate) carried in the bitstream's data structure to decode the bitstream (dequantization operation) and obtain reconstructed data of redundant audio data.

[0314] The following section, using the structure of the AI ​​encoding model shown in Figure 7a as an example, will introduce the encoding process of the AI ​​encoding model shown in Figure 8a.

[0315] As shown in Figure 8a, the AI ​​encoding model can encode the redundant audio data corresponding to the current frame to obtain a multi-dimensional feature vector.

[0316] As shown in Figure 7a, the encoding module 601 of the AI ​​encoding model (specifically a convolutional neural network including m convolutional layers) can receive redundant audio data (PCM data) and obtain multidimensional feature vectors through convolution operations of the m convolutional layers. Each of the m convolutional layers converts the input data into a corresponding multidimensional feature vector, where m is a positive integer greater than 1. For example, as shown in Figure 7a, after the redundant audio data is input into the AI ​​encoding model, it first passes through the convolution operation of convolutional layer 1 to obtain feature vector 1; this feature vector 1 is then input into convolutional layer 2 for convolution to obtain feature vector 2, and so on, until finally, after passing through the convolution operation of the m convolutional layers, a multidimensional feature vector is obtained. This multidimensional feature vector can be one or more. It can be a high-dimensional feature vector corresponding to the redundant audio data. The network parameters (e.g., weight parameters) in the multiple convolutional layers are obtained by convergence after training the AI ​​encoding model.

[0317] Returning to Figure 8a, the AI ​​encoding model can quantize multidimensional feature vectors (one or more) to obtain the first encoded data.

[0318] As shown in Figure 7a, the quantization module 602 of the AI ​​encoding model can perform vector quantization processing on the input multidimensional feature vector to obtain the first encoded data. The quantization module 602 can adopt the RVQ quantization technique, so that the quantization module 602 can include the codebook shown in Figure 8a, so as to quantize the multidimensional feature vector based on the codebook.

[0319] As shown in Figure 8a, the codebook may include multiple feature vectors, each with a unique index. The quantization module 602 can search the codebook for the feature vector most similar to the input multidimensional feature vector, thereby determining an index corresponding to that feature vector in the codebook, and thus quantizing the multidimensional feature vector into a single index. Of course, when multiple codebooks are used for quantization at the encoding end, the quantization module 602 can quantize the multidimensional feature vector into multiple indices using multiple codebooks, with the number of indices being the same as the number of codebooks.

[0320] The feature vector searched by the quantization module 602 can be a feature vector in the codebook that is most similar to the multidimensional feature vector (e.g., the Euclidean distance between the vectors is the smallest; the specific strategy for judging similarity is not restricted) (also called a similar feature vector). This yields at least one index (or multiple indices if there are multiple codebooks) corresponding to the similar feature vector in the codebook, such as index 1 shown in Figure 8a. In this way, the multidimensional feature vector can be quantized into at least one index, thereby obtaining the first encoded data.

[0321] For example, the encoder can encode at least one index (e.g., entropy coding) to obtain first encoded data, thereby obtaining a bitstream. Alternatively, the encoder can use at least one index as first encoded data to obtain a bitstream.

[0322] Referring to the data structure of the bitstream described above, the first encoded data in this data structure may include at least one index.

[0323] Referring to any of the embodiments in Figures 2 to 8a, please refer to Figure 9, which is a flowchart illustrating another encoding method provided by an embodiment of this application. This encoding method can be applied to the encoding end (e.g., an AI encoder) shown in Figure 1. As shown in Figure 9, this encoding method may include S201 and S202 as shown in Figure 2, and optionally S301. Descriptions of S201, S202, and S301 can be found in the relevant descriptions in the embodiments corresponding to Figure 2, and will not be repeated here.

[0324] As shown in Figure 9, compared to the embodiment corresponding to Figure 2 above, the method may further include the following steps:

[0325] S401: The encoding end obtains channel state information, which indicates the packet loss rate of the transmission channel of the code stream.

[0326] The transmission channel can be a transmission channel between the encoding end (or the sending end that sends the code stream) and the decoding end (or the receiving end that receives the code stream).

[0327] Depending on the transmission protocol, when the transmission protocol is a short-range wireless transmission protocol, the transmission channel can be a short-range wireless transmission channel, such as a Bluetooth transmission channel, an infrared transmission channel, or a Wi-Fi transmission channel.

[0328] When the transmission protocol is a long-distance wireless transmission protocol, the transmission channel can be a long-distance wireless transmission channel, such as a satellite communication channel, a terrestrial microwave channel, a fourth-generation (4G) mobile communication channel, a fifth-generation (5G) mobile communication channel, etc.

[0329] In some embodiments, when the encoding end obtains channel state information, it can obtain the state information of the buffer unit of the transport protocol stack. The buffer unit is used to buffer the bit stream, and the state information of the buffer unit can indicate the packet loss rate of the transport channel.

[0330] The bitstream can be transmitted through the transmission protocol stack. For example, if the transmission protocol is Bluetooth, then the transmission protocol stack can be a Bluetooth protocol stack. This application does not make any specific limitations on this.

[0331] The buffer unit can be a Buffer. Unlike Buffer1, which is used to buffer the audio sequence to be encoded as described in the above embodiments, the buffer used to buffer the bitstream to be sent can be named Buffer2.

[0332] After obtaining the bitstream (including the first encoded data and the second encoded data, also known as the mixed bitstream) corresponding to each audio frame, the encoding end can cache it in Buffer2 of the transport protocol stack.

[0333] To improve the transmission efficiency of the bitstream, the encoding end writes the bitstream corresponding to each audio frame it encodes into a buffer. When transmitting the bitstream, the encoding end can read the bitstream from Buffer2 according to the encoding order (e.g., the audio playback order) and send the read bitstream to the decoding end through the aforementioned transmission channel. This reduces the latency caused by waiting for the bitstream to be generated to a certain extent, thereby improving the overall bitstream transmission efficiency.

[0334] The status information of the buffer unit can be information such as the congestion level of Buffer2, to indicate the packet loss rate of the transmission channel.

[0335] For example, if the size of Buffer2 is fixed, then when Buffer2 is congested, such as when Buffer2 is close to full of data, it indicates that the buffered bitstream cannot be sent out in time, the transmission channel has strong interference with the transmission of the bitstream (e.g., low bandwidth, obvious jitter, etc.), and the packet loss rate of the bitstream is high. When Buffer2 is idle (e.g., the amount of buffered data is less than half the size of Buffer2), it indicates that the buffered bitstream can be sent out in time, the transmission channel has low interference with the transmission of the bitstream, and the packet loss rate of the bitstream is low.

[0336] The encoding end can obtain the status information of the buffer unit through hardware or software. For example, it can obtain the status information of Buffer2 through the physical interface or through related instructions.

[0337] In some embodiments, when the encoding end obtains channel state information, it can obtain signal strength information of the transmission channel, which can indicate the packet loss rate of the transmission channel.

[0338] Signal strength information of a transmission channel can be used to measure the quality and reliability of the transmitted signal. Higher signal strength indicates higher data transmission reliability and a lower packet loss rate. Conversely, weaker signal strength makes data more susceptible to interference and noise during transmission, resulting in a higher packet loss rate.

[0339] The encoding end can use hardware to detect the signal strength information of the transmission channel.

[0340] For example, taking Bluetooth as the transmission protocol, the Bluetooth chip on the encoding end can monitor the signal strength of the Bluetooth channel. Then, the AI ​​encoder on the encoding end can obtain the signal strength monitored by the Bluetooth chip through software or hardware.

[0341] In some embodiments, when the encoding end obtains channel state information, it can obtain the packet loss rate of the transmission channel.

[0342] The encoding end can use the aforementioned Bluetooth chip to obtain the packet loss rate information of the Bluetooth channel.

[0343] This application does not impose any restrictions on the specific implementation method for obtaining the packet loss rate of the transmission channel.

[0344] It should be understood that the methods for obtaining channel state information are not limited to those listed in the above embodiments, and may include other methods. Furthermore, the methods for obtaining channel state information mentioned in the above embodiments can be freely combined to comprehensively determine the channel state information. In this way, various information obtained through different methods can be cross-checked, improving the accuracy of the determined channel state information.

[0345] In addition, in some embodiments, the encoding end can acquire channel state information in real time, or periodically, or under certain conditions. This application does not specifically limit this.

[0346] S402: The encoding end obtains the first data length of redundant audio data based on the channel state information.

[0347] There is a positive correlation between the packet loss rate indicated by the channel state information and the length of redundant audio data. For example, when the packet loss rate indicated by the channel state information is high, redundant audio data with a longer data length can be obtained; when the packet loss rate indicated by the channel state information is low, redundant audio data with a shorter data length can be obtained.

[0348] For example, referring to Figure 3a, the audio sequence encoded by the encoder is frame 1... frame N-3, frame N-2, frame N, frame N+1, frame N+2, frame N+3..., where frame N is the current frame. As mentioned above, the data length of each frame is 160 sampling points (i.e., 10ms).

[0349] When the channel state information indicates that the packet loss rate of the transmission channel is high (e.g., greater than the preset packet loss rate), the first data length can be two or more audio frames, or the number of sampling points can exceed 320. For example, the redundant audio data consisting of two frames of redundant audio data as shown in Figure 3a(3), and the redundant audio data consisting of two frames (160 sampling points) and the first 80 sampling points of the N+3 frame as shown in Figure 3a(4).

[0350] When the channel state information indicates that the packet loss rate of the transmission channel is low, the first data length can be one frame, or the number of points can be less than 320. For example, as shown in Figure 3a(1), there is a frame of redundant audio data (the N-1th frame), and as shown in Figure 3a(6), there is redundant audio data consisting of the N-1th frame and the first 20 sampling points of the Nth frame.

[0351] The above examples are only used to illustrate the positive correlation between packet loss rate and the length of redundant audio data. This application does not limit whether the positive correlation is linear or non-linear.

[0352] The encoding end can obtain the first data length in real time based on the real-time channel state information, or it can obtain the first data length periodically based on the periodic channel state information. This application embodiment does not specifically limit this.

[0353] In other words, the encoding method of this application embodiment can flexibly and dynamically determine the first data length of the redundant audio data to be encoded based on channel state information, thereby dynamically adjusting the data length of the encoded redundant audio data. Thus, when the transmission channel is severely interfered with, more redundant audio data can be encoded, allowing the bitstream to include more redundant information, thereby improving the anti-interference capability of the audio data during transmission. Even when the transmission channel is severely interfered with, the decoding end can still receive sufficient redundant information to recover lost data.

[0354] The encoding method of this application embodiment can flexibly determine the length of the redundant audio data to be encoded according to the channel state information, so that the length of the encoded redundant audio data is adapted to the channel state. When the channel is severely interfered with, more redundant audio data can be encoded, thereby increasing the possibility that the decoding end can receive the redundant audio data, and thus increasing the possibility that the decoding end can recover the audio data normally, which can improve the anti-interference ability.

[0355] Furthermore, in related technologies, the method of obtaining channel state information is to wait for the decoding end to send back the information. This operation has a certain time delay, which makes it impossible to obtain the channel state information in a timely manner, and consequently, the first data length cannot be obtained in a timely manner. Consequently, it is impossible to obtain the redundant audio data of the first data length in a timely manner, resulting in a decrease in overall encoding efficiency. However, in the embodiment of this application, the method of obtaining channel state information is to actively detect it at the encoding end. Therefore, the embodiment of this application can obtain the channel state information in a timely manner, and the overall encoding efficiency will not be affected by the time delay caused by this operation.

[0356] In other embodiments, the first data length of the redundant audio data to be encoded can also be a preset value (e.g., determined based on prior information, such as 3 frames), rather than dynamically determined based on channel state information or other information. This can reduce the computational overhead of the encoder in determining the first data length, thereby improving encoding efficiency.

[0357] Referring to Figure 9, after the encoding end executes S402 to obtain the first data length, it can obtain the redundant audio data of the corresponding first data length of the current frame when executing S201, so that the data length of the encoded redundant audio data can match the state of the transmission channel and improve the anti-packet loss capability.

[0358] Optionally, as shown in Figure 9, the encoding method may also include S403 after S402.

[0359] S403: The encoding end can encode the first data length of redundant audio data into the bitstream.

[0360] It should be understood that this application does not restrict the execution order between S403 and S201, nor does it restrict the execution order between S403 and S202. S403, S201, and S202 can all be executed after S402.

[0361] In some embodiments, the first data length may include the number of complete audio frames and the number of sample points of incomplete audio frames.

[0362] A complete audio frame refers to an entire audio frame. A complete audio frame can be any one or more previous frames, optionally including the current frame; or it can be any one or more subsequent frames, optionally including the current frame; or it can be at least one previous frame and at least one subsequent frame, optionally including the current frame.

[0363] An incomplete audio frame means that the audio data is a portion of the audio data extracted from a complete audio frame, rather than the complete audio data of that audio frame.

[0364] The incomplete audio frame can also be selected from previous and / or subsequent frames, and optionally, further selected from the current frame.

[0365] For example, if an audio frame has 160 sampling points (i.e., the number of sampling points is 160), then the incomplete audio frame can be any number of sampling points (excluding the boundary values ​​0 and 160) within that audio frame. Furthermore, this arbitrary number of sampling points can be continuous or discontinuous audio data.

[0366] For example, if an audio frame has 160 sampling points, and three audio data points are selected from the audio frame to form part of the redundant audio data, then the three audio data points can be the first three data points at the beginning of the audio frame or the last three data points at the end of the audio frame, or three consecutive data points that are neither at the beginning nor the end, or the audio data points 1, 3, and 5 of the audio frame.

[0367] The following examples illustrate different cases of the first data length:

[0368] Example 4: Extracting redundant audio data from previous frames

[0369] The first data length is: 1 frame (i.e., the number of complete audio frames is 1) and 40 sample points (i.e., the number of sample points of incomplete audio frames is 40).

[0370] As shown in Figure 3a(2), the current frame is the Nth frame. The encoding end can extract the audio data of the last 40 sampling points of the previous frame N-2 from the buffer in Buffer1 according to the first data length, and extract the N-1th frame to obtain 1 frame and 40 sampling points of redundant audio data.

[0371] Example 5: Extract redundant audio data from previous and current frames. The extracted data for the current frame is a non-complete frame.

[0372] The first data length is: 1 frame (i.e., the number of complete audio frames is 1) and 20 sample points (i.e., the number of sample points of incomplete audio frames is 20).

[0373] As shown in Figure 3a(6), the current frame is the Nth frame. The encoding end can extract the N-1th frame from the previous frame cached in Buffer1 according to the first data length, and extract the audio data of 20 sampling points in the header of the Nth frame to obtain redundant audio data of 1 frame and 20 sampling points.

[0374] Example 6: Extract redundant audio data from previous frames and the current frame, and obtain the data of the current frame as a complete frame.

[0375] The first data length is: 2 frames (i.e., the number of complete audio frames is 2) and 40 sampling points (i.e., the number of sampling points of incomplete audio frames is 40).

[0376] As shown in Figure 3a(8), the current frame is the Nth frame. The encoding end can extract the audio data of the last 40 sampling points of the N-2th frame from the previous frame cached in Buffer1 according to the first data length, as well as extract the N-1th frame and the Nth frame, to obtain 2 frames and 40 sampling points of redundant audio data.

[0377] Example 7: Extracting redundant audio data from subsequent frames

[0378] The first data length is: 2 frames (i.e., the number of complete audio frames is 2) and 0 sample points (i.e., the number of sample points for incomplete audio frames is 0).

[0379] As shown in Figure 3a(3), the current frame is the Nth frame. The encoding end can extract the N+1th frame and the N+3th frame from the previous frame cached in Buffer1 according to the first data length to obtain 2 frames of redundant audio data.

[0380] Example 8: Extract redundant audio data from subsequent frames and the current frame, and the extracted data of the current frame is the complete frame.

[0381] The first data length is: 3 frames (i.e., the number of complete audio frames is 3) and 80 sampling points (i.e., the number of sampling points of incomplete audio frames is 80).

[0382] As shown in Figure 3a(9), the current frame is the Nth frame. The encoding end can extract the Nth frame, the N+1th frame, the N+2th frame, and the audio data of the 80 sampling points of the header of the N+3th frame from the previous frames cached in Buffer1 according to the first data length, so as to obtain redundant audio data of 3 frames and 80 sampling points.

[0383] Example 9: Extract redundant audio data from subsequent frames and the current frame. The extracted data for the current frame is a non-complete frame.

[0384] The first data length is: 1 frame (i.e., the number of complete audio frames is 1) and 40 sample points (i.e., the number of sample points of incomplete audio frames is 40).

[0385] As shown in Figure 3a(10), the current frame is the Nth frame. The encoding end can extract the audio data of the 40 sampling points of the header of the Nth frame from the previous frame cached in Buffer1 according to the first data length, and extract the N+1th frame to obtain 1 frame and 40 sampling points of redundant audio data.

[0386] Example 10: Extracting redundant audio data from previous and subsequent frames

[0387] The first data length is: 3 frames (i.e., the number of complete audio frames is 3) and 0 sample points (i.e., the number of sample points for incomplete audio frames is 0).

[0388] As shown in Figure 3a(5), the current frame is the Nth frame. The encoding end can extract the N-1th frame, the N+1th frame, and the N+2th frame from the previous frames cached in Buffer1 according to the first data length to obtain 3 frames of redundant audio data.

[0389] Example 11: Extract redundant audio data from previous frames, subsequent frames, and the current frame. The extracted data for the current frame is a non-complete frame.

[0390] The first data length is: 3 frames (i.e., the number of complete audio frames is 3) and 20 sampling points (i.e., the number of sampling points of incomplete audio frames is 20).

[0391] As shown in Figure 3a(11), the current frame is the Nth frame. The encoding end can extract the audio data of the N-1th frame and the 20 sampling points of the head of the Nth frame from the previous frame cached in Buffer1 according to the first data length, as well as extract the N+1th frame and the N+2th frame, to obtain 3 frames and 20 sampling points of redundant audio data.

[0392] Example 12: Extract redundant audio data from previous frames, subsequent frames, and the current frame. The extracted data of the current frame is a complete frame with a first data length of 4 frames (i.e., the number of complete audio frames is 3) and 0 sampling points (i.e., the number of sampling points of a non-complete audio frame is 0).

[0393] As shown in Figure 3a(12), the current frame is the Nth frame. The encoding end can extract the N-1th frame, the Nth frame, the N+1th frame, and the N+2th frame from the previous frames cached in Buffer1 according to the first data length, so as to obtain 4 frames of redundant audio data.

[0394] Of course, the first data length is not limited to the examples above, and the redundant audio data extracted under different first data lengths is not limited to the examples above, and can be determined according to the actual application.

[0395] The encoding end can use an AI encoder to encode the first data length to obtain the aforementioned bitstream.

[0396] Considering that the AI ​​encoder can obtain the first data length in order to acquire redundant audio data of the first data length, the AI ​​encoder can encode the first data length into the bitstream after obtaining the first data length, thereby eliminating the overhead of additional data transmission (such as transmitting the first data length information to the traditional encoder) and improving encoding efficiency.

[0397] Alternatively, the encoding end can also encode the first data length into the bitstream when encoding the current frame via S301 as shown in Figures 2 and 9.

[0398] When S301 is executed by a conventional encoder, the AI ​​encoder can transmit the obtained first data length to the conventional encoder so that the conventional encoder can encode the first data length into the aforementioned bitstream.

[0399] In some embodiments, the information on the number of complete audio frames in the first data length may include first indication information (e.g., flag1 and the corresponding m1 below), which can be used to indicate that the complete audio frame includes the previous frame and / or subsequent frame of the current frame.

[0400] In some embodiments, the information on the number of samples of incomplete audio frames in the first data length may carry second indication information (e.g., flag2 and the corresponding m2 below), which may be used to indicate whether the number of samples of incomplete audio frames is the number of samples of the previous frame or the number of samples of the subsequent frame.

[0401] For example: The first data length includes flag1, and the number of complete audio frames m1 corresponding to different values ​​of flag1, and optionally includes flag2 and the number of sampling points m2 of incomplete audio frames corresponding to different values ​​of flag2.

[0402] For example, flag1 can take values ​​of 0 and 1 to represent complete audio frames as the previous frame and the subsequent frame, respectively. flag2 can take values ​​of 0 and 1 to represent incomplete audio frames as the previous frame and the subsequent frame, respectively.

[0403] Specifically, taking the current frame as frame N as an example, the first data length includes the following information: flag1 = 0, m1 = 1; flag1 = 1, m1 = 2; flag2 = 0, m2 = 20. This first data length information can be interpreted as follows: redundant audio data includes 3 complete audio frames (frames N-1, N+1, and N+2 respectively), and redundant audio data includes the previous frame with 20 sample points (counting the corresponding points backward from the current frame according to the agreed-upon rules). This previous frame with 20 sample points is the last 20 sample points of frame N-2.

[0404] In this way, the encoding end can encode the first data length of the redundant data into the bitstream, so that the decoding end can obtain the first data length of the redundant audio data encoded in the bitstream. Since the first data length includes the number of complete audio frames and the number of sampling points of incomplete audio frames, the decoding end can determine which audio frames the redundant audio data belongs to based on the first data length, and thus use it to reconstruct which lost audio frames.

[0405] Based on the data structure of the bitstream shown in the above embodiments, the following further illustrates a data structure of the bitstream according to an embodiment of this application, which may further include:

[0406] 10) First data length of redundant audio data (optional): For example, the first data length may include the number of complete audio frames and the number of sample points of incomplete audio frames.

[0407] In other embodiments, the bitstream may include more or fewer fields.

[0408] Referring to any of the embodiments in Figures 2 to 9, please refer to Figure 10, which is a flowchart illustrating another encoding method provided by an embodiment of this application. This encoding method can be applied to the encoding end (e.g., an AI encoder) shown in Figure 1. As shown in Figure 10, this encoding method may include S201 and S202 as shown in Figure 2, and optionally S301. Descriptions of S201, S202, and S301 can be found in the relevant descriptions in the embodiments corresponding to Figure 2, and will not be repeated here.

[0409] As shown in Figure 10, compared to the embodiment corresponding to Figure 2 above, the method may further include the following steps:

[0410] S501: The encoder obtains the first bitrate, which is the encoding bitrate of the AI ​​model (e.g., the AI ​​encoding model).

[0411] When the encoding end executes S202, it can encode the redundant audio data corresponding to the current frame according to the first bit rate to obtain the above bitstream.

[0412] In other embodiments, the encoding rate of the AI ​​encoding model can be fixed and does not need to be determined by the encoding end during the encoding process, thereby improving encoding efficiency.

[0413] It should be understood that this application does not specify the execution order of the three steps S501, S201, and S301.

[0414] The encoding bitrate of this AI encoding model is matched with the compression ratio of the AI ​​model (i.e., the AI ​​encoding model) mentioned in the embodiment of Figure 2 above. That is, adjusting the encoding bitrate of the AI ​​encoding model can cause a change in the compression ratio of the AI ​​encoding model.

[0415] In some embodiments, the encoder can obtain the first bitrate mentioned above based on the total bitrate of the current frame (also known as the encoding bitrate of the current frame).

[0416] The total bitrate is the bitrate used by the encoder when encoding the current frame. The encoder can encode the current frame and its redundant audio data according to the total bitrate to obtain the bitstream mentioned above. The bitstream includes not only the second encoded data of the current frame, but also the first encoded data of the redundant audio data corresponding to the current frame.

[0417] When the encoding end executes the encoding method of the embodiment of this application, it can receive total bitrate information. Then, the encoding end can allocate a lower first bitrate from the total bitrate to the AI ​​encoding model for encoding redundant audio data, and allocate a higher second bitrate for encoding the current frame corresponding to the redundant audio data.

[0418] In some embodiments, when the AI ​​encoding model performs AI encoding on the redundant audio data corresponding to the current frame (e.g., S202 as shown in Figure 2 above), it can encode the redundant audio data using the same bit rate (e.g., the first bit rate) to improve encoding efficiency and enhance the ability to resist packet loss.

[0419] In some embodiments, when the AI ​​encoding model performs AI encoding on the redundant audio data corresponding to the current frame (e.g., S202 as shown in Figures 2, 9, and 10 above), it can also use different first bit rates to encode different numbers of points in the redundant audio data corresponding to the same current frame or different frames of audio data.

[0420] In some embodiments, as the encoding progresses, the current frame encoded by the encoder may change. For example, the current frame may be frame 1, frame 2, frame 3, etc., in the order of encoding. Then, when the AI ​​encoding model encodes redundant audio data corresponding to different audio frames (e.g., frame 1, frame 2, frame 3), it may use the same first bit rate for encoding, or it may use different first bit rates for encoding.

[0421] The encoding end can determine to use a fixed bitrate as the total bitrate for each frame (including the current frame). In this way, encoding each frame with the same total bitrate (including AI encoding and traditional encoding) saves the time consumed in determining the total bitrate for each frame, thereby improving encoding efficiency.

[0422] The total bit rate can also be flexibly adjusted according to changes in channel capacity or channel state information. In this way, different first bit rates can be used to encode the corresponding redundant audio data of different audio frames in the audio sequence.

[0423] For example, when the total bitrate of the current frame is higher than the total bitrate of the previous frame (e.g., the previous frame), the bitrate of the AI ​​encoding model can be increased to obtain a higher bitrate than the bitrate when AI encoding the redundant audio data corresponding to the previous frame, which can then be used as the first bitrate of the current frame.

[0424] As a concrete example, if the total bitrate of the previous frame is 128Kbps and the AI ​​encoding bitrate of the previous frame is 4Kbps, and the total bitrate is increased from 128Kbps to 256Kbps when encoding the current frame, then the encoding end can adjust the encoding bitrate of the current frame and subsequent frames from 4Kbps to a larger bitrate (such as 8Kbps, 10Kbps, etc.) to serve as the encoding bitrate for the redundant audio data corresponding to the current frame and the redundant audio data corresponding to subsequent frames (i.e., the first bitrate used by the AI ​​encoding model to encode the redundant audio data).

[0425] Thus, in this embodiment of the application, when using an AI encoding model to encode redundant audio data, the encoding bitrate can be flexibly adjusted according to the change in the total bitrate of the audio frames, thereby enabling the bitstream to carry more redundant information and improving the anti-interference capability of the audio data during transmission.

[0426] In some embodiments, the encoding bitrate of the second encoded data of the current frame is the second bitrate, and the sum of the first bitrate and the second bitrate is the total bitrate.

[0427] The encoding end can allocate a higher bitrate from the total bitrate as the first bitrate, provided that the size of the second bitrate does not significantly affect the audio quality. This allows the bitstream to carry more redundant information without significantly impacting audio quality, thereby improving the audio data's resistance to interference during transmission.

[0428] The encoding end can dynamically allocate the total bitrate of the audio encoding into a first bitrate (e.g., for encoding in an AI encoder) and a second bitrate (e.g., for encoding in a traditional encoder).

[0429] The traditional encoder is an L2HC encoder, and the total bitrate of the traditional encoder and AI encoder can be dynamically allocated. For example, with a total bitrate of 32Kbps, if the AI ​​encoder redundantly encodes a 20ms digital audio signal, consuming 20 bits, and its bitrate is 20Kbps, then the L2HC allocates 320bit - 20bit = 300bits, meaning its bitrate is 30Kbps. In constant bitrate mode (CBR), the sum of the total bitrates of the L2HC and AI encoders remains constant. In CBR mode, to avoid affecting the audio quality of normal L2HC decoding, the bitrate allocated to the AI ​​encoder can decrease as the total bitrate decreases.

[0430] Thus, in this embodiment of the application, when performing traditional encoding on audio frames and AI encoding on redundant audio data, the bitrate of traditional encoding and the bitrate of AI encoding can be flexibly adjusted within the total bitrate of the audio frames. This allows the bitstream to carry more redundant information and improve the anti-interference of audio data during transmission without significantly affecting the audio quality of the traditional encoding bitrate.

[0431] The encoder can obtain channel state information and determine the fixed total code rate to be used during encoding based on the obtained channel state information.

[0432] When the channel state information indicates a low packet loss rate on the transmission channel, the encoder can set a higher fixed total code rate. Otherwise, the encoder can set a lower fixed total code rate.

[0433] A lower packet loss rate indicates a higher quality transmission channel and a lower probability of packet loss during data transmission. In this embodiment, when the packet loss rate of the transmission channel is low, a higher fixed total bit rate can be set, thereby increasing the amount of data transmitted in the same amount of time and improving data transmission efficiency.

[0434] In some embodiments, the encoder can determine the total bit rate of the current frame based on real-time channel state information.

[0435] For frames other than the current frame, the encoder can determine the total bitrate of the frame in a similar way. In this way, the encoder can dynamically determine the total bitrate of each frame, so that the bandwidth occupied by the total encoded data corresponding to each audio frame can be flexibly changed to meet the needs of various scenarios.

[0436] When the channel state information indicates a high packet loss rate on the transmission channel, the encoder can set a lower total bit rate for the current frame. Otherwise, the encoder can set a higher total bit rate for the current frame, thereby increasing the amount of data transmitted in the same amount of time and improving data transmission efficiency.

[0437] For an introduction to the channel state information and the method of obtaining it, please refer to the relevant description in the embodiment corresponding to Figure 9, which will not be repeated here.

[0438] In conjunction with the description of channel state information in the embodiment corresponding to Figure 9, in the embodiment shown in Figure 10 of this application and the embodiment shown in Figure 9 above, when the encoding end obtains channel state information (e.g., packet loss rate), it can actively detect the packet loss rate at the encoding end without involving the packet loss rate being transmitted back from the decoding end to the encoding end. Therefore, when determining the total bit rate or the first bit rate based on the channel state information, it will not cause delays in audio data transmission or fluctuations in sound quality.

[0439] In some embodiments, the encoder may periodically determine the total bitrate, rather than repeatedly determining the total bitrate before encoding each audio frame or its redundant audio data. In some embodiments, the total bitrate may remain constant if the channel environment does not change.

[0440] In some embodiments, the encoder may also determine the first code rate based on channel state information.

[0441] When the channel state information indicates that the packet loss rate of the transmission channel is high, a higher first code rate can be determined, thereby encoding more redundant information into the code stream. This ensures that even when the packet loss rate of the transmission channel is high, the decoding end can still receive the necessary redundant information to recover the audio data, which in turn improves the anti-interference capability of the audio data during transmission.

[0442] In some embodiments, the encoder may also determine the first bitrate based on the bitrate supported by the AI ​​model of the encoder (e.g., the AI ​​encoding model described above).

[0443] For information on the bitrates supported by the AI ​​encoding model, please refer to the relevant descriptions in the corresponding embodiment in Figure 2, which will not be repeated here.

[0444] With sufficient bandwidth, the encoder can select a higher bitrate from the bitrates supported by the AI ​​model as the first bitrate, thereby encoding more redundant information into the bitstream and improving the anti-interference capability of audio data during transmission.

[0445] The process of determining the first bitrate can be implemented by an AI encoder. Optionally, after determining the first bitrate, the AI ​​encoder also inputs the first bitrate into the AI ​​encoding model so that the AI ​​encoding model encodes the redundant audio data according to the first bitrate.

[0446] In some embodiments, the encoder may also determine the first bitrate based on the first data length of the redundant audio data (e.g., a preset data length, or the first data length of the redundant audio data obtained by S402 as shown in FIG9). When the first data length of the redundant audio data corresponding to the current frame received during AI model inference is large, a lower first bitrate can be determined. In this way, a higher compression rate can be applied to a larger amount of redundant audio data so that the amount of data in the first encoded data of the redundant audio data is not too large, thereby reducing bandwidth consumption.

[0447] It should be understood that the implementation methods for obtaining the first bit rate in this application are not limited to those listed in the above embodiments, and may include other implementation methods. Furthermore, the implementation methods for obtaining the first bit rate mentioned in the above embodiments can be freely combined to comprehensively determine the first bit rate, thereby ensuring that the amount of encoded redundant information meets the requirements of the current scenario.

[0448] Corresponding to the encoding methods of the various embodiments described above, this application also provides a decoding method, which can be executed by a decoding end. For example, this decoding method can be applied to the decoding end 1 and / or decoding end 2 shown in FIG1.

[0449] The decoding end may include, but is not limited to, an AI decoder corresponding to the AI ​​encoder described above, and optionally, a conventional decoder corresponding to the conventional encoder described above.

[0450] The decoding methods of the various embodiments of this application can be implemented by an AI decoder. Optionally, some steps can be implemented by a conventional decoder (e.g., decoding the second encoded data in the bitstream).

[0451] When decoding the second encoded data in the bitstream, the AI ​​decoder of this application can be used, or an encoder for audio encoding that has been disclosed or will be disclosed in the future (i.e., a "traditional decoder") can be used. This application does not limit the use of the decoder.

[0452] Traditional decoders consume less computing resources and have lower computational overhead during operation, and the audio quality of the decoded audio is relatively good. Therefore, traditional decoders can be used to decode the second encoded data.

[0453] The lowest bitrate (or highest compression ratio) used for decoding by a traditional decoder can be higher than the highest bitrate (or lowest compression ratio) used for decoding by an AI decoder.

[0454] An AI decoder can be an encoder that includes an AI model for audio decoding. This application does not impose any restrictions on the internal structure of the AI ​​decoder. This AI model is hereinafter referred to as the AI ​​decoding model, to distinguish it from the AI ​​model at the encoding end (hereinafter referred to as the AI ​​encoding model).

[0455] The network structure of the AI ​​decoding model and the AI ​​encoding model is symmetrical.

[0456] In some embodiments, the decoding end of this application may include a separate AI decoder and a traditional decoder. Alternatively, the AI ​​decoder may be used as a component of a traditional decoder. This application does not limit the adaptation method between the AI ​​decoder and the traditional decoder, and it should be understood that the AI ​​decoder provided in this application can be adapted to various traditional decoders.

[0457] In this embodiment of the application, after the decoding end obtains the first information indicating the loss of the first audio frame, it can obtain the first reconstructed data of the redundant audio data based on the bitstream (including received and unreceived data), thereby realizing the recovery of the lost audio data and finally obtaining continuous audio data.

[0458] Please refer to Figure 11, which is a flowchart of a decoding method provided in an embodiment of this application. As shown in Figure 11, the decoding method may include, but is not limited to, the following steps: S601, S602, and S603.

[0459] S601: The decoding end obtains the first information, which indicates that the first audio frame in the bitstream is lost.

[0460] In some embodiments, the decoder may receive the bitstream corresponding to each audio frame (e.g., periodically). For example, each bitstream corresponding to a frame has a frame identifier (e.g., frame index, or frame number), and the decoder may identify whether any audio frames are missing based on the frame identifier in the received bitstream.

[0461] The first audio frame is frame N. If the decoder received the bitstream corresponding to frame N-1 in the previous cycle, but did not receive the bitstream corresponding to frame N in the current cycle, the wireless transmission chip (e.g., a Bluetooth chip) of the decoder can generate a signal indicating that frame N is lost and send this signal to the AI ​​decoder. The AI ​​decoder then receives this signal to obtain the initial information indicating that frame N is lost. In this scenario, the loss of data packets (e.g., the bitstream corresponding to frame N) is mainly due to poor transmission channel conditions.

[0462] In some embodiments, the decoding end also performs error checking on the received bitstream (e.g., cyclic redundancy check, CRC check, etc.). If the decoding end confirms that the bitstream corresponding to the Nth frame has errors, the data packets composed of that bitstream can be discarded. In this case, optionally, the physical layer of the decoding end can also generate a signal indicating that the Nth frame has been lost and send the signal to the AI ​​decoder so that the AI ​​decoder can obtain the first information.

[0463] In some embodiments, after the decoding end performs error checking on the received bitstream, although it confirms that the bitstream corresponding to the Nth frame has bit errors, unlike related technologies, the decoding end of this application embodiment may not discard the data packets composed of the bitstream to ensure continuous reception of data packets and avoid audio quality stuttering. This is because the AI ​​decoder of this application, after training, can have a certain degree of robustness. Even if the bitstream corresponding to the Nth frame has bit errors, after decoding the bitstream by the AI ​​decoder, it can still obtain reconstructed data of the Nth frame with an audio quality equivalent to that without bit errors.

[0464] It should be understood that the above only lists some possible implementations of obtaining the first information indicating the loss of audio frames. In other embodiments, the first information can also be obtained by other methods. This application does not limit the specific method of obtaining the first information.

[0465] S602: The decoding end obtains the first reconstructed data of redundant audio data based on this bitstream.

[0466] S603: The decoding end obtains the second reconstruction data of the first audio frame based on the first reconstruction data.

[0467] The redundant audio data may include the first audio frame.

[0468] The aforementioned bitstream may include second encoded data of the second audio frame corresponding to the redundant audio data.

[0469] The first reconstructed data is obtained by decoding the first encoded data of the redundant audio data in the bitstream using an AI model (such as the AI ​​decoding model mentioned above).

[0470] For ease of understanding, the first audio frame will be described as a "lost frame" and the second audio frame corresponding to the redundant audio data in S602 will be described as a "correlated frame".

[0471] When encoding the "related frame" (i.e., the "current frame" described by the encoding method), the aforementioned encoding end not only encodes the "related frame," but also uses an AI encoding model to encode the redundant audio data corresponding to the "related frame" to obtain the second encoded data. Therefore, the aforementioned bitstream may include the second encoded data of the "related frame." In addition, the bitstream may also include the first encoded data of the aforementioned redundant audio data corresponding to the "related frame."

[0472] In order to reconstruct the "lost frame", the decoding end can obtain the first reconstruction data containing the redundant audio data of the "lost frame". In this way, the second reconstruction data of the "lost frame" can be obtained using the first reconstruction data.

[0473] Considering that the redundant audio data corresponding to an "associated frame" may not necessarily contain the data of all the sampling points of the "lost frame", taking a frame with 160 sampling points as an example, the total number of sampling points is 160 sampling points. Therefore, in the embodiments of this application, the "associated frame" involved in the first reconstructed data of the redundant audio data obtained based on the bitstream by S602 can be one frame or multiple frames.

[0474] When the first reconstructed data of the redundant audio data involves a single “associated frame”, it means that the redundant audio data corresponding to the “associated frame” can contain the audio data of all the sample points of the “lost frame”.

[0475] When the first reconstructed data of the redundant audio data involves multiple "associated frames", it means that the redundant audio data corresponding to each "associated frame" in the multiple "associated frames" can contain audio data of a portion of the "lost frame", and the set of redundant audio data corresponding to the multiple "associated frames" can contain audio data of all the sampling points of the "lost frame".

[0476] In some embodiments, corresponding to the encoding end described above, the decoding end may also have a buffer for caching the bitstream, denoted as Buffer3. Buffer3 can be used to cache undecoded bitstreams from the encoding end, or undecoded data (e.g., first encoded data) from the bitstreams of the encoding end.

[0477] In some embodiments, the decoding end may further include a buffer, denoted as Buffer4, for buffering reconstructed data of decoded audio frames. The reconstructed data stored in Buffer4 may be obtained by decoding the first encoded data of redundant audio data in the bitstream, or by decoding the second encoded data of "associated frames" in the bitstream.

[0478] In some embodiments, when the decoding end executes the above S602, it can read the first reconstructed data containing the redundant audio data of the "lost frame" (the redundant audio data may include part of the number of points or all of the sampling points of the "lost frame") from Buffer4. The first reconstructed data is also obtained by the decoding end using an AI decoding model to decode the first encoded data in the corresponding bitstream.

[0479] In some embodiments, when the decoding end performs the above S602, it can read the bitstream from Buffer3. The redundant audio data corresponding to the first encoded data in the bitstream can include data of part or all of the sampling points of the "lost frame". Then, the decoding end can use an AI decoding model to decode the read first encoded data so as to obtain the first reconstructed data containing the redundant audio data of the "lost frame" (the redundant audio data can include data of part or all of the sampling points of the "lost frame").

[0480] In some embodiments, when the decoding end executes S602 above, it can decode the bitstream corresponding to the subsequent frame of the received "lost frame". For example, the bitstream has not yet been written to the Buffer3 above, wherein the redundant audio data corresponding to the first encoded data in the bitstream includes data of part or all of the sample points of the "lost frame". Then the decoding end can decode the first encoded data in the bitstream to obtain first reconstructed data containing the redundant audio data of the "lost frame" (the redundant audio data may include data of part or all of the sample points of the "lost frame").

[0481] In some embodiments, considering that the encoder can encode multiple frames and / or any number of sampling points of audio data as redundant audio data when encoding redundant audio data, the first reconstructed data obtained in S602 may not only include the reconstructed data of the "lost frame", but may also include the reconstructed data of audio data extracted from the previous frame and / or the subsequent frame of the "lost frame".

[0482] It should be understood that when the decoding end detects the loss of the first audio frame (e.g., when the first information is obtained), the current frame (i.e. the audio frame to be decoded) corresponding to the current bitstream received by the decoding end can be any subsequent frame of the first audio frame, and this application does not impose any restrictions on this.

[0483] Thus, in the event that the first audio frame in the bitstream is lost (represented by the lost frame), the decoding method of this application can obtain the first reconstructed data containing the redundant audio data of the lost frame based on the bitstream. The first reconstructed data is obtained by decoding the first encoded data of the redundant audio data in the bitstream using an AI model (e.g., an AI decoding model). This enables the rapid recovery of lost audio frames in the event of channel interference, allowing the decoding end to obtain continuous audio frames and improve anti-interference capability.

[0484] In some embodiments, the second audio frame may include at least one of the following: a previous frame of the first audio frame, or a subsequent frame of the first audio frame.

[0485] In other words, the "related frame" can be a frame preceding the "lost frame" and / or a frame following the "lost frame".

[0486] The second audio frame will be illustrated below with different examples, but is not limited to the following three examples.

[0487] Example 1

[0488] In order to reconstruct the first audio frame (i.e., the "lost frame"), if the redundant audio data corresponding to the previous frame (one or more frames, without limitation) of the "lost frame" contains audio data of all the sampling points of the "lost frame", then the decoding end can obtain the first reconstructed data of the redundant audio data corresponding to the previous frame of the "lost frame" in order to reconstruct the "lost frame".

[0489] For example, the "lost frame" is the Nth frame as shown in Figure 3a(1). The preceding frame consists of two frames, namely the N-1th frame and the N-2th frame. For instance, the redundant audio data corresponding to the N-1th frame contains 20 sampling points at the beginning of the "lost frame", and the redundant audio data corresponding to the N-2th frame contains 140 sampling points at the end of the "lost frame". Then, the decoding end can obtain the reconstructed data 1 of the redundant audio data corresponding to the N-1th frame and the reconstructed data 1 of the redundant audio data corresponding to the N-2th frame to obtain the first reconstructed data (containing reconstructed data 1 and reconstructed data 2), so that the second reconstructed data containing the complete number of points of the "lost frame" can be obtained using the first reconstructed data. Among them, the complete "lost frame" has 160 sampling points.

[0490] Example 2

[0491] In order to reconstruct the first audio frame (i.e., the "lost frame"), if the redundant audio data corresponding to the subsequent frames (one or more frames, without limitation) of the "lost frame" contains audio data of all the sampling points of the "lost frame", then the decoding end can obtain the first reconstructed data of the redundant audio data corresponding to the subsequent frames of the "lost frame" in order to reconstruct the "lost frame".

[0492] For example, the "lost frame" is the Nth frame as shown in Figure 3a(1). The subsequent frames are two frames, namely the N+2th frame and the N+3th frame. For example, the redundant audio data corresponding to the N+2th frame contains 60 sampling points of the beginning of the "lost frame", and the redundant audio data corresponding to the N+3th frame contains 100 sampling points of the end of the "lost frame". Then the decoding end can obtain the reconstructed data 1 of the redundant audio data corresponding to the N+2th frame and the reconstructed data 1 of the redundant audio data corresponding to the N+3th frame to obtain the first reconstructed data (containing reconstructed data 1 and reconstructed data 2), so that the second reconstructed data containing the complete number of points of the "lost frame" can be obtained using the first reconstructed data. Among them, the complete "lost frame" has 160 sampling points.

[0493] Example 3

[0494] In order to reconstruct the first audio frame (i.e., the "lost frame"), if the redundant audio data A corresponding to the previous frame (one or more frames, not limited) of the "lost frame" (e.g., a total of 160 sampling points) contains audio data of a portion of the "lost frame" (e.g., 60 sampling points), and the redundant audio data B corresponding to the subsequent frame (one or more frames, not limited) of the "lost frame" contains audio data of another portion of the "lost frame" (e.g., 100 sampling points), then the decoding end can obtain the reconstructed data A of the redundant audio data A and the reconstructed data B of the redundant audio data B to obtain the first reconstructed data (a combination of reconstructed data A and reconstructed data B) in order to reconstruct the "lost frame".

[0495] In this embodiment, to reconstruct a lost audio frame, the decoding method can obtain the reconstructed data of the lost audio frame from the first reconstructed data of the redundant audio data (containing the lost audio frame) corresponding to the previous and / or subsequent frames of the lost audio frame. This allows for flexible selection of the reconstructed data of the lost audio frame from the redundant audio data corresponding to the previous and / or subsequent frames, based on the latency requirements of wireless audio transmission scenarios. This enables the decoding end to utilize the reconstructed data of the redundant audio data obtained from AI model decoding to restore the lost audio data in packet loss scenarios, thereby improving the channel's anti-interference and packet loss capabilities, providing users with a continuous audio experience, and reducing audio stuttering.

[0496] In some embodiments, the redundant audio data in S602 shown in FIG11 may include data of any length in the first audio frame (i.e., the “lost frame”) described above.

[0497] The arbitrary length can be any number of sampling points. For example, if each audio frame in the audio sequence to be decoded has 160 sampling points, then the arbitrary length can be any number of sampling points in the audio data of the "lost frame" from 0 to 160 (excluding 0).

[0498] For example, the second audio frame (also called the "associated frame") is a frame, and the redundant audio data packet corresponding to the "associated frame" contains audio data with 160 sampling points of the "lost frame", that is, the redundant audio data packet contains the complete "lost frame". Then the decoding method of this application embodiment can obtain the reconstructed data (i.e. the first reconstructed data) of the redundant audio data corresponding to the "associated frame" so as to obtain the second reconstructed data of the "lost frame".

[0499] For example, if the second audio frame (also called the "associated frame") is K frames (K≥2, K is a positive integer), then the redundant audio data corresponding to each "associated frame" can contain audio data of the corresponding length li of the "lost frame" of arbitrary length li (e.g., any number of sampling points from 0 to 160 (excluding 0 and 160) here), i=1,2,3,…,K. Then the sum of any length li corresponding to K frames can be greater than or equal to the data length of the "lost frame" (e.g., 160 sampling points).

[0500] The reason why the sum of any lengths li corresponding to K frames is greater than the data length of the "lost frame" (e.g., 160 sampling points) is that, in some embodiments, there may be some identical audio data among the redundant audio data corresponding to different "associated frames".

[0501] For example, the "lost frame" is the Nth frame as shown in Figure 3a(1), and the "associated frames" are the N-1th frame and the N-2th frame, respectively. For example, the redundant audio data 1 corresponding to the N-1th frame contains 30 sampling points of the head of the "lost frame", and the redundant audio data 2 corresponding to the N-2th frame contains 140 sampling points of the tail of the "lost frame". In this way, there is an overlap of 10 sampling points of the "lost frame" between the redundant audio data 1 and the redundant audio data 2.

[0502] In this embodiment, the redundant audio data corresponding to a single audio frame can contain a first audio frame of arbitrary length (i.e., a lost frame). Thus, the redundant audio data corresponding to a single audio frame can contain complete audio frames or incomplete audio frames, making the granularity of the redundant audio data encoded in the bitstream not limited to frame granularity but also usable at the sampling point level. Therefore, in different scenarios of short-range and long-range wireless transmission, and considering the presence and magnitude of latency in traditional encoders, the decoding end can flexibly decode the first encoded data of the redundant audio data corresponding to a single audio frame to obtain data of arbitrary length in the current frame, meeting the audio encoding and decoding requirements of various scenarios, resisting packet loss, and improving anti-interference capabilities.

[0503] In this embodiment of the application, the decoding end may include an AI decoder (e.g., an AI decoding model) corresponding to the AI ​​encoder mentioned in the above encoding method, and optionally may also include a conventional decoder corresponding to the conventional encoder mentioned above.

[0504] In this embodiment, the bitstream obtained by the encoding end through the above encoding method can be decoded by an AI decoder and / or a traditional decoder to obtain the reconstructed data of the current frame; in addition, in this embodiment, the bitstream can be decoded by an AI decoder to obtain the reconstructed data of the redundant audio data corresponding to the current frame.

[0505] The process of a traditional decoder decoding the second encoded data of the current frame can be found in the introduction of the traditional decoder on the encoding method side above (e.g., Figure 5b, Figure 6(2)).

[0506] The following example illustrates the structure and decoding process of an AI decoder by using an AI decoder to decode the first encoded data of redundant audio data corresponding to the current frame in the bitstream.

[0507] The AI ​​decoder may include an AI decoding model (an example of an AI model mentioned in the decoding method of this application embodiment). It should be understood that the AI ​​encoding model and the AI ​​decoding model are mutually corresponding and used in combination.

[0508] Please refer to Figures 7b and 8b. Figure 7b is a structural schematic diagram of an AI decoding model provided in an embodiment of this application, and Figure 8b is a schematic diagram of the process of decoding the first encoded data of redundant audio data by an AI decoding model provided in an embodiment of this application.

[0509] As shown in Figure 7b, the AI ​​decoding model may include, but is not limited to, the inverse quantization module 701 and the decoding module 702.

[0510] As shown in Figure 7b, the dequantization module 701 can be implemented using dequantization techniques such as vector dequantization (VDQ), scalar dequantization (SDQ), or residual vector dequantization (RVDQ). Typically, the dequantization module 701 corresponds to the technique used by the aforementioned quantization module 602 to ensure proper data decoding.

[0511] Similar to the quantization module 602 described above, the dequantization module 701 may optionally further include one or more codebooks, depending on the dequantization technique used. When the dequantization module 701 is implemented using VDQ or RVDQ, it may include one or more codebooks.

[0512] For an introduction to the codebook, please refer to the relevant explanations when introducing the AI ​​encoding model above, which will not be repeated here.

[0513] Furthermore, the decoding module 702 and the encoding module 602 can be symmetrical. For example, the encoding module 602 is a trained CNN, and the decoding module 702 is also a trained CNN, with the same m convolutional layers as the encoding module 601, but in reverse order. This symmetrical structure simplifies design and facilitates implementation. Alternatively, the decoding module 702 and the encoding module 602 can also be asymmetrical. For example, the decoding module 702 is a trained CNN with m' convolutional layers, where m' is an integer greater than 1, and the number of m' can be flexibly set according to the application scenario.

[0514] Furthermore, the decoding module 701 in this application embodiment is not limited to the above-described convolutional neural network, but can also be other network structures. For specific examples, please refer to the description of the network structure of the encoding module 601 above. This application does not limit the network structure of the decoding module 702.

[0515] The codebook used by the inverse quantization module 701 and the network parameters of the decoding module 702 in the AI ​​decoding model can be obtained by pre-training the AI ​​decoding model using the method of the embodiments of this application, so that the AI ​​decoding model converges.

[0516] The codebook used by the quantization module 602 and the codebook used by the dequantization module 701 can be the same.

[0517] In some embodiments, the AI ​​encoding model and AI decoding model described above can be obtained by training them separately or by joint training. This application does not impose any limitations on this.

[0518] Of course, during the process of using the AI ​​decoding model to decode (i.e. inference) the input bitstream, the network parameters and codebook information in the AI ​​decoding model can be continuously optimized. This application does not restrict the training time of the AI ​​decoding model.

[0519] The dataset used for training the AI ​​decoding model can be found in the above introduction to the AI ​​encoding model, and will not be repeated here.

[0520] In some embodiments, similar to AI encoding models, the method of this application can train multiple AI decoding models corresponding to different audio sampling rates in scenarios with different audio sampling rates. For example, one AI decoding model can correspond to a unique audio sampling rate.

[0521] The decoding end of this application embodiment can parse the bitstream and obtain the audio sampling rate information carried by the bitstream (refer to the relevant introduction in the AI ​​encoding model section). Then, based on the obtained audio sampling rate, an AI decoding model matching the audio sampling rate is selected to decode the bitstream and obtain the reconstructed data of the corresponding redundant audio data (PCM data sampled according to the audio sampling rate).

[0522] Similar to AI encoding models, the method in this application embodiment, after training the AI ​​decoding model, can not only obtain the codebook and the network parameters of the AI ​​decoding model, but also the bitrate information supported by the AI ​​decoding model.

[0523] The AI ​​encoding model and the AI ​​decoding model each support the same bitrate information. For more information on bitrate information, please refer to the relevant introduction of the AI ​​encoding model mentioned above.

[0524] After receiving the bitstream, the AI ​​decoding model can extract bitrate information from the bitstream. This bitrate information can indicate the encoding bitrate (i.e., the first bitrate) corresponding to the redundant audio data. The first bitrate may include the encoding bitrate of each frame in the redundant audio data; or, the first bitrate may include an encoding bitrate, which is the uniform encoding bitrate of the redundant audio data.

[0525] Then, the AI ​​decoding model can decode the bitstream based on the acquired bitrate information to obtain reconstructed data of redundant audio data (e.g., PCM data sampled according to the audio sampling rate).

[0526] The bitrate information can include the encoding bitrate of each frame in the redundant audio data, for example, a bitrate range of 2Kpbs to 10Kpbs (no limitation is made here). In this way, during the inference process, the AI ​​decoding model can flexibly use the corresponding bitrate to decode the corresponding frames of data within the redundant audio data based on the encoding bitrate of each frame in the redundant audio data (i.e., an example of the first bitrate) to obtain the reconstructed data of the redundant audio data.

[0527] The following section, using the structure of the AI ​​decoding model shown in Figure 7b as an example, will introduce the decoding process of the AI ​​decoding model shown in Figure 8b.

[0528] As shown in Figure 8b, the AI ​​decoding model can extract the first encoded data from the bitstream and perform inverse quantization on it to obtain a multidimensional feature vector (also called a similar feature vector) that is similar to the multidimensional feature vector generated during the encoding process of the AI ​​encoding model.

[0529] As shown in Figure 7b, the inverse quantization module 701 of the AI ​​decoding model can perform inverse quantization on the first encoded data to obtain a multidimensional feature vector.

[0530] The dequantization module 701 can use RVDQ dequantization technology to perform dequantization operations. Specifically, the dequantization module 701 can include a codebook as shown in Figure 8b, to perform dequantization operations on the first encoded data based on the codebook.

[0531] The codebook shown in Figure 8b can be described in the above description of the codebook shown in Figure 8a. The AI ​​encoding model and the AI ​​decoding model can share a single codebook.

[0532] The first encoded data can be in the form of an index, as shown in Figure 8b. The dequantization module 701 can search for the same index as the first encoded data (e.g., index1) in the codebook, and find one or more feature vectors (referred to as "similar feature vectors") corresponding to the index in the codebook, thereby dequantizing index1 into similar feature vectors (also known as multidimensional feature vectors as shown in Figure 7b).

[0533] Referring again to Figure 8b, the AI ​​decoding model can decode multi-dimensional feature vectors to obtain the first reconstructed data of redundant audio data.

[0534] As shown in Figure 7b, the decoding module 702 of the AI ​​decoding model (specifically a convolutional neural network including m' convolutional layers) can receive multi-dimensional feature vectors and obtain the first reconstructed data of redundant audio data through the convolution operation of the m' convolutional layers.

[0535] For example, the first m'-1 layers in the m' convolutional layer can transform the input feature vector into a corresponding lower-dimensional feature vector, and the last layer (convolutional layer m') can transform the input feature vector into the first reconstruction number (PCM data) of redundant audio data, where m' is a positive integer greater than 1.

[0536] For example, as shown in Figure 7b, after the multidimensional feature vector is input into the decoding module 702, it first undergoes a convolution operation in convolutional layer 1' to obtain feature vector 2'; this feature vector 2' is then input into convolutional layer 2' for convolution operation to obtain feature vector 3', and so on. Finally, after convolution operations in m' convolutional layers, the first reconstructed data of the redundant audio data can be obtained. The network parameters (e.g., weight parameters) in the multiple convolutional layers can be obtained by training the AI ​​decoding model and then converging.

[0537] In some embodiments, when the decoding end receives the bitstream corresponding to the current frame, if no audio frame loss is detected (e.g., the first audio frame mentioned above), the decoding end can decode the second encoded data of the current frame in the bitstream corresponding to the current frame to obtain the reconstructed data of the current frame; and the decoding end can use an AI model (e.g., an AI decoding model) to decode the first encoded data of the redundant audio data corresponding to the current frame in the bitstream to obtain the reconstructed data of the redundant audio data corresponding to the current frame.

[0538] In some embodiments, the decoding end can cache the reconstructed data of the redundant audio data obtained from decoding, and / or the reconstructed data of the current frame, for example, by writing it into Buffer4 mentioned above.

[0539] In some embodiments, when the decoding end receives the bitstream corresponding to the current frame, if no audio frame loss is detected (e.g., the first audio frame mentioned above), the decoding end can decode the second encoded data in the bitstream to obtain the reconstructed data of the current frame.

[0540] In some embodiments, the decoding end can write the reconstructed data of the current frame obtained by decoding into the Buffer4 mentioned above. In addition, the decoding end can also write the first encoded data of the bitstream corresponding to the current frame or the redundant audio data corresponding to the current frame in the bitstream into the buffer (e.g., the Buffer3 mentioned above).

[0541] In some embodiments, not every audio frame may carry first encoded data with redundant audio data in the bitstream. In such cases, the decoder can determine whether the bitstream has been redundantly encoded and requires AI decoding based on the flag in the bitstream.

[0542] When the value of the flag is 1, it means that the bitstream includes the second encoded data of the current frame, but does not include the first encoded data of AI encoding. For example, a bitstream with the data structure shown in Figure 4(2) can be decoded by the decoder using a traditional decoder (for example, the second encoded data can be decoded using the parsed Header2) to obtain the reconstructed data of the current frame.

[0543] When the value of this flag is 0, it indicates that the bitstream includes the second encoded data of the current frame and the first encoded data encoded by AI. For example, a bitstream with the data structure shown in Figure 4(3) can be decoded by parsing the position information in the bitstream. This position information can indicate the position of the first encoded data and / or the second encoded data in the bitstream. In this way, the decoder can use the position information to identify the positions of the second encoded data and the first encoded data in the bitstream, so that the second encoded data can be decoded by a traditional decoder to obtain the reconstructed data of the current frame, and the first encoded data can be decoded by an AI decoder to obtain the reconstructed data of the redundant audio data corresponding to the current frame.

[0544] The parsing of flag and location information can be achieved by traditional decoders or AI decoders, and this application does not impose any restrictions on this.

[0545] In some embodiments, when the decoding end obtains the first reconstructed data of redundant audio data based on the above-mentioned bitstream (for example, a possible implementation of S602 as shown in FIG11), it can determine the second audio frame corresponding to the first audio frame; then, the AI ​​model of the decoding end (for example, an AI decoding model) can be used to decode the first encoded data of the redundant audio data corresponding to the second audio frame in the above-mentioned bitstream to obtain the first reconstructed data.

[0546] The bitstream decoded by the AI ​​decoding model can be the bitstream corresponding to the received current frame, the bitstream corresponding to the previous frame of the current frame cached in Buffer3, or the bitstream corresponding to the subsequent frame of the current frame. This application does not impose any restrictions on this.

[0547] In this embodiment, the bitstream encoded by the encoding end includes not only the encoded data of the "associated frame" but also the encoded data of the redundant audio data corresponding to the "associated frame". In order to obtain the first reconstructed data containing the redundant audio data of the first audio frame (also known as the "lost frame"), the decoding end can first identify the "associated frame" that constitutes the redundant audio data with the "lost frame", that is, determine the second audio frame corresponding to the first audio frame. Then, the AI ​​decoding model can be used to decode the first encoded data of the redundant audio data (including the "lost frame") corresponding to the "associated frame" in the bitstream corresponding to the "associated frame" to obtain the first reconstructed data of the redundant audio data containing the "lost frame", so as to realize the reconstruction of the "lost frame".

[0548] In some embodiments, the encoder and decoder may agree that the redundant audio data corresponding to the current frame of the encoder is the previous frame, the subsequent frame, or both, and the length of the redundant audio data (e.g., a preset length) may be agreed upon. Then the decoder can find the "related frame" with the "lost frame" as redundant audio data, i.e., the second audio frame, based on the agreement, so as to improve decoding efficiency.

[0549] In other embodiments, the decoding end can decode information about the first data length of redundant audio data corresponding to any audio frame from the bitstream, and determine whether the redundant audio data contains part or all of the sampled audio data of the "lost frame" based on the first data length, so as to find the "related frame" in which the redundant audio data contains the "lost frame".

[0550] In this embodiment, by first determining the second audio frame (i.e., the "associated frame") corresponding to the first audio frame, and then performing AI decoding on the first encoded data of the redundant audio data corresponding to the second audio frame, the first reconstructed data of the redundant audio data corresponding to the second audio frame is obtained, and then the first reconstructed data of the first audio frame (i.e., the "lost frame") is obtained from it. This allows only the encoded data that helps to reconstruct the first audio to be decoded, avoiding useless decoding operations and improving the overall decoding efficiency.

[0551] Referring to Figure 11 or any of the above embodiments, please refer to Figure 12, which is a flowchart illustrating another decoding method provided in this application embodiment. This decoding method can be applied to the decoding end 1 and / or decoding end 2 (e.g., an AI decoder) shown in Figure 1. As shown in Figure 12, this decoding method may include, but is not limited to, the following steps:

[0552] S601: The decoding end obtains the first information, which indicates that the first audio frame in the bitstream is lost.

[0553] Please refer to the description in the embodiment shown in Figure 11 above for this step, which will not be repeated here.

[0554] S701: Based on the above bitstream, the decoding end obtains the first encoded data of the redundant audio data corresponding to the second audio frame.

[0555] The redundant audio data includes the first audio frame mentioned above, and the bitstream also includes the second encoded data of the second audio frame;

[0556] S702: The decoding end uses an AI model to decode the first encoded data to obtain the first reconstructed data of the redundant audio data.

[0557] After executing S701 and S702 sequentially at the decoding end, S603 as shown in Figure 11 above can be executed.

[0558] S603, the decoding end can obtain the second reconstruction data of the first audio frame (i.e. the "lost frame" mentioned above) based on the first reconstruction data.

[0559] Based on the above description, the first audio frame is also called the "lost frame", and the second audio frame is also called the "related frame".

[0560] It should be understood that S701 and S702 above can be one implementation of S602 shown in Figure 11. For "lost frame" and "associated frame", please refer to the relevant description in the embodiment of Figure 11.

[0561] Based on the above description, the source of the first encoded data of the redundant audio data of the "associated frame" can be the undecoded bitstream buffered in Buffer3, i.e., the "associated frame" is the previous frame. Alternatively, it can come from the bitstream corresponding to the subsequent frame of the current frame that has not yet been received by the decoding end, i.e., the "associated frame" is the subsequent frame.

[0562] The implementation methods formed by S601, S701, S702 and S603 will be introduced below with reference to Figure 3b as a specific example.

[0563] As shown in Figure 3b(1), taking the Nth frame as an example of a "lost frame":

[0564] Example 4: "Related frame" refers to a subsequent frame (e.g., frame N+1).

[0565] Please refer to Figure 3b(2). When the encoding end encodes the (N+1)th frame, the redundant audio data corresponding to the encoded (N+1)th frame is the Nth frame. Then, when the decoding end finds the Nth frame to be a "lost frame", it can determine that the "associated frame" (an example of the second audio frame) is the (N+1)th frame shown in Figure 3b(2). When the decoding end determines that the Nth frame is lost through S601, it can decode the bitstream corresponding to the currently received (N+1)th frame. Specifically, it can decode the first encoded data of the redundant audio data carried in the bitstream, thereby reconstructing the second reconstructed data of the Nth frame. In this example 4, since the redundant audio data of the (N+1)th frame is only the complete Nth frame and does not contain other frames, the second reconstructed data is the same as the first reconstructed data.

[0566] When the "associated frame" is an audio frame after the N+1th frame, the implementation method is the same as in Example 4, and will not be repeated here.

[0567] Example 5: The “related frame” is the previous frame of the “lost frame”.

[0568] Please refer to the dashed arrows from S701 to S702 in Figure 12. In the case of the loss of the Nth frame, the first encoded data obtained by the decoding end through S701 may not be obtained by decoding the bitstream corresponding to the current frame (e.g., in the case of Example 4) or the bitstream corresponding to the subsequent frames of the current frame. Instead, it may be the first encoded data of the redundant audio data corresponding to the previous frame of the current frame (e.g., the N-1th frame) extracted from Buffer3.

[0569] Specifically, please refer to Figure 3b(3). When the encoding end encodes the (N-1)th frame, the redundant audio data of the (N-1)th frame is the Nth frame. Then, when the decoding end finds the Nth frame to be a "lost frame", it can be determined that the "related frame" (an example of the second audio frame) is the (N-1)th frame shown in Figure 3b(3). Since the redundant audio data corresponding to the (N-1)th frame is only the Nth frame, when the decoding end recovers the previously lost audio frame, it does not decode the first encoded data of the redundant audio data of the (N-1)th frame. This first encoded data is buffered in Buffer3. Therefore, when the decoding end determines that the Nth frame is lost through S601, it can extract the first encoded data of the redundant audio data of the (N-1)th frame from Buffer3 and decode the first encoded data to obtain the second reconstructed data of the Nth frame. In this example 5, since the redundant audio data of the (N-1)th frame is only the complete Nth frame and does not contain other frames, the second reconstructed data is the same as the first reconstructed data.

[0570] When the "associated frame" is an audio frame before the (N-1)th frame, the implementation method is the same as in Example 5, and will not be repeated here.

[0571] Optionally, as shown in Figure 12, after obtaining the first reconstructed data of the redundant audio data of the second audio frame through S702, S901 can also be executed.

[0572] S901: The decoder stores the first reconstructed data of redundant audio data to obtain the cached reconstructed data.

[0573] The decoding end can save the first reconstructed data to Buffer4 on the decoding end.

[0574] Referring to Figure 3a(9), when the redundant audio data encoded at the encoding end for the Nth frame is the data of the first 80 sampling points of the Nth to N+2th and N+3th frames, after the decoding end obtains the first reconstructed data of the redundant audio data corresponding to the Nth frame (as the "associated frame", i.e. the second audio frame) through S702, it can write the reconstructed data of the first 80 sampling points of the Nth to N+2th and N+3th frames into Buffer4.

[0575] In this embodiment, when an audio frame is lost (hereinafter referred to as a "lost frame"), the first reconstructed data containing the redundant audio data of the lost frame can be saved. Thus, if subsequent frames after the first audio frame are lost at the decoding end, the decoding end can directly extract part or all of the reconstructed data of the lost subsequent frames from the saved reconstructed data of the decoded redundant audio data, without needing to decode the currently received, future, or already received bitstream to obtain the reconstructed data of the lost subsequent frames, thereby improving decoding efficiency.

[0576] The execution order of S901, S801, and S1001 is not limited in this embodiment.

[0577] Optionally, referring to Figure 12, when the "associated frame" containing the "lost frame" is a previous frame of the current frame received by the decoding end, the decoding end may obtain the first reconstructed data of redundant audio data through S801 instead of S701 and S702.

[0578] S801: The decoding end can obtain the first reconstructed data of the redundant audio data corresponding to the previous frame that has been decoded and saved by the above-mentioned bitstream (bitstream in S601).

[0579] As shown in Figure 12, the decoder can extract the first reconstructed data containing the "lost frame" from the reconstructed data (e.g., Buffer4) of the cached redundant audio data.

[0580] S801 is in the order of S603. S801 can be another decoding implementation method that is parallel to the technical means constituted by S701 and S702.

[0581] The implementation of S601, S801, and S603 will be introduced below with reference to Figure 3b and specific example 6.

[0582] As shown in Figure 3b(1), let's continue to use the "lost frame" as the Nth frame as an example:

[0583] Example 6: The “associated frame” is the previous frame of the “lost frame” (e.g., the (N-1)th frame), and before the Nth frame is determined to be lost by S601, the redundant audio data corresponding to the “associated frame” has been decoded and saved (e.g., saved to Buffer4).

[0584] Please refer to Figure 3b(4). When the encoder encodes the (N-1)th frame, the redundant audio data of the (N-2)th frame is the (N-2)th frame and the Nth frame. When the decoder finds the Nth frame to be a "lost frame", it can determine that the "related frame" (an example of the second audio frame) is the (N-1)th frame shown in Figure 3b(4). Since the redundant audio data corresponding to the (N-1)th frame includes the Nth frame and the (N-2)th frame, when the decoder recovers the previously lost (N-2)th frame using the first encoded data of the redundant audio data of the (N-1)th frame, it has already decoded the first encoded data of the redundant audio data of the (N-1)th frame and cached the first reconstructed data obtained from the decoding in Buffer4. Therefore, when the decoder determines that the Nth frame is lost through S601, it can extract the first reconstructed data of the redundant audio data of the (N-1)th frame from Buffer4 and extract the second reconstructed data of the Nth frame from the first reconstructed data. In this example 6, since the redundant audio data of the (N-1)th frame includes not only the complete Nth frame but also the complete N-2th frame, the second reconstructed data is only a part of the first reconstructed data.

[0585] The above example illustrates a scenario where both traditional encoders and AI encoders have no latency (i.e., both have 0 latency).

[0586] In Example 4 above, the "related frame" (frame N+1) is the frame following the "lost frame" (frame N). In Examples 5 and 6 above, the "related frame" (frame N-1) is the frame preceding the "lost frame" (frame N). In practical applications, the "related frame" can also be other subsequent frames such as frame N+2 or frame N+3, or multiple frames among the subsequent frames. The "related frame" can also be other previous frames such as frame N-2 or frame N-3, or multiple frames among the previous frames. This application embodiment does not limit this.

[0587] In the above example, the redundant audio data corresponding to the "associated frame" includes all the data of the Nth frame. In practical applications, the redundant audio data corresponding to the "associated frame" may also include some points of the Nth frame. In this case, at least two examples from Examples 4 to 6 can be combined to obtain all the reconstructed data of the "lost frame", which will not be elaborated here.

[0588] In this embodiment of the application, when an audio frame is lost (hereinafter referred to as "lost frame"), the decoding end can directly extract the first reconstructed data containing the "lost frame" from the reconstructed data of the redundant audio data obtained by the cached decoding, without having to decode the currently received or future received or already received bitstream to obtain the reconstructed data of the "lost frame", thereby improving the overall decoding efficiency.

[0589] Optionally, as shown in Figure 12, the method may further include S1001.

[0590] S1001: The decoding end decodes the target data length of the first audio frame in the redundant audio data from the bitstream. The target data length indicates the amount of data of the first audio frame included in the redundant audio data.

[0591] When the encoding end executes S603 as shown in Figure 12, it can obtain the second reconstructed data of the first audio frame based on the target data length and the first reconstructed data.

[0592] In some embodiments, when the encoding end encodes redundant audio data, it can encode the first data length of the redundant audio data into the bitstream. The decoding end can decode the first data length of the redundant audio data containing the "lost frame" from the bitstream.

[0593] In this way, the decoding end can accurately determine the specific amount of data of the lost first audio frame contained in the redundant audio data, so as to achieve accurate reconstruction of the first audio frame.

[0594] In some embodiments, the first data length may include the number of complete audio frames and the number of sample points of incomplete audio frames.

[0595] For example, the encoder and decoder can agree that the redundant audio data to be encoded is the previous frame of the current frame and the current frame (data of some or all of the sample points). Under this agreed encoding rule, taking Figure 3a (6) as an example, when encoding the Nth frame, the encoder encodes the redundant audio data (the N-1th frame and the first 20 sample points of the Nth frame) using an AI model, and encodes the first data length (specifically 1 frame and 20 sample points) to obtain the bitstream. In the event of a loss of the (N-1)th frame, the decoding end can decode the bitstream to obtain the first data length. According to the agreed rule, the target data length of the redundant audio data corresponding to the Nth frame encoded in the bitstream, which can include the lost (N-1)th frame (an example of the first audio frame), can be determined based on the first data length. The target data length can indicate that the data volume of the (N-1)th frame is one frame (i.e., 160 sampling points). Thus, the first encoded data of the redundant audio data in the bitstream can be decoded to obtain the first reconstructed data of the redundant audio data. Then, based on the target data length, the second reconstructed data of the lost (N-1)th frame can be extracted from the first reconstructed data.

[0596] In this embodiment, since the audio frames in the audio sequence are encoded sequentially (e.g., by frame number), and the encoder and decoder can agree on the types of audio frames that can be carried in the redundant audio data, such as previous frames, subsequent frames, etc. (or information indicating the data length of previous audio data and / or subsequent audio data in the redundant audio data is carried in the bitstream), as well as information on the first data length of the redundant audio data that can be carried in the bitstream. Based on this, the decoder can decode the bitstream to obtain the target data length of the "lost frame" in the redundant audio data of the "associated frame". This target data length can be, for example, information about the number of sampling points or frames of the "lost frame" in the redundant audio data. Then, the decoder can combine the first reconstructed data of the redundant audio data with the target data length to obtain the second reconstructed data of the "lost frame", so as to ensure the accurate and complete reconstruction of the "lost frame" at the decoder.

[0597] When the redundant audio data corresponding to a "correlated frame" contains data of a partial number of points of the "lost frame", that is, the target data length of the "lost frame" corresponding to the "correlated frame" is a partial number of points, the decoding end can comprehensively extract the complete second reconstructed data of the "lost frame" from the multiple first reconstructed data based on the first reconstructed data of the redundant audio data corresponding to multiple "correlated frames" that can cover all the sampling points of the "lost frame", and the target data length of the "lost frame" corresponding to each of the multiple "correlated frames".

[0598] Optionally, referring to Figure 12, the method may further include S1101 and S1102. S1102 is performed after S603. This embodiment does not specifically limit the execution order of S1101, as long as it precedes S1102.

[0599] S1101: The decoding end acquires the third reconstructed data of the third audio frame adjacent to the first audio frame.

[0600] The third reconstructed data of the third audio frame can be obtained by decoding the second encoded data based on the third audio frame, or it can be obtained by decoding the first encoded data containing redundant audio data of the third audio frame in the case of loss of the third audio frame, according to the decoding method of the embodiments of this application. This application does not limit the reconstructed data of the third audio frame.

[0601] For example, if each frame is 10ms, then the third reconstructed data of the third audio frame is the PCM data (i.e., reconstructed data) of the third audio frame for 10ms.

[0602] In some embodiments, the third audio frame may be an adjacent audio frame to the lost first audio frame, such as the frame preceding or following the "lost frame".

[0603] In some embodiments, the first audio frame may be multiple frames, i.e., multiple frames may be lost consecutively. For example, the multiple first audio frames may be the Nth frame, the (N-1)th frame, and the (N-2)th frame. Then the third audio frame may be the (N-3)th frame or the (N+1)th frame.

[0604] S1102: The decoding end performs smoothing processing on the third reconstructed data and the second reconstructed data of the first audio frame obtained through S603 to obtain audio sequence data.

[0605] When smoothing the third and second reconstructed data at the decoding end, one or more of the following methods can be used: windowing functions (such as Hamming window, Hanning window, etc.), moving average filters, Kalman filters, adaptive filters, frequency domain smoothing, etc. The decoding end can select an appropriate smoothing technique according to the specific application scenario and requirements to smooth the reconstructed data of different audio frames.

[0606] For example, as shown in Figure 14, in the case of the loss of the Nth frame, the AI ​​decoder can decode the first encoded data of the redundant audio data corresponding to the N+1th frame as shown in Figure 14(2), thereby obtaining the 10ms decoded data of the Nth frame and the 6.25ms decoded data of the beginning of the N+1th frame as shown in Figure 14(4). Due to the 2.5ms delay, the 10ms decoded data of the Nth frame can include the 2.5ms reconstructed data at the end of the N-1th frame and the 7.5ms reconstructed data at the beginning of the Nth frame.

[0607] Then, the 10ms decoded data of the Nth frame can be smoothed by performing a 2.5ms temporal overlap on the reconstructed data of the previous frame (i.e., the N-1th frame). The 2.5ms temporal overlap data are the first 2.5ms of the decoded data of the Nth frame and the last 2.5ms of the reconstructed data of the N-1th frame. This makes the reconstructed data of the Nth frame obtained by AI decoding more smoothly stitched with the reconstructed data of the adjacent previous frame.

[0608] The above embodiment takes the case of time delay in traditional codec as an example, so that the decoded data of the lost frame obtained by AI decoding (also called reconstructed data in some scenarios) includes not only part of the data of the lost frame, but also part of the data of the previous or next frame (referred to as redundant data). Therefore, this redundant data can be smoothed with the previous or next frame of the lost frame to make the transition between adjacent frames smoother and the listening experience more natural.

[0609] In other embodiments, where there is no latency in the traditional codec, for example, the decoded data of the lost frame obtained by AI decoding is the 10ms PCM data of the lost frame itself, without including part of the data of the previous or next frame, then the decoded data of the lost frame and the reconstructed data of the previous (or next) frame can also be smoothed by means of windowing functions, etc., without limitation.

[0610] In this embodiment of the application, by smoothing the reconstructed data of the lost first audio frame and the reconstructed data of its adjacent audio frames, problems such as unnatural sound, uneven connection between adjacent frames or multiple frames, and unnatural transitions in the reconstructed data of each audio frame obtained by decoding can be avoided, making the sound of the obtained audio sequence more natural and smooth.

[0611] Furthermore, the above embodiments all use the lost first audio frame as an example for illustration. The encoding and decoding method of this application embodiment can support the scheme of reconstructing multiple consecutive frames. In the case where the lost first audio frame is multiple frames, since the redundant audio data corresponding to a single "associated frame" can carry multiple audio frames, the decoding method of this application embodiment can also execute the decoding method of this application embodiment once when multiple first audio frames are detected to be lost, so as to realize the restoration and reconstruction of the lost multiple first audio frames, so as to have the ability to resist and restore continuous packet loss.

[0612] Referring to any of the embodiments in Figures 11 and 12, please refer to Figure 13, which is a flowchart illustrating another decoding method provided by an embodiment of this application. This decoding method can be applied to the decoding end 1 and / or decoding end 2 (e.g., an AI decoder) shown in Figure 1. As shown in Figure 13, the decoding method may include S601 as shown in Figure 11, and S702 and S801 as shown in Figure 12.

[0613] As shown in Figure 12, the method may also include, but is not limited to, the following step S1201. The order of S1201 is after S601.

[0614] S1201: The decoding end can query the cache unit. If the cache unit has the first encoded data (a) of the redundant audio data (including the data of the first audio frame) corresponding to the previous frame, then S702 is executed to perform AI decoding on the first encoded data to obtain the first reconstructed data of the redundant audio data. If the cache unit has the first reconstructed data (b) of the redundant audio data (including the data of the first audio frame) corresponding to the previous frame, then S801 is executed to obtain the first reconstructed data of the redundant audio data. If the cache unit does not have the first encoded data (a) of the redundant audio data (including the data of the first audio frame) corresponding to the previous frame, and the cache unit also does not have the first reconstructed data (b) of the redundant audio data (including the data of the first audio frame) corresponding to the previous frame, then S1301 is executed.

[0615] The buffer units include Buffer3 and Buffer4. Referring to the relevant description above, Buffer3 can buffer undecoded bitstream from the encoder, or undecoded data (e.g., first encoded data) from the bitstream from the encoder. Buffer4 can buffer reconstructed data obtained by decoding the first encoded data of redundant audio data in the bitstream. When the decoder queries the buffer units, it can first query Buffer4. If Buffer4 does not contain the aforementioned (b), it will then query Buffer3, thereby minimizing decoding operations and improving overall decoding efficiency.

[0616] S1301: The decoding end obtains the first encoded data of the redundant audio data corresponding to the subsequent frame, which includes the data of the first audio frame.

[0617] It is understandable that the decoding end may need to wait for a period of time before it can receive the bitstream corresponding to the subsequent frame that is the "associated frame".

[0618] After executing S1301, the decoding end can execute S702 to perform AI decoding on the first encoded data to obtain the first reconstructed data of the redundant audio data.

[0619] After executing S701 or S801, the decoding end can execute S603, referring to the embodiment corresponding to Figure 11 or Figure 12 above, to obtain the second reconstruction data of the first audio frame based on the first reconstruction data.

[0620] In this embodiment, when the reconstructed data corresponding to the first audio frame already exists in the cache unit, it can be directly obtained; when the first encoded data has been received at the decoding end, AI decoding can be performed directly. If neither of the above two conditions applies, the system waits for the bitstream corresponding to the future audio frame and performs AI decoding on the first encoded data therein. This maximizes the overall decoding efficiency.

[0621] The encoding and decoding method of this application will be described below with reference to specific embodiments:

[0622] Specific Implementation Example 1

[0623] The following uses a traditional encoder, L2HC, as an example to illustrate the implementation process of the encoding method in this application:

[0624] In traditional codecs, each audio frame to be processed is the PCM data corresponding to the audio frame. The duration of the PCM data corresponding to each audio frame is 10ms, that is, the duration of the current frame to be encoded is 10ms. Taking the current frame as the N+1th frame as an example.

[0625] As shown in Figure 6, although the current frame input to the traditional encoder is 10ms, due to the mathematical principle of MDCT and the effect of the low-delay window function, the actual data size encoded by the traditional encoder (specifically L2HC) is 20ms, which is 10ms for the current frame, 6.25ms for the end of the previous frame (i.e., the Nth frame), and 3.75ms for the beginning of the next frame (i.e., the N+2th frame).

[0626] Referring to Figure 6, the decoded data obtained by the traditional decoder for the current frame is 10ms, which includes the first 7.5ms of the current frame and the last 2.5ms of the previous frame. In other words, although the 10ms encoded by the traditional encoder is the PCM data of the current frame, the traditional decoder has a 2.5ms delay in decoding the 10ms of the current frame, and can only decode the first 7.5ms of the current frame and the last 2.5ms of the previous frame.

[0627] Furthermore, referring to Figure 6, in the conventional decoder, the last 6.25ms of temporal aliasing data in the 16.5ms temporal aliasing data obtained from decoding the current frame's bitstream is superimposed with the temporal aliasing data of the next frame to obtain the 10ms of decoded data for the next frame (specifically including the 6.25ms of decoded data for the next frame). However, the conventional codec has a 2.5ms delay, meaning that the 6.25ms of decoded data for the next frame actually includes the last 2.5ms of data from the current frame and the first 3.75ms of data from the next frame. Therefore, if the current frame is lost, it will not only affect the reconstruction of the current frame but also the reconstruction of the first 3.75ms of the next frame.

[0628] To combat packet loss in the Nth frame, please refer to Figure 14. As shown in Figure 14(1), the traditional encoder can encode the current frame, specifically the N+1th frame, and use the AI ​​encoder to encode the redundant audio data corresponding to the current frame (N+1th frame). This redundant audio data is the audio data affected by the loss of the Nth frame, specifically the first 3.75ms of the current frame, the previous frame (specifically the Nth frame at 10ms), and the last 2.5ms of the frame before that, as shown in Figure 14(2). This is because when the encoded data obtained by the traditional encoder for the Nth frame is lost, as described in Figure 6 above, the loss of the Nth frame will not only affect the reconstructed data of the Nth frame, but also the reconstructed data of the N+1th frame and the N-1th frame. Based on this, the redundant audio data corresponding to the current frame encoded by the AI ​​encoder must at least cover these data. Optionally, the redundant audio data corresponding to the current frame encoded by the AI ​​encoder may also include more data. For example, the data length of the audio data at the end of the (N-1)th frame in the redundant audio data may be greater than 2.5ms, and / or the data length of the audio data at the beginning of the (N+1)th frame in the redundant audio data may be greater than 3.75ms. This facilitates smoothing processing on the decoding side.

[0629] As shown in Figure 14(3), the traditional decoder can decode the current frame (the N+1th frame) shown in Figure 14(1). As shown by the dashed line, the actual decoded data of the N+1th frame is the 2.5ms at the end of the Nth frame and the 7.5ms at the beginning of the N+1th frame, with a delay of 2.5ms.

[0630] As shown in Figure 14(4), the AI ​​decoder can decode 16.5ms of redundant audio data corresponding to the current frame as shown in Figure 14(2). As shown in Figure 14(4), this redundant audio data includes the decoded data of the Nth frame and the decoded data of the first 6.25ms of the N+1th frame. However, due to a 2.5ms delay, as shown by the dashed line, the decoded data of the Nth frame is actually the last 2.5ms of the N-1th frame and the first 7.5ms of the Nth frame. The decoded data of the first 6.25ms of the N+1th frame is actually the last 2.5ms of the Nth frame and the first 3.75ms of the N+1th frame. In summary, the redundant audio data that the AI ​​decoder can decode is the last 2.5ms of the N-1th frame, the 10ms of the Nth frame where packet loss occurred, and the first 3.75ms of the current frame (the N+1th frame).

[0631] In this way, if the Nth frame is lost, the 10ms data of the Nth frame can be reconstructed by decoding the redundant audio data corresponding to the next frame (specifically the N+1th frame).

[0632] Specific Implementation Example 2

[0633] The contents of Specific Embodiment 2 are mostly the same as those of Specific Embodiment 1. The main difference is that the redundant audio data of the current frame can resist not only the loss of the previous frame, but also the loss of the next frame.

[0634] In this embodiment, taking the current frame as the Nth frame as an example, the data length of the redundant audio data corresponding to the current frame encoded by the AI ​​encoder can be determined based on the channel conditions and the buffer on the encoding side.

[0635] When channel conditions are poor or the buffer is congested, packet loss is more likely to occur. In this case, the AI ​​encoder can cache more audio frames to prevent packet loss. This allows it to extract redundant audio data with a length of (2.5ms + k * 10ms + 3.75ms) from the cached audio frames, where k is an integer greater than or equal to 1.

[0636] Taking k=3 as an example, as shown in Figure 15, the current frame is the Nth frame. The Nth frame shown in Figure 15(1) can be encoded by a traditional encoder, and the redundant audio data corresponding to the Nth frame shown in Figure 15(2) can be encoded by an AI encoder. The redundant audio data can include the last 2.5ms of the N-2th frame, the N-1th frame, the Nth frame, the N+1th frame, and the first 3.75ms of the N+2th frame to resist packet loss in the N-1th and N+1th frames.

[0637] In one possible implementation, the AI ​​encoder can determine the length of redundant audio data based on the signal strength of the transmission channel at the encoding end, and / or the buffer state in the AI ​​encoder's buffer.

[0638] For example, with the signal strength based on RSSI-80dBm, the value of k increases by 1 for every 3dB decrease, thus adding a redundant audio frame.

[0639] For example, if the buffer typically outputs one frame before inputting another, and if no audio frames can be written to the buffer within a preset duration, the AI ​​encoder can increment the number of redundant audio frames based on the duration.

[0640] In some embodiments, under normal or minimally interfered wireless transmission channel conditions, the redundant audio data encoded by the AI ​​encoder can be 1 to 2 frames to provide active packet loss resistance. When the channel is interfered with, the number of redundant audio data frames encoded by the AI ​​encoder can be increased. However, due to dynamic bitrate allocation, the bitrate sampled by the AI ​​encoder should not exceed the difference between the total bitrate and the minimum bitrate of the traditional codec (to ensure sound quality), in order to guarantee the sound quality of the audio from the traditional codec.

[0641] Referring to Figure 15, after the decoding end receives the bitstream, the traditional decoder can decode the Nth frame as shown in Figure 15(3). The decoded data is actually the last 2.5ms of the N-1th frame and the first 7.5ms of the Nth frame.

[0642] In some embodiments, under normal no-packet-loss scenarios, the decoding side only needs to decode the L2HC encoded data in the bitstream.

[0643] In this embodiment, for example, if the (N-1)th frame is lost, as shown in Figure 15(4), the AI ​​decoder can decode the redundant audio data corresponding to the current frame, specifically including 10ms of decoded data for the current frame (the Nth frame), 10ms of decoded data for the previous frame (the N-1th frame), 10ms of decoded data for the next frame (the N+1th frame), and 6.25ms of decoded data at the beginning of the N+2th frame. These decoded data are actually: the last 2.5ms of the N-2th frame as shown in Figure 15(2), 10ms of data for the N-1th frame, 10ms of data for the Nth frame, 10ms of data for the N+1th frame, and 3.75ms of data at the beginning of the N+2th frame.

[0644] After the N-1 frame is lost, the AI ​​decoder can use the decoding method of this application embodiment to obtain the 2.5ms at the end of the N-2 frame, the 10ms data of the N-1 frame, the 10ms data of the N frame, the 10ms data of the N+1 frame, and the 3.75ms data at the beginning of the N+2 frame as shown in Figure 15(2), and cache these reconstructed data in Buffer4, thereby realizing the reconstruction of the lost N-1 frame.

[0645] If packet loss also occurs in the next N+1 frame, the 10ms data of the N+1 frame (i.e., reconstructed data) obtained by the AI ​​decoder from the current cache can be used directly without performing AI decoding again.

[0646] Furthermore, in some embodiments, the decoding end can also use the 10ms data of the N+1th frame and the first 3.75ms data of the N+2th frame obtained by the AI ​​decoder to smooth the reconstructed data (10ms of PCM data) of the N+2th frame, thereby improving the smoothness of the audio. Specifically, this involves smoothing data that overlaps in the temporal domain, for example, smoothing the first 3.75ms data of the N+2th frame obtained by the AI ​​decoder with the first 3.75ms data of the reconstructed data of the N+2th frame.

[0647] In addition, in some embodiments, the decoding end can also use the 2.5ms data at the end of the N-2th frame and the 10ms data of the N-1th frame, as shown in Figure 15(2), which are decoded by the AI ​​decoder, to smooth the reconstructed data (10ms of PCM data) of the N-2th frame that has already been decoded. Specifically, the data that overlaps in the temporal domain is smoothed, for example, the 2.5ms data at the end of the N-2th frame obtained by the AI ​​decoder is smoothed with the 2.5ms data at the end of the reconstructed data of the N-2th frame that has already been decoded.

[0648] When smoothing the preceding and following frames, one possible implementation is that the AI ​​encoder can use a sample point weighted average to smooth the overlapping audio data and finally output PCM audio data.

[0649] For example, the reconstructed data of frame N+1 obtained by AI decoding, the data of the first 3.75ms of frame N+2 (e.g., M sampling points), and the data of the first 3.75ms of the reconstructed data of frame N+2 (e.g., M sampling points) are smoothed. This smoothing can be done using a sampling point weighted average strategy.

[0650] For example, the M sampling points (i.e., sequence A) of the first 3.75ms of data in the (N+2)th frame are denoted as ai, i = 1, 2, 3, ..., M; the M sampling points (i.e., sequence B) of the first 3.75ms of data in the reconstructed data of the (N+2)th frame obtained from decoding are denoted as bi, i = 1, 2, 3, ..., M. In this example, the number of sampling points that need to be smoothed in both frames is M, so the smoothing step size can be 1 / M. Then, the sampling point value ci of the M sampling points of the sequence C containing M sampling points obtained after smoothing the A and B sequences is shown in Formula 1, Formula 2 and Formula 3: ci = (1 - (i-1) / M) * ai + (i-1) / M * bi, Formula 1;

[0651] In Formula 1, i = 2, 3, ..., M-1. c1 = a1 (Formula 2); cM = bM (Formula 3).

[0652] Of course, the weighted average strategy is used as an example to illustrate the smoothing method. In other embodiments, the way to calculate the weights of ai and bi is not limited to the example in Formula 1, and the smoothing strategy is not limited to the weighted average strategy. Other strategies can be used, and no restrictions are placed here.

[0653] This application embodiment also provides an encoding device 1600. Please refer to FIG16, which is a schematic diagram of the structure of the encoding device 1600. The encoding device 1600 may include, but is not limited to, a data acquisition module 1601 and a data encoding module 1602.

[0654] The data acquisition module 1601 is used to acquire redundant audio data of the first data length corresponding to the audio frame to be encoded.

[0655] The data encoding module 1602 is used to encode redundant audio data using an AI model to obtain a bitstream. The bitstream includes first encoded data of the redundant audio data and second encoded data of the audio frame to be encoded. The compression rate of the AI ​​model is higher than the compression rate of the audio frame to be encoded.

[0656] In some possible implementations, the redundant audio data is extracted from previous audio frames and / or subsequent audio frames of the audio frame to be encoded, and the first data length includes any number of frames, or any number of frames plus any number of sampling points.

[0657] In some possible implementations, the redundant audio data may also include data of arbitrary length from the audio frame to be encoded.

[0658] In some possible implementations, the data encoding module 1602 is further configured to encode a first data length of the redundant audio data into the bitstream, the first data length including the number of complete audio frames and the number of sampling points of incomplete audio frames.

[0659] In some possible implementations, the data acquisition module 1601 is further configured to acquire channel state information, which indicates the packet loss rate of the transmission channel of the code stream; and acquire a first data length based on the channel state information.

[0660] In some possible implementations, the data acquisition module 1601 is further configured to obtain a first bitrate based on the encoding bitrate of the audio frame to be encoded, wherein the first bitrate is the encoding bitrate of the AI ​​model; the data encoding module 1602 is further configured to use the AI ​​model to encode redundant audio data according to the first bitrate to obtain first encoded data.

[0661] In some possible implementations, the code rate of the second encoded data is the second code rate, and the sum of the first code rate and the second code rate is the encoding code rate.

[0662] It should be understood that the encoding device 1600 may also include other modules such as a communication module.

[0663] It should be understood that the encoding device 1600 can also be used to perform the encoding methods described in the above embodiments, which will not be repeated here.

[0664] The effects and implementation methods of the encoding device 1600 in the above embodiments are similar to those of the encoding methods in the above embodiments, and will not be repeated here.

[0665] This application embodiment also provides a decoding device 1700. Please refer to FIG17, which is a schematic diagram of the structure of the decoding device 1700. The decoding device 1700 may include, but is not limited to, an information acquisition module 1701 and a decoding module 1702.

[0666] The information acquisition module 1701 is used to acquire first information, which indicates that the first audio frame in the bitstream is lost.

[0667] The decoding module 1702 is used to obtain first reconstructed data of redundant audio data based on the bitstream. The redundant audio data includes a first audio frame. The first reconstructed data is obtained by decoding the first encoded data of the redundant audio data in the bitstream using an AI model. The bitstream also includes second encoded data of the second audio frame corresponding to the redundant audio data.

[0668] The decoding module 1702 is also used to obtain second reconstructed data of the first audio frame based on the first reconstructed data.

[0669] In some possible implementations, the second audio frame includes at least one of the following: a previous audio frame of the first audio frame, or a subsequent audio frame of the first audio frame.

[0670] In some possible implementations, the redundant audio data may also include data of arbitrary length in the first audio frame.

[0671] In some possible implementations, the decoding module 1702 is specifically used to: determine the second audio frame corresponding to the first audio frame; and use an AI model to decode the first encoded data of the redundant audio data corresponding to the second audio frame in the bitstream to obtain the first reconstructed data.

[0672] In some possible implementations, the second audio frame includes the previous audio frame, and the decoding module 1702 is specifically used to: acquire first reconstructed data of the redundant audio data corresponding to the previous audio frame that has been decoded and saved by the bitstream.

[0673] In some possible implementations, the decoding module 1702 is also used to store the first reconstructed data.

[0674] In some possible implementations, the decoding module 1702 is further configured to decode from the bitstream to obtain the target data length of the first audio frame in the redundant audio data, the target data length indicating the amount of data of the first audio frame included in the redundant audio data.

[0675] In some possible implementations, the decoding module 1702 is specifically used to: obtain second reconstructed data of the first audio frame based on the target data length and the first reconstructed data.

[0676] In some possible implementations, the decoding module 1702 is further configured to acquire third reconstructed data of a third audio frame adjacent to the first audio frame; and to smooth the third reconstructed data and the second reconstructed data to obtain audio sequence data.

[0677] It should be understood that the decoding device 1700 may also include other modules such as a communication module.

[0678] It should be understood that the decoding device 1700 can also be used to perform the decoding methods described in the above embodiments, which will not be repeated here.

[0679] The effects and implementation methods of the decoding device 1700 in the above embodiments are similar to those of the decoding methods in the above embodiments, and will not be described again here.

[0680] In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware encoding processor, or by a combination of hardware and software modules in the encoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0681] The memory mentioned in the above embodiments can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0682] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0683] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0684] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0685] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0686] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0687] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0688] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

An encoding method characterized by comprising: The method comprises: acquiring redundant audio data corresponding to a first data length of a to-be-encoded audio frame; encoding the redundant audio data by using an artificial intelligence (AI) model to obtain a code stream, wherein the code stream comprises first encoded data of the redundant audio data and second encoded data of the to-be-encoded audio frame, and a compression rate of the AI model is higher than a compression rate corresponding to the to-be-encoded audio frame. The method of claim 1, wherein The redundant audio data is extracted from a previous audio frame of the to-be-encoded audio frame and / or a subsequent audio frame of the to-be-encoded audio frame, and the first data length comprises an arbitrary number of frames or an arbitrary number of frames plus an arbitrary number of sampling points. The method according to claim 2, characterized in that The redundant audio data further comprises data of an arbitrary length in the to-be-encoded audio frame. The method according to any one of claims 1 to 3, characterized in that The method further comprises: encoding the first data length of the redundant audio data into the code stream, wherein the first data length comprises a number of complete audio frames and a number of sampling points of an incomplete audio frame. The method according to any one of claims 1 to 4, characterized in that Before the encoding of the redundant audio data by using the AI model to obtain the code stream, the method further comprises: acquiring channel state information, wherein the channel state information indicates a packet loss rate of a transmission channel of the code stream; acquiring the first data length based on the channel state information. The method according to any one of claims 1 to 5, characterized in that The method further comprises: obtaining a first code rate according to an encoding code rate of the to-be-encoded audio frame, wherein the first code rate is an encoding code rate of the AI model; encoding the redundant audio data by using the AI model according to the first code rate to obtain the first encoded data. The method according to claim 6, characterized in that A code rate of the second encoded data is a second code rate, and a sum of the first code rate and the second code rate is the encoding code rate. A decoding method characterized by comprising: The method comprises: acquiring first information indicating a loss of a first audio frame in a code stream; acquiring first reconstructed data of redundant audio data based on the code stream, wherein the redundant audio data comprises the first audio frame, the first reconstructed data is obtained by decoding, by using an artificial intelligence (AI) model, first encoded data of the redundant audio data in the code stream, and the code stream further comprises second encoded data of a second audio frame corresponding to the redundant audio data; obtaining second reconstructed data of the first audio frame based on the first reconstructed data. The method of claim 8, wherein The second audio frame comprises at least one of a previous audio frame of the first audio frame and a subsequent audio frame of the first audio frame. The method of claim 9, wherein The redundant audio data further comprises data of an arbitrary length in the first audio frame. The method according to any one of claims 8 to 10, characterized in that The acquiring of the first reconstructed data of the redundant audio data based on the code stream comprises: determining the second audio frame corresponding to the first audio frame; decoding, by using the AI model, first encoded data of the redundant audio data corresponding to the second audio frame in the code stream to obtain the first reconstructed data. The method of claim 9, wherein The second audio frame comprises the previous audio frame, and the acquiring of the first reconstructed data of the redundant audio data based on the code stream comprises: acquiring first reconstructed data of the redundant audio data corresponding to the previous audio frame, which has been saved after decoding the code stream. The method according to any one of claims 8 to 12, characterized in that The method further comprises: saving the first reconstructed data. The method according to any one of claims 8 to 13, characterized in that The method further includes: decoding, from the bitstream, a target data length of the first audio frame in the redundant audio data, the target data length indicating an amount of data of the first audio frame included in the redundant audio data. The method of claim 14, wherein The second reconstruction data of the first audio frame is obtained based on the first reconstruction data, including: obtaining the second reconstruction data of the first audio frame based on the target data length and the first reconstruction data. The method according to any one of claims 8 to 15, characterized in that The method further includes: obtaining third reconstruction data of a third audio frame adjacent to the first audio frame; performing smoothing processing on the third reconstruction data and the second reconstruction data to obtain audio sequence data. An encoding device characterized by comprising: The encoding device includes: a data obtaining module configured to obtain redundant audio data of a first data length corresponding to an audio frame to be encoded; a data encoding module configured to encode the redundant audio data by using an artificial intelligence (AI) model to obtain a bitstream, wherein the bitstream includes first encoded data of the redundant audio data and second encoded data of the audio frame to be encoded, and a compression rate of the AI model is higher than a compression rate corresponding to the audio frame to be encoded. A decoding device characterized by comprising: The decoding device includes: an information obtaining module configured to obtain first information indicating that a first audio frame in a bitstream is lost; a decoding module configured to obtain first reconstruction data of redundant audio data based on the bitstream, the redundant audio data including the first audio frame, the first reconstruction data being obtained by decoding first encoded data of the redundant audio data in the bitstream by using an artificial intelligence (AI) model, and the bitstream further including second encoded data of a second audio frame corresponding to the redundant audio data; the decoding module is further configured to obtain second reconstruction data of the first audio frame based on the first reconstruction data. A bitstream, characterized in that The bitstream is generated according to the method in any one of claims 1 to 7. An encoder characterized by comprise: a memory and a processor, the memory being coupled to the processor; the memory stores program instructions, when the program instructions are executed by the processor, causing the processor to perform the steps of the method in any one of claims 1 to 7. A decoder, characterized in that comprise: a memory and a processor, the memory being coupled to the processor; the memory stores program instructions, when the program instructions are executed by the processor, causing the processor to perform the steps of the method in any one of claims 8 to 16. A computer-readable storage medium, characterized by The computer readable storage medium stores a computer program, when the computer program runs on a computer or a processor, causing the computer or the processor to perform the method in any one of claims 1 to 7, or causing the computer or the processor to perform the method in any one of claims 8 to 16. A computer program product, characterized in that The computer program product comprises computer instructions which, when executed by a computer or processor, cause the steps of the method according to any one of claims 1 to 7 to be performed, or cause the steps of the method according to any one of claims 8 to 16 to be performed. A computer-readable storage medium, characterized by, The computer readable storage medium stores a bitstream generated according to the method of any one of claims 1 to 7.

Images