Coding method and apparatus during AIGC generation, and decoding method and apparatus during AIGC generation

Abstract

Provided in the present application are a coding method and apparatus during AIGC generation, and a decoding method and apparatus during AIGC generation. The decoding method during AIGC generation in the present application comprises: on the basis of generated feature information outputted by an AIGC generative model, obtaining first feature information of the current frame and motion information, wherein the motion information is used for representing inter-frame correlation; on the basis of the first feature information of the current frame and the motion information, performing motion modeling, so as to obtain second feature information of the current frame, wherein the motion modeling is used for performing compensation on the first feature information of the current frame on the basis of the motion information; and on the basis of the second feature information of the current frame, acquiring reconstructed content of the current frame. The present application can improve the time-domain compression efficiency and the quality of reconstructed content in AIGC.

Description

Encoding and Decoding Methods and Apparatus in AIGC Generation Technical Field

[0001] This application relates to AI Generated Content (AIGC) generation technology, and more particularly to an encoding / decoding method and apparatus for AIGC generation. Background Technology

[0002] The two-stage generative model (encoder + generative model) is the mainstream paradigm in multimodal AIGC generation tasks, widely used for generating various media formats such as images and videos. This generative paradigm first compresses high-dimensional data into a latent space, and then uses the generative model to generate data in the latent space, greatly improving generation efficiency and quality. Furthermore, this method separates the encoding and generation processes, allowing for independent optimization for different modalities of data (e.g., images, videos), making it suitable for various scenarios such as cross-modal generation, video generation, and image generation.

[0003] However, in implementing the above paradigm, the related technologies have limitations in capturing complex motion patterns between frames, resulting in insufficient temporal reference information and thus affecting temporal compression efficiency. Summary of the Invention

[0004] This application provides an encoding / decoding method and apparatus for AIGC generation to improve temporal compression efficiency and the quality of reconstructed content in AIGC.

[0005] In a first aspect, this application provides a decoding method in AIGC generation, comprising: obtaining first feature information and motion information of the current frame based on the generated feature information output by the AIGC generation model, wherein the motion information is used to characterize inter-frame correlation; performing motion modeling based on the first feature information and the motion information of the current frame to obtain second feature information of the current frame, wherein the motion modeling is used to compensate the first feature information of the current frame based on the motion information; and obtaining the reconstructed content of the current frame based on the second feature information of the current frame.

[0006] In this embodiment, the first feature information and motion information of the current frame are obtained by generating feature information based on the output of the AIGC generation model. In the decoding process, the motion information is used to compensate for the first feature information, which can enhance the first feature information and make it closer to the original content of the current frame. On the one hand, it can better realize inter-frame reference in media content encoding and decoding and improve temporal compression efficiency. On the other hand, the reconstructed content is obtained by reconstructing from the enhanced feature information in AIGC, which can improve the quality of the reconstructed content.

[0007] In this application, the current frame can be used as a temporal concept to represent a time slice corresponding to the media content currently being processed by the decoder. It should be understood that the reference frame mentioned below is also used to represent a time slice; in the temporal domain, the reference frame can be earlier than the current frame, and / or, the reference frame can be later than the current frame.

[0008] Motion information is used to characterize inter-frame correlation; that is, it describes the difference between the current frame and the reference frame. For example, motion information can be optical flow information, inter-frame difference information, etc. Furthermore, motion information can include other information, without specific limitations.

[0009] The first feature information can be obtained by processing user-input text or media content through an AIGC generative model (hereinafter referred to as the generative model). Additionally, motion information can also be obtained by processing user-input text or media content through an AIGC generative model (hereinafter referred to as the generative model). The aforementioned media content can include audio, speech, images, 2D / 3D video, etc.

[0010] Optionally, the generator model can output the first feature information and the motion information of the current frame. In this case, the generator model can be improved to enable it to perform temporal and spatial domain processing, changing from outputting only one type of feature information to outputting both the first feature information and the motion information.

[0011] Optionally, the generated feature information output by the generative model can be decomposed to obtain the first feature information and motion information of the current frame. This feature decomposition process can be performed using a spatiotemporal feature extractor (e.g., a Casual 3D Convolution network, 3D CNN, or other neural networks, without specific limitations) to process the generated feature information output by the generative model in both the temporal and spatial domains, thereby obtaining the first feature information and motion information of the current frame. In this case, further feature decomposition of the generated feature information output by the generative model allows for the acquisition of the first feature information and motion information without modifying the structure of the generative model, thus achieving compatibility with the generative model. Furthermore, further feature decomposition of the generated feature information output by the generative model yields motion information that can compensate for the first feature information, making it closer to the original content of the current frame. In AIGC, the reconstructed content is obtained from the enhanced feature information, improving the quality of the reconstructed content.

[0012] Motion modeling is used to compensate for the primary feature information of the current frame based on motion information. For example, motion modeling establishes inter-frame correlations by analyzing motion information (e.g., optical flow, inter-frame differences, displacement fields, block matching information, depth information, etc.) between frames in video or other media. The core task of motion modeling is to compensate for the features of different frames using this motion information. Compensation methods can include transformations (e.g., warping) and fusion (e.g., weighted summation) to enhance inter-frame consistency, improve reconstruction quality, and optimize inter-frame visual performance. Motion modeling not only includes the analysis of inter-frame motion but also involves the process of enhancing the current frame based on this motion information: such as temporal compensation, spatial compensation, and multi-frame fusion. Therefore, motion modeling can enhance the primary feature information, making it closer to the original content of the current frame. This allows for better inter-frame reference in the encoding and decoding of media content such as audio, speech, images, and 2D / 3D video during the decoding process, thereby improving the quality of the decoder's output features and increasing temporal compression efficiency.

[0013] In one possible implementation, the feature information of the reference frame can be obtained first; then temporal compensation can be performed based on the motion information and the feature information of the reference frame to obtain the prediction information of the current frame; then the prediction information and the first feature information of the current frame can be fused to obtain the second feature information of the current frame.

[0014] In this application, the reference frame includes at least one of the following frames:

[0015] (1) The reference frame includes the previous n frames in the video sequence, where n ≥ 1. For example, if the current frame is the 3rd frame in the video sequence, its reference frame could be the 2nd frame in the video sequence; or, if the current frame is the 3rd frame in the video sequence, its reference frames could be the 1st and 2nd frames in the video sequence. It should be understood that n can also be other values, and this application does not specifically limit them.

[0016] (2) The reference frame includes the m frames following the current frame in the video sequence, where m ≥ 1. For example, if the current frame is the first frame in the video sequence, its reference frame could be the second frame in the video sequence; or, if the current frame is the third frame in the video sequence, its reference frames could be the second and third frames in the video sequence. It should be understood that m can also be other values, and this application does not specifically limit them.

[0017] (3) The reference frame includes a specified frame in the image frame group to which the current frame belongs (e.g., the first frame in the image frame group to which the current frame belongs, or the middle frame in the image frame group to which the current frame belongs, or the last frame in the image frame group to which the current frame belongs, etc.). The image frame grouping is obtained by grouping the image frames contained in the video sequence. For example, the video sequence contains 9 frames 1-9, which are divided into 3 image frame groups. The first group includes image frames 1-3, the second group includes image frames 4-6, and the third group includes image frames 7-9. In the first group, image frames 2-3 reference image frame 1, and image frame 1 references itself. In the second group, image frames 5-6 reference image frame 4, and image frame 4 references itself. In the third group, image frames 8-9 reference image frame 7, and image frame 7 references itself. It should be understood that the number of image frames contained in the image frame group can also be other values, and the number of image frame groups obtained from the video sequence can also be other values. Furthermore, the number of image frames contained in each image frame group can be the same or different. This application does not specifically limit the foregoing content.

[0018] (4) When the current frame is the first frame in the video sequence, the reference frame includes the current frame. For example, if the current frame is the first frame in the video sequence, its reference frame is itself. Optionally, the first frame can be copied as the 0th frame, in which case the reference frame of the first frame is the 0th frame.

[0019] (5) When the current frame is the last frame in the video sequence, the reference frame includes the current frame. For example, if the current frame is the 9th frame in the video sequence (the video sequence contains 9 frames 1-9), its reference frame is itself. Optionally, the 9th frame can be copied as the 10th frame, in which case the reference frame of the 9th frame is the 10th frame.

[0020] In this application, when AIGC generates an image, the reference frame for the current frame is the current frame.

[0021] It should be noted that the reference frame of the current frame in this application can be one of the above five types of reference frames. For example, if it is a forward single-frame reference, the reference frame of the current frame can be the first type (1), or if it is a backward single-frame reference, the reference frame of the current frame can be the second type (2). The reference frame of the current frame in this application can also be a combination of two or more of the above five types of reference frames. For example, if it is a bidirectional single-frame reference, the reference frame of the current frame can be a combination of the first type (1) and the second type (2). This application does not make any specific limitation on this.

[0022] In one possible implementation, when the current frame is the first or last frame of a video sequence, the current frame does not require a reference frame; that is, it does not need to reference itself or other frames, and motion modeling is unnecessary. It can be directly reconstructed. In this application, the temporal compensation method varies depending on the different motion information.

[0023] (1) When the motion information includes optical flow information, the temporal compensation performed includes warping operations based on the motion information and the feature information of the reference frame. For example, 2D Warp and DCN Warp introduced above.

[0024] (2) When the motion information includes inter-frame difference information, the temporal compensation performed includes the addition or weighted summation operation based on the motion information and the feature information of the reference frame. For example, the motion information and the feature information of the reference frame are added together.

[0025] It should be noted that since other forms of motion information can be used in this application, the time-domain compensation used accordingly can also include other methods, and no specific limitation is made.

[0026] In this application, the prediction information of the current frame and the first feature information of the current frame can be fused to generate the second feature information of the current frame. The aforementioned fusion method includes inputting the two types of information into a neural network for fusion, such as CNN, Transformer, etc. This application does not specifically limit this.

[0027] Optionally, the feature information of the reference frame includes the original feature information of the reference frame. This original feature information can correspond to the first feature information mentioned above, that is, the feature information that has not undergone motion modeling.

[0028] That is, in the process of obtaining the second feature information of the current frame, the first feature information of the reference frame can be used as a factor affecting the temporal compensation result (the prediction information of the current frame). In this way, the prediction information of the current frame will not be affected by the motion modeling process of the reference frame. Each frame in the video sequence can be reconstructed independently and in parallel, thereby improving the decoding efficiency.

[0029] The first feature information of the reference frame can be obtained by referring to the method for obtaining the first feature information of the current frame described above.

[0030] Optionally, the feature information of the reference frame includes feature information compensated for by adjusting the original feature information of the reference frame. This compensated feature information can correspond to the second feature information mentioned above, that is, the feature information after motion modeling.

[0031] That is, in the process of obtaining the second feature information of the current frame, the second feature information of the reference frame can be used as a factor affecting the temporal compensation result (the prediction information of the current frame). In this way, the prediction information of the current frame depends on the reconstructed features of the reference frame, and the high-quality feature information after temporal compensation can be passed to the next frame, thereby improving the decoding and reconstruction quality.

[0032] The second feature information of the reference frame can be obtained by referring to the method for obtaining the second feature information of the current frame described above.

[0033] In one possible implementation, when the second feature information of the current frame corresponds to a pixel field, the reconstructed content of the current frame is the second feature information of the current frame.

[0034] The neural network used in the above fusion process can realize feature reconstruction and directly output the second feature information corresponding to the pixel domain. At this time, the feature value in the second feature information corresponds to the pixel value of the image. Therefore, the second feature information of the current frame can be directly used as the reconstruction content of the current frame.

[0035] In one possible implementation, when the second feature information of the current frame corresponds to the feature domain, feature reconstruction is performed on the second feature information of the current frame to obtain the reconstructed content of the current frame.

[0036] The neural network used in the above fusion process can focus solely on processing feature information, outputting second feature information corresponding to the feature domain. In this case, feature reconstruction is needed to obtain pixel domain information from the second feature information in order to obtain the reconstructed content of the current frame. For example, the second feature information can be input into a feature reconstruction network (e.g., a 2D convolutional network) to output the reconstructed content of the current frame. It should be noted that other networks can also be used for feature reconstruction, and this application does not specifically limit this.

[0037] As can be seen, whether or not the second feature information needs to be post-processed (i.e., feature reconstruction or feature transformation) in this application depends on the output of motion modeling, and this application does not make specific limitations on this.

[0038] In one possible implementation, motion modeling of multiple frames in the video sequence to which the current frame belongs, based on their respective first feature information and motion information, is processed in parallel. When processing the video sequence, the feature information based on the reference frame can be the original feature information of the reference frame. Therefore, there is no need to wait for the motion modeling of the reference frame to complete. In this case, motion modeling of multiple frames in the video sequence can be processed in parallel, synchronously based on their respective first feature information and motion information. That is, multiple frames in the video sequence synchronously acquire the feature information of their respective reference frames, then perform temporal compensation based on the feature information of their respective reference frames to obtain their respective prediction information, and then reconstruct the video.

[0039] Secondly, this application provides an encoding method in AIGC generation, comprising: acquiring first feature information and motion information of a current frame, wherein the motion information is used to characterize inter-frame correlation; performing motion modeling based on the first feature information and the motion information of the current frame to obtain second feature information of the current frame, wherein the motion modeling is used to compensate the first feature information of the current frame based on the motion information; and acquiring encoded feature information of the current frame based on the second feature information and the motion information of the current frame, wherein the encoded feature information is used to input to an AIGC generation model.

[0040] In this embodiment, by using motion information to compensate for the first feature information during the encoding process of AIGC generation, the first feature information can be enhanced, making it closer to the original content of the current frame. This allows for better inter-frame reference in media content encoding and decoding, thereby improving the quality of the encoder output features and assisting in improving temporal compression efficiency.

[0041] Motion information is used to characterize inter-frame correlation; that is, it describes the difference between the current frame and the reference frame. For example, motion information can be optical flow information, inter-frame difference information (also known as inter-frame difference information), etc. In addition, motion information can also include other information, without specific limitations.

[0042] In one possible implementation, the feature information can be obtained through a feature extraction network (e.g., a 2D convolutional network).

[0043] In one possible implementation, motion information can be obtained through a motion information extraction network (e.g., an optical flow estimation network).

[0044] It should be noted that the aforementioned neural network can also employ other AI technologies, and this application does not impose specific limitations on them.

[0045] In this application, the temporal compensation during the encoding process can be referred to the description of the decoding process, and will not be repeated here.

[0046] In this application, the second feature information and motion information of the current frame can be input into a spatiotemporal feature extractor (e.g., a Casual 3DConvolution network) to obtain the encoded feature information of the current frame. It should be noted that the spatiotemporal feature extractor can also use other networks, and this application does not specifically limit it.

[0047] In one possible implementation, motion modeling of multiple frames in the video sequence to which the current frame belongs, based on their respective first feature information and motion information, is processed in parallel. When processing the video sequence, the feature information based on the reference frame can be the original feature information of the reference frame. Therefore, there is no need to wait for the motion modeling of the reference frame to complete. In this case, motion modeling of multiple frames in the video sequence can be processed in parallel, synchronously based on their respective first feature information and motion information. That is, multiple frames in the video sequence synchronously acquire the feature information of their respective reference frames, then perform temporal compensation based on the feature information of their respective reference frames to obtain their respective prediction information, and then reconstruct the video.

[0048] Thirdly, this application provides a decoding apparatus, comprising: an acquisition module, configured to obtain first feature information and motion information of a current frame based on generated feature information output by an AIGC generation model, wherein the motion information is used to characterize inter-frame correlation; a motion modeling module, configured to perform motion modeling based on the first feature information and the motion information of the current frame to obtain second feature information of the current frame, wherein the motion modeling is used to compensate for the first feature information of the current frame based on the motion information; and a reconstruction module, configured to obtain reconstructed content of the current frame based on the second feature information of the current frame.

[0049] In one possible implementation, the acquisition module is specifically used to perform feature decomposition on the generated feature information to obtain the first feature information of the current frame and the motion information.

[0050] In one possible implementation, the acquisition module is specifically used to output the first feature information and the motion information of the current frame based on the AIGC generation model.

[0051] In one possible implementation, the motion modeling module is specifically used to acquire feature information of a reference frame of the current frame; perform temporal compensation based on the motion information and the feature information of the reference frame to obtain prediction information of the current frame; and fuse the prediction information and the first feature information of the current frame to obtain the second feature information of the current frame.

[0052] In one possible implementation, the reference frame includes at least one of the following frames:

[0053] The reference frame includes the previous n frames of the current frame in the video sequence, where n ≥ 1; or,

[0054] The reference frame includes the m frames following the current frame in the video sequence, where m ≥ 1; or...

[0055] The reference frame includes a designated frame within the image frame group to which the current frame belongs, wherein the image frame grouping is obtained by grouping the image frames contained in the video sequence; or...

[0056] When the current frame is the first frame in the video sequence, the reference frame includes the current frame; or,

[0057] When the current frame is the last frame in the video sequence, the reference frame includes the current frame.

[0058] In one possible implementation, when the AIGC generates an image, the reference frame for the current frame is the current frame.

[0059] In one possible implementation, the feature information of the reference frame includes the original feature information of the reference frame; or, the feature information of the reference frame includes feature information after compensation of the original feature information of the reference frame.

[0060] In one possible implementation, when the motion information includes optical flow information, the temporal compensation includes a warping operation based on the motion information and feature information of the reference frame; or, when the motion information includes inter-frame difference information, the temporal compensation includes a weighted summation operation based on the motion information and feature information of the reference frame.

[0061] In one possible implementation, when the video sequence to which the current frame belongs comprises multiple frames, the motion modeling of the multiple frames based on their respective first feature information and motion information is performed in parallel.

[0062] In one possible implementation, the reconstruction module is specifically used to: when the second feature information of the current frame corresponds to a pixel field, the reconstructed content of the current frame is the second feature information of the current frame; or, when the second feature information of the current frame corresponds to a feature field, to perform feature reconstruction on the second feature information of the current frame to obtain the reconstructed content of the current frame.

[0063] Fourthly, this application provides an encoding apparatus, comprising: an acquisition module, configured to acquire first feature information and motion information of a current frame, wherein the motion information is used to characterize inter-frame correlation; a processing module, configured to perform motion modeling based on the first feature information and the motion information of the current frame to obtain second feature information of the current frame, wherein the motion modeling is used to compensate the first feature information of the current frame based on the motion information; and to acquire encoded feature information of the current frame based on the second feature information and the motion information, wherein the encoded feature information is used to input to an AIGC generation model.

[0064] In one possible implementation, the processing module is specifically configured to acquire feature information of a reference frame of the current frame; perform temporal compensation based on the motion information and the feature information of the reference frame to obtain prediction information of the current frame; and fuse the prediction information and the first feature information of the current frame to obtain the second feature information of the current frame.

[0065] In one possible implementation, the reference frame includes at least one of the following frames:

[0066] The reference frame includes the previous n frames of the current frame in the video sequence, where n ≥ 1; or,

[0067] The reference frame includes the m frames following the current frame in the video sequence, where m ≥ 1; or...

[0068] The reference frame includes the first frame in the image frame group to which the current frame belongs, and the image frame grouping is obtained by grouping the image frames contained in the video sequence; or...

[0069] When the current frame is the first frame in the video sequence, the reference frame includes the current frame; or,

[0070] When the current frame is the last frame in the video sequence, the reference frame includes the current frame.

[0071] In one possible implementation, when the AIGC generates an image, the reference frame for the current frame is the current frame.

[0072] In one possible implementation, when the motion information includes optical flow information, the temporal compensation includes a warping operation based on the motion information and feature information of the reference frame; or, when the motion information includes inter-frame difference information, the temporal compensation includes a weighted summation operation based on the motion information and feature information of the reference frame.

[0073] In one possible implementation, when the video sequence to which the current frame belongs comprises multiple frames, the motion modeling of the multiple frames based on their respective first feature information and motion information is performed in parallel.

[0074] In one possible implementation, the feature information of the current frame is obtained through a feature extraction network.

[0075] In one possible implementation, the motion information is obtained through a motion information extraction network.

[0076] Fifthly, this application provides a decoder, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described in any one of the first aspects above.

[0077] In a sixth aspect, this application provides an encoder comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described in any one of the second aspects above.

[0078] In a seventh aspect, this application provides a computer-readable storage medium including a computer program that, when executed on a computer, causes the computer to perform the method described in any one of the first to second aspects above.

[0079] Eighthly, this application provides a computer program product comprising computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first to second aspects.

[0080] Ninthly, this application provides an AIGC (Artificial Intelligence Generated Content) generation system, comprising: a decoder and an AIGC generation model; wherein the decoder employs the decoding device described in any one of the third aspects above, or the decoder described in the fifth aspect above.

[0081] One possible implementation also includes an encoder.

[0082] In one possible implementation, the encoder employs the encoding device described in any one of the fourth aspects above, or the encoder described in the sixth aspect above. Attached Figure Description

[0083] Figure 1a is a schematic diagram of an application scenario of this application;

[0084] Figure 1b is a schematic diagram of another application scenario of this application;

[0085] Figure 2 shows a schematic diagram of the terminal device 200;

[0086] Figure 3 is a software structure block diagram of the terminal device 200 of this application;

[0087] Figure 4 is a schematic diagram of the architecture of the network model acquisition system 400 according to an embodiment of this application;

[0088] Figure 5 is a schematic diagram of the CNN structure according to an embodiment of this application;

[0089] Figure 6 is a flowchart of the decoding method 600 in the AIGC generation provided in this application;

[0090] Figure 7 is a flowchart of the encoding method 700 in AIGC generation provided in this application;

[0091] Figure 8 is a framework diagram of the encoder in the AIGC generation of this application;

[0092] Figure 9 is a framework diagram of the decoder in the AIGC generation of this application;

[0093] Figure 10a is a schematic diagram of the inter-frame reference scheme of this application;

[0094] Figure 10b is a flowchart of the encoding method in AIGC generation of this application;

[0095] Figure 10c is a flowchart of the decoding method in the AIGC generation of this application;

[0096] Figure 11a is a schematic diagram of the inter-frame reference scheme of this application;

[0097] Figure 11b is a flowchart of the encoding method in AIGC generation of this application;

[0098] Figure 11c is a flowchart of the decoding method in the AIGC generation of this application;

[0099] Figure 12 is a schematic diagram of the inter-frame reference scheme of this application;

[0100] Figure 13a is a schematic diagram of the inter-frame reference scheme of this application;

[0101] Figure 13b is a flowchart of the encoding method in AIGC generation of this application;

[0102] Figure 13c is a flowchart of the decoding method in the AIGC generation of this application;

[0103] Figure 14 is a flowchart of the encoding method in AIGC generation of this application;

[0104] Figure 15 is a structural schematic diagram of the decoding device 1500 of this application;

[0105] Figure 16 is a structural schematic diagram of the encoding device 1600 of this application;

[0106] Figure 17 is a schematic block diagram of an apparatus 1700 according to an embodiment of this application. Detailed Implementation

[0107] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0108] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0109] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0110] AI-generated content (AIGC) refers to content created using artificial intelligence (AI) technology, including various forms such as text, images, audio, and video. With the emergence of Generative Adversarial Networks (GANs) and denoising diffusion models, the underlying technology of AIGC has continuously broken through, enabling it to generate corresponding text and realistic images or videos based on prompts. AI-generated content has a wide range of applications, from providing richer user experiences to improving content creation efficiency and creating new art forms; its possibilities are endless, and it has great development potential in generating personalized, customized, and real-time content. AIGC generation tasks can be applied to large-scale models in the media field, such as edge AI, video albums, and Huawei Cloud. For example, Figure 1a is a schematic diagram of an application scenario of this application. As shown in Figure 1a, this application scenario includes an AIGC generation model (hereinafter referred to as the generation model) and a decoder. The user-input content (e.g., text or media content) is input into the generation model, processed by the generation model, and outputs the feature domain features corresponding to the input. The decoder then reconstructs the features and outputs the final media content corresponding to the input. Figure 1b illustrates another application scenario of this application. As shown in Figure 1b, this application scenario includes an encoder, an AIGC generation model (hereinafter referred to as the generation model), and a decoder. The user inputs text using a terminal device, and prompts are fed to the encoder (e.g., a text encoder) to generate word embeddings. The generation model then generates feature domain features, which are reconstructed by the decoder to generate the final media content (corresponding to the user's input text). Alternatively, the user inputs media content using a terminal device, and the encoder (e.g., a media encoder) generates pixel domain features, which are then fed to the generation model to generate feature domain features. The decoder reconstructs the features to generate the final media content (corresponding to the user's input media content). The encoder and decoder in this application scenario can correspond to the same standard or different standards, without specific limitations. It should be noted that the aforementioned media content can include audio, speech, images, 2D / 3D video, etc.

[0111] In one possible implementation, to obtain the decoder described above, a training process can be performed in advance; that is, a network containing an encoder and a decoder is constructed and trained using machine learning methods to obtain the decoder. The encoder and decoder during the training process can correspond to the same standard.

[0112] Therefore, before introducing the technical solution of this application, the relevant technologies involved in this application will be explained first.

[0113] 1. Transformer

[0114] The Transformer is a deep learning model architecture originally designed for sequence-to-sequence tasks, such as machine translation. It uses a self-attention mechanism to process input sequences and has achieved great success in natural language processing. Most large language models, such as BERT, GPT, and T5, are based on the Transformer architecture.

[0115] 2. Diffusion Model

[0116] Diffusion models are a type of generative model that generates complex data, such as images or videos, by progressively adding noise. Backdiffusion, on the other hand, is used to generate high-quality samples from random noise and is widely used in image generation tasks.

[0117] The diffusion model consists of two processes: a forward process and a reverse process, where the forward process is also known as the diffusion process. Both the forward and reverse processes are parameterized Markov chains (i.e., the current state depends only on the previous state), and the reverse process can be used to generate data.

[0118] 3. Latent Diffusion Models (LDM)

[0119] Latent diffusion models are efficient generative models that generate data by applying a diffusion process to a latent feature space. While traditional diffusion models perform well in image generation, their computational cost of directly performing diffusion and generation in a high-dimensional space is very high.

[0120] To address this issue, the latent diffusion model first maps the data (pixel space) to a latent feature space, and then performs the diffusion process within that space. This improvement significantly reduces computational complexity while maintaining the quality of the generated data. The key to LDM lies in using a pre-trained autoencoder to compress high-dimensional data into the latent feature space, then generating new latent variables in that space through the diffusion process, and finally using a decoder to restore these latent variables into a high-resolution image or video.

[0121] Two-stage generative models (encoder + generative model) are gradually becoming the mainstream paradigm in multimodal generation tasks, widely applied to various forms of generation tasks such as images and videos. This generative paradigm first compresses high-dimensional data into a latent space for encoding, and then uses a generative model to generate and reconstruct the data in the latent space, greatly improving generation efficiency and quality. Another advantage is that this method can independently optimize for different modalities of data (e.g., images, videos) by separating the encoding and generation processes, making it suitable for various scenarios such as cross-modal generation, video generation, and image generation. Many classic methods have adopted this two-stage generative strategy, such as VQ-GAN, DIT, and Video-LDM.

[0122] 4. Motion Modeling

[0123] Motion modeling establishes inter-frame correlations by analyzing motion information (e.g., optical flow, inter-frame differences, displacement fields, block matching information, depth information, etc.) between frames in video or other media. The core task of motion modeling is to compensate for the features of different frames using this motion information. Compensation methods can include transformations (e.g., warping) and fusion (e.g., weighted summation) to enhance inter-frame consistency, improve reconstruction quality, and optimize inter-frame visual representation. Motion modeling not only includes the analysis of inter-frame motion but also involves enhancing the current frame based on this motion information: such as temporal compensation, spatial compensation, and multi-frame fusion.

[0124] 5. Temporal Compensation

[0125] Temporal compensation is an important task in motion modeling. It primarily utilizes inter-frame motion information (e.g., optical flow, inter-frame differences, displacement fields, block matching information, depth information, etc.) to predict and compensate for the current frame (e.g., compensation based on optical flow and warp operations, interpolation, etc.). Temporal compensation aims to reduce redundant information between video frames, improving video compression efficiency or reconstruction quality. During temporal compensation, a reference frame is predicted as the current frame using motion information, thereby reducing redundancy in data storage and transmission or improving reconstruction quality.

[0126] 6. Optical Flow

[0127] Optical flow is a technique for tracking pixel motion between video frames to represent the motion patterns of objects in a video. It is used to estimate the motion vectors of pixels between adjacent video frames, representing the direction and magnitude of each pixel's movement from one frame to another.

[0128] 7. Warp

[0129] Warp is a technique commonly used in image or video processing. It is mainly used to spatially rearrange or offset pixels in each frame according to a certain transformation rule. Its goal is to perform geometric transformations on the image to adapt it to a desired shape or align it with other frames.

[0130] Optical flow-based 2D warp is a technique that uses optical flow information to perform temporal compensation of pixels between frames. It offsets the positions of pixels in a reference frame based on optical flow information, using this offset as prediction information for the current frame. This compensates for motion differences between adjacent frames, thereby aligning the images at two different moments. Specifically, optical flow calculates a displacement vector for each pixel from the current frame to the reference frame, and then applies these vectors to pixels in the reference frame to generate an image aligned with the current frame. This process is commonly used in video processing or generation tasks to solve misalignment problems caused by motion between frames. Optical flow-based 2D warp performs well when handling simple, regular motions such as translation and rotation.

[0131] Warp based on Deformable Convolutional Networks (DCNs) provides a flexible warp technique suitable for complex non-rigid motion. Traditional convolutions have fixed kernel sampling positions when processing spatial features, while DCNs, by introducing deformable convolutions, allow the kernel sampling positions to be dynamically adjusted based on optical flow or motion information. In optical flow-based motion modeling, DCN Warp combines the results of optical flow estimation and adjusts the kernel sampling positions to ensure that the reference frame can be effectively aligned to the current frame even in more complex motion scenes. This makes DCN Warp more flexible and robust when handling complex scenes, such as non-rigid deformations, occlusions, or large-scale object motion. DCN Warp can be seen as an extension of 2D Warp, making inter-frame alignment more accurate by dynamically adjusting the sampling positions, especially providing better results when dealing with scenes with local deformations and complex motions.

[0132] 8. Variational Autoencoder (VAE)

[0133] Visual Advanced Encoding (VAE) is a generative model that encodes input data into a latent representation and then reconstructs the data from the latent space using a decoder. A key feature of VAEs is the introduction of a probability distribution, which allows for the generation of diverse samples.

[0134] 9. Causal 3D Variational Autoencoder (Casual 3D VAE)

[0135] Casual 3D VAE can effectively handle spatiotemporal dependencies in video. Its main features include:

[0136] Causal Modeling: In video data processing, the core module of Casual 3D VAE is Casual 3D Convolution. Casual 3D Convolution is a special type of 3D convolution operation widely used in video processing tasks. By performing convolution operations in both temporal and spatial dimensions, it captures the spatiotemporal relationships within a video sequence. Unlike ordinary 3D convolution, Casual 3D Convolution only considers the current frame and previous frames, ignoring information from future frames. This causal structure ensures temporal consistency during model inference, making it particularly suitable for tasks such as online video streaming or real-time video analysis. Causal models need to ensure that the generation process follows a natural temporal order, avoiding reliance on future information that has not yet been generated during frame modeling, thus ensuring the rationality and continuity of the video reconstruction process. Furthermore, causal modeling guarantees the independence of the first frame, making it suitable for image modeling.

[0137] 3D Convolutional Neural Network (CNN): 3D CNNs can process both the spatial and temporal dimensions of video. 3D convolution operations can simultaneously capture the temporal dependencies between video frames and the spatial structure within frames, ensuring that the spatiotemporal information of the video is preserved during compression and reconstruction.

[0138] The Causal 3D VAE structure can extract spatiotemporal features from variable-length video sequences (becoming an image codec when the number of frames is 1). Each frame depends only on the earlier frame in the time sequence during encoding or decoding, solving the spatiotemporal dependency problem in video compression and generation. It realizes variable-length video compression and image-video joint modeling, effectively improving the compression and reconstruction performance of video.

[0139] 10. Autoregressive Model

[0140] Autoregressive models are a type of sequence model that predicts the input of the next time step based on the output of the previous time step. Autoregressive models are widely used in sequence generation tasks, including text and image generation.

[0141] 11. Upsampling / Downsampling

[0142] Upsampling increases the resolution of data through interpolation or convolution operations, while downsampling decreases the resolution of data. They are widely used in image / video generation and compression tasks.

[0143] This application can be implemented by a terminal device, or as a system service of the terminal device, or as an application (APP) installed on the terminal device, or in other forms, without specific limitation. Figure 2 shows a schematic diagram of the structure of the terminal device 200. It should be understood that the terminal device 200 shown in Figure 2 is only an example, and the terminal device 200 may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in Figure 2 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0144] Terminal device 200 may include: processor 210, external memory interface 220, internal memory 221, universal serial bus (USB) interface 230, charging management module 240, power management module 241, battery 242, antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, audio module 270, speaker 270A, receiver 270B, microphone 270C, headphone jack 270D, sensor module 280, button 290, motor 291, indicator 292, camera 293, display screen 294, and subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, a barometric pressure sensor 280C, a magnetic sensor 280D, an accelerometer sensor 280E, a distance sensor 280F, a proximity sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0145] Processor 210 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0146] The controller can serve as the central nervous system and command center of the terminal device 200. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0147] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0148] In some embodiments, the processor 210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0149] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 210 may include multiple I2C buses. The processor 210 can couple to the touch sensor 280K, charger, flash, camera 293, etc., through different I2C bus interfaces. For example, the processor 210 can couple to the touch sensor 280K through the I2C interface, enabling the processor 210 and the touch sensor 280K to communicate through the I2C bus interface, thereby realizing the touch function of the terminal device 200.

[0150] The I2S interface can be used for audio communication. In some embodiments, the processor 210 may include multiple I2S buses. The processor 210 can be coupled to the audio module 270 via the I2S bus to enable communication between the processor 210 and the audio module 270. In some embodiments, the audio module 270 can transmit audio signals to the wireless communication module 260 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0151] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 270 and the wireless communication module 260 can be coupled via the PCM bus interface. In some embodiments, the audio module 270 can also transmit audio signals to the wireless communication module 260 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0152] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 210 and the wireless communication module 260. For example, the processor 210 communicates with the Bluetooth module in the wireless communication module 260 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 270 can transmit audio signals to the wireless communication module 260 via the UART interface to enable music playback through Bluetooth headphones.

[0153] The MIPI interface can be used to connect the processor 210 to peripheral devices such as the display screen 294 and the camera 293. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 210 and the camera 293 communicate via the CSI interface to enable the shooting function of the terminal device 200. The processor 210 and the display screen 294 communicate via the DSI interface to enable the display function of the terminal device 200.

[0154] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 210 to a camera 293, a display screen 294, a wireless communication module 260, an audio module 270, a sensor module 280, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0155] USB port 230 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, or USB Type-C port. USB port 230 can be used to connect a charger to charge terminal device 200, and can also be used for data transfer between terminal device 200 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other terminal devices, such as AR devices.

[0156] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the terminal device 200. In other embodiments of this application, the terminal device 200 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0157] The charging management module 240 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 receives charging input from the wired charger via a USB interface 230. In some wireless charging embodiments, the charging management module 240 receives wireless charging input via the wireless charging coil of the terminal device 200. While charging the battery 242, the charging management module 240 can also supply power to the terminal device via the power management module 241.

[0158] The power management module 241 connects the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, providing power to the processor 210, internal memory 221, external memory, display screen 294, camera 293, and wireless communication module 260. The power management module 241 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 241 may also be located within the processor 210. In other embodiments, the power management module 241 and the charging management module 240 may be housed in the same device.

[0159] The wireless communication function of the terminal device 200 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor, etc.

[0160] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 200 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0161] The mobile communication module 250 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the terminal device 200. The mobile communication module 250 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 250 may be housed in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 250 and at least some modules of the processor 210 may be housed in the same device.

[0162] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 270A, receiver 270B, etc.) or displays images or videos through the display screen 294. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 210 and may be housed in the same device as the mobile communication module 250 or other functional modules.

[0163] The wireless communication module 260 can provide solutions for wireless communication applications on the terminal device 200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 210. The wireless communication module 260 can also receive signals to be transmitted from processor 210, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0164] In some embodiments, antenna 1 of terminal device 200 is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, enabling terminal device 200 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0165] The terminal device 200 implements display functions through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0166] Display screen 294 is used to display images, videos, etc. Display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 200 may include one or N displays 294, where N is a positive integer greater than 1.

[0167] The terminal device 200 can perform shooting functions through an ISP, camera 293, video codec, GPU, display screen 294, and application processor.

[0168] The ISP (Image Signal Processor) is used to process data fed back from the camera 293. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 293.

[0169] Camera 293 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the terminal device 200 may include one or N cameras 293, where N is a positive integer greater than 1.

[0170] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when the terminal device 200 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0171] Video codecs are used to compress or decompress digital video. Terminal device 200 may support one or more video codecs. Thus, terminal device 200 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0172] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in terminal devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0173] The external storage interface 220 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device 200. The external storage card communicates with the processor 210 through the external storage interface 220 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0174] Internal memory 221 can be used to store computer executable program code, which includes instructions. Processor 210 executes various functional applications and data processing of terminal device 200 by running the instructions stored in internal memory 221. Internal memory 221 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of terminal device 200 (such as audio data, phonebook, etc.). Furthermore, internal memory 221 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0175] Terminal device 200 can implement audio functions, such as music playback and recording, through audio module 270, speaker 270A, receiver 270B, microphone 270C, headphone jack 270D, and application processor.

[0176] The audio module 270 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 270 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 270 may be located in the processor 210, or some functional modules of the audio module 270 may be located in the processor 210.

[0177] The speaker 270A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal device 200 can listen to music or make hands-free calls through the speaker 270A.

[0178] The receiver 270B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal device 200 receives a telephone call or voice message, the receiver 270B can be brought close to the listener's ear to receive the voice message.

[0179] Microphone 270C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 270C, inputting the sound signal into microphone 270C. Terminal device 200 may be equipped with at least one microphone 270C. In some embodiments, terminal device 200 may be equipped with two microphones 270C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, terminal device 200 may be equipped with three, four, or more microphones 270C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0180] The headphone jack 270D is used to connect wired headphones. The headphone jack 270D can be a USB 230 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0181] Pressure sensor 280A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 280A can be disposed on display screen 294. There are many types of pressure sensors 280A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 280A, the capacitance between the electrodes changes. Terminal device 200 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 294, terminal device 200 detects the intensity of the touch operation based on pressure sensor 280A. Terminal device 200 can also calculate the touch position based on the detection signal from pressure sensor 280A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example: when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0182] The gyroscope sensor 280B can be used to determine the motion attitude of the terminal device 200. In some embodiments, the gyroscope sensor 280B can determine the angular velocity of the terminal device 200 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 280B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 280B detects the angle of the terminal device 200's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the terminal device 200 through reverse movement, thus achieving image stabilization. The gyroscope sensor 280B can also be used in navigation and motion-sensing game scenarios.

[0183] The barometric pressure sensor 280C is used to measure air pressure. In some embodiments, the terminal device 200 calculates altitude using the air pressure value measured by the barometric pressure sensor 280C to assist in positioning and navigation.

[0184] The magnetic sensor 280D includes a Hall sensor. The terminal device 200 can use the magnetic sensor 280D to detect the opening and closing of the flip cover. In some embodiments, when the terminal device 200 is a flip phone, the terminal device 200 can detect the opening and closing of the flip cover using the magnetic sensor 280D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0185] The accelerometer 280E can detect the magnitude of acceleration of the terminal device 200 in various directions (generally three axes). When the terminal device 200 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the attitude of the terminal device and applied to applications such as landscape / portrait switching and pedometers.

[0186] A distance sensor 280F is used to measure distance. The terminal device 200 can measure distance via infrared or laser. In some embodiments, during a shooting scene, the terminal device 200 can utilize the distance sensor 280F to measure distance for rapid focusing.

[0187] The proximity sensor 280G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The terminal device 200 emits infrared light outward through the LED. The terminal device 200 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that an object is near the terminal device 200. When insufficient reflected light is detected, the terminal device 200 can determine that no object is near the terminal device 200. The terminal device 200 may use the proximity sensor 280G to detect when a user holds the terminal device 200 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 280G can also be used in holster mode and pocket mode for automatic unlocking and screen locking.

[0188] The ambient light sensor 280L is used to sense ambient light intensity. The terminal device 200 can adaptively adjust the brightness of the display screen 294 according to the sensed ambient light intensity. The ambient light sensor 280L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 280L can also work with the proximity sensor 280G to detect whether the terminal device 200 is in a pocket to prevent accidental touches.

[0189] The fingerprint sensor 280H is used to collect fingerprints. The terminal device 200 can use the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0190] Temperature sensor 280J is used to detect temperature. In some embodiments, terminal device 200 uses the temperature detected by temperature sensor 280J to execute a temperature processing strategy. For example, when the temperature reported by temperature sensor 280J exceeds a threshold, terminal device 200 reduces the performance of the processor located near temperature sensor 280J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, terminal device 200 heats battery 242 to prevent abnormal shutdown of terminal device 200 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, terminal device 200 boosts the output voltage of battery 242 to prevent abnormal shutdown due to low temperature.

[0191] Touch sensor 280K, also known as a "touch panel," can be located on display screen 294. The touch sensor 280K and display screen 294 together form a touchscreen, also known as a "touchscreen." Touch sensor 280K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 294. In other embodiments, touch sensor 280K may also be located on the surface of terminal device 200, in a different position than display screen 294.

[0192] The bone conduction sensor 280M can acquire vibration signals. In some embodiments, the bone conduction sensor 280M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 280M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 280M can also be incorporated into headphones to form bone conduction headphones. The audio module 270 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 280M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 280M to realize heart rate detection functionality.

[0193] Buttons 290 include a power button, volume buttons, etc. Buttons 290 can be mechanical buttons or touch-sensitive buttons. Terminal device 200 can receive button input and generate key signal inputs related to user settings and function control of terminal device 200.

[0194] Motor 291 can generate vibration alerts. Motor 291 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 291 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 294. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0195] Indicator 292 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0196] The SIM card interface 295 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 295 to make contact with and separate from the terminal device 200. The terminal device 200 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, and other SIM cards. Multiple cards can be inserted into the same SIM card interface 295 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 295 is also compatible with different types of SIM cards. The SIM card interface 295 is also compatible with external memory cards. The terminal device 200 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the terminal device 200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 200 and cannot be separated from the terminal device 200.

[0197] The software system of terminal device 200 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the system structure of terminal device 200.

[0198] Figure 3 is a software structure block diagram of the terminal device 200 of this application.

[0199] The layered architecture of the terminal device 200 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0200] The application layer can include a series of application packages.

[0201] As shown in Figure 3, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0202] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0203] As shown in Figure 3, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0204] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0205] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0206] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0207] The phone manager is used to provide communication functions for terminal device 200. For example, it manages call status (including connection and disconnection).

[0208] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0209] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0210] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0211] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0212] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0213] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0214] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0215] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0216] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0217] A 2D graphics engine is a graphics engine for 2D drawing.

[0218] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0219] It is understood that the components included in the system framework layer, system library, and runtime layer shown in Figure 3 do not constitute a specific limitation on the terminal device 200. In other embodiments of this application, the terminal device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or have different component arrangements.

[0220] This application also relates to the application of artificial intelligence (AI). For ease of understanding, some terms or nouns used in AI technology are explained below, and these terms or nouns are also part of the invention.

[0221] (1) Neural Network

[0222] Neural Networks (NNs) are machine learning models. A neural network can be composed of neural units, which are computational units that take xs and an intercept of 1 as input. The output of such a computational unit can be:

[0223] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0224] (2) Deep Neural Networks

[0225] Deep neural networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0226] (3) Convolutional Neural Network

[0227] A convolutional neural network (CNN) is a deep neural network with convolutional structures. It is a deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, where each neuron responds to an input image. A CNN contains a feature extractor consisting of convolutional layers and pooling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as performing convolution with a trainable filter and an input image or a convolutional feature map.

[0228] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution processing on the input signal. A convolutional layer can contain multiple convolution operators, also called kernels. In image processing, these operators act as filters, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during the convolution operation, the weight matrix extends to the entire depth of the input image. Therefore, convolving with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same size (rows × columns) are used instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. This dimension can be understood as being determined by the "multiple" factors mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix can be used to extract edge information, another to extract specific colors, and yet another to blur unwanted noise. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these weight matrices also have the same size. These extracted feature maps are then merged to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by these trained weight values ​​can be used to extract information from the input image, enabling the convolutional neural network to make correct predictions. When a convolutional neural network has multiple convolutional layers, the initial convolutional layers often extract more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by later convolutional layers become increasingly complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem being solved.

[0229] Because it's often necessary to reduce the number of training parameters, pooling layers are frequently introduced periodically after convolutional layers. This can be a single convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range as the result of average pooling. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0230] After processing by convolutional / pooling layers, a convolutional neural network (CNN) is still insufficient to output the required information. As mentioned earlier, convolutional / pooling layers only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the CNN needs to utilize neural network layers to generate one or a set of desired classes in the output. Therefore, the neural network can include multiple hidden layers, the parameters of which can be pre-trained based on relevant data for a specific task type, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0231] Optionally, after the multiple hidden layers in the neural network, there is also an output layer of the entire convolutional neural network. This output layer has a loss function similar to the classification cross-entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the backpropagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0232] (4) Recurrent Neural Network

[0233] Recurrent neural networks (RNNs) are used to process sequential data. In traditional neural network models, the layers from the input layer to the hidden layer and then to the output layer are fully connected, but the nodes within each layer are unconnected. While this type of neural network has solved many difficult problems, it remains inadequate for many others. For example, predicting the next word in a sentence generally requires using the preceding words because words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is related to the outputs of previous sequences. Specifically, the network memorizes previous information and applies it to the calculation of the current output; that is, nodes within the same hidden layer are no longer unconnected but connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous time step. Theoretically, RNNs can process sequential data of any length. Training an RNN is similar to training a traditional CNN or DNN. This algorithm also uses the backpropagation algorithm, but with one key difference: when an RNN is expanded, its parameters, such as W, are shared; however, this is not the case with traditional neural networks as illustrated above. Furthermore, in gradient descent, the output at each step depends not only on the network at the current step but also on the states of the network in previous steps. This learning algorithm is called Backpropagation Through Time (BPTT).

[0234] Since we already have convolutional neural networks (CNNs), why do we need recurrent neural networks (RNNs)? The reason is simple. CNNs rely on the fundamental assumption that elements are independent of each other, and that input and output are also independent—like a cat and a dog. However, in the real world, many elements are interconnected. For example, stock prices fluctuate over time. Or, imagine someone saying, "I love traveling, and my favorite place is Yunnan. I definitely want to go there someday." Humans know the answer to this question is "Yunnan." Humans can infer from context, but how can machines do the same? This is where RNNs come in. RNNs aim to give machines the ability to remember, just like humans. Therefore, the output of an RNN depends on both the current input information and historical memory information.

[0235] (5) Loss Function

[0236] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0237] (6) Backpropagation algorithm

[0238] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0239] (7) Generative Adversarial Networks

[0240] Generative adversarial networks (GANs) are a type of deep learning model. This model comprises at least two modules: a generative model and a discriminative model. These two modules learn from each other through a game-like interaction, resulting in better outputs. Both the generative and discriminative models can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GANs is as follows: Taking an image-generating GAN as an example, suppose there are two networks, G (Generator) and D (Discriminator). G is a network that generates images by receiving random noise z and using this noise, denoted as G(z). D is a discriminative network used to determine whether an image is "real." Its input parameter is x, representing an image, and its output D(x) represents the probability that x is a real image. A value of 1 indicates that the image is 100% real, while a value of 0 indicates that the image is impossible to be real. During the training of this generative adversarial network (GAN), the goal of the generative network G is to generate realistic images to deceive the discriminator network D, while the goal of the discriminator network D is to distinguish the images generated by G from real images as much as possible. Thus, G and D constitute a dynamic "game," which is the "adversarial" aspect of the GAN. Ideally, the game will result in G generating images G(z) that are sufficiently realistic, while D struggles to determine whether the images generated by G are real or not, i.e., D(G(z)) = 0.5. This yields a superior generative model G that can be used to generate images.

[0241] Regardless of which AI technology is used, the network model in this embodiment can be pre-trained. For example, Figure 4 is a schematic diagram of the architecture of the network model acquisition system 400 in this embodiment. As shown in Figure 4, the data acquisition device 460 collects data and stores it in the database 430, and the training device 420 generates a target model / rule 401 based on the data maintained in the database 430. The following will describe in more detail how the training device 420 obtains the target model / rule 401 based on the data. The target model / rule 401 can perform related functions and output the required data or graphics.

[0242] The function of each layer in a deep neural network can be expressed mathematically. To describe it: From a physical perspective, the work of each layer in a deep neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are... The operation 4 is completed using +b, and the operation 5 is implemented using a(). The term "space" is used here because the objects being classified are not individual things, but a class of things; space refers to the set of all individuals within this class of things. Here, W is the weight vector, where each value represents the weight of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space, as described above; that is, the weights W of each layer control how the space is transformed. The purpose of training a deep neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (a weight matrix formed by the vectors W from many layers). Therefore, the training process of a neural network is essentially learning how to control spatial transformation, more specifically, learning the weight matrix.

[0243] Because the goal is for the output of a deep neural network to be as close as possible to the actual predicted value, we can compare the current network's prediction with the desired target value and update the weight vector of each layer based on the difference. (Of course, there's usually an initialization process before the first update, where parameters are pre-configured for each layer in the deep neural network). For example, if the network's prediction is too high, the weight vector is adjusted to predict a lower value, and this adjustment continues until the neural network can predict the actual target value. Therefore, it's necessary to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss as much as possible.

[0244] The target model / rule 401 obtained from training device 420 can be applied to different systems or devices.

[0245] The execution device 410 is equipped with an I / O interface 412 for data interaction with external devices. The "user" can input data to the I / O interface 412 through the terminal device 440.

[0246] The execution device 410 can call data, code, etc. in the data storage system 450, and can also store data, instructions, etc. in the data storage system 450.

[0247] The calculation module 411 uses the target model / rule 401 to process the input data in order to realize the function of the network model in this application embodiment.

[0248] The associated function module 413 and associated function module 414 can respectively implement related functions in the training process, such as preprocessing and filtering.

[0249] Finally, the I / O interface 412 returns the processing result to the terminal device 440 for the user.

[0250] At a deeper level, the training device 420 can generate corresponding target models / rules 401 based on different data for different objectives, in order to provide users with better results.

[0251] In the scenario shown in Figure 4, the user can manually specify the data to be input into the execution device 410, for example, by operating through the interface provided by the I / O interface 412. Alternatively, the terminal device 440 can automatically input data into the I / O interface 412 and obtain the results. If the terminal device 440 requires user authorization to automatically input data, the user can set the corresponding permissions in the terminal device 440. The user can view the results output by the execution device 410 on the terminal device 440, which can be presented in various ways such as display, sound, or animation. The terminal device 440 can also act as a data acquisition terminal, storing the acquired data into the database 430.

[0252] It is worth noting that Figure 4 is merely a schematic diagram of a network model acquisition system provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, in Figure 4, the data storage system 450 is an external memory relative to the execution device 410. In other cases, the data storage system 450 can also be placed in the execution device 410. As another example, in Figure 4, the terminal device 440 and the execution device 410 are two devices. In other cases, the terminal device 440 and the execution device 410 can also be integrated into one device.

[0253] A convolutional neural network (CNN) is a deep neural network with convolutional structures. It is a deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network where each neuron responds to overlapping regions in the input image.

[0254] For example, FIG5 is a schematic diagram of the structure of a CNN according to an embodiment of the present application. As shown in FIG5, the CNN 500 may include an input layer 410, a convolutional layer / pooling layer 520, wherein the pooling layer is optional, and a neural network layer 530.

[0255] 520 convolutional / pooling layers:

[0256] Convolutional layers:

[0257] As shown in Figure 5, the convolutional / pooling layer 520 may include layers 521-526 as in Examples 521. In one implementation, layer 521 is a convolutional layer, layer 522 is a pooling layer, layer 523 is a convolutional layer, layer 524 is a pooling layer, layer 525 is a convolutional layer, and layer 526 is a pooling layer. In another implementation, layers 521 and 522 are convolutional layers, layer 523 is a pooling layer, layers 524 and 525 are convolutional layers, and layer 526 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0258] Taking convolutional layer 521 as an example, it can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same dimension are applied instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors of the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same dimension, and the feature maps extracted by these multiple weight matrices also have the same dimension. The extracted feature maps with the same dimension are then merged to form the output of the convolution operation.

[0259] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the trained weight values ​​can extract information from the input image, thereby helping CNN 500 to make correct predictions.

[0260] When a CNN 500 has multiple convolutional layers, the initial convolutional layers (e.g., 521) tend to extract more general features, which can also be called low-level features. As the depth of the CNN 500 increases, the features extracted by later convolutional layers (e.g., 526) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0261] Pooling layer:

[0262] Because it's often necessary to reduce the number of training parameters, pooling layers are frequently introduced periodically after convolutional layers, as illustrated in layers 521-526 of Figure 5 (example 520). This can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0263] Neural network layer 530:

[0264] After processing by the convolutional / pooling layers 520, CNN 500 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 520 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), CNN 500 needs to utilize neural network layers 530 to generate one or a set of required class numbers of output. Therefore, neural network layers 530 can include multiple hidden layers (541, 542 to 53n as shown in Figure 5) and an output layer 540. The parameters contained in these hidden layers can be pre-trained based on relevant data for a specific task type, such as image recognition, image classification, or image super-resolution reconstruction.

[0265] After the multiple hidden layers in neural network layer 530, which is the last layer of the entire CNN 500, is the output layer 540. The output layer 540 has a loss function similar to the classification cross-entropy, which is used to calculate the prediction error. Once the forward propagation of the entire CNN 500 is completed (as shown in Figure 5, the propagation from 410 to 540 is the forward propagation), the back propagation (as shown in Figure 5, the propagation from 540 to 410 is the back propagation) will begin to update the weight values ​​and biases of the aforementioned layers, in order to reduce the loss of CNN 500 and the error between the result output by CNN 500 through the output layer and the ideal result.

[0266] It should be noted that the CNN 500 shown in Figure 5 is only an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models. For example, as shown in Figure 5, multiple convolutional / pooling layers can be parallelized, and the extracted features can be input into the full neural network layer 530 for processing.

[0267] Based on the above, this application provides an encoding and decoding method in AIGC generation to better realize inter-frame reference in the encoding and decoding of media content such as audio, speech, images, and 2D / 3D video.

[0268] Figure 6 is a flowchart of process 600 in the decoding method for AIGC generation provided in this application. Process 600 can be executed by the terminal device described above. Process 600 is described as a series of steps or operations. It should be understood that process 600 can be executed in various orders and / or occur simultaneously, and is not limited to the execution order shown in Figure 6. Process 600 may include:

[0269] Step 601: Obtain the first feature information and motion information of the current frame based on the generated feature information output by the AIGC generation model.

[0270] In this application, the current frame can be used as a temporal concept to represent a time slice corresponding to the media content currently being processed by the decoder. It should be understood that the reference frame mentioned below is also used to represent a time slice; in the temporal domain, the reference frame can be earlier than the current frame, and / or, the reference frame can be later than the current frame.

[0271] Motion information is used to characterize inter-frame correlation; that is, it describes the difference between the current frame and the reference frame. For example, motion information can be optical flow information, inter-frame difference information, etc. Furthermore, motion information can include other information, without specific limitations.

[0272] Referring to Figures 1a and 1b, the first feature information can be obtained by processing user-input text or media content through an AIGC generation model (hereinafter referred to as the generation model). Furthermore, motion information can also be obtained by processing user-input text or media content through an AIGC generation model (hereinafter referred to as the generation model). The aforementioned media content can include audio, speech, images, 2D / 3D video, etc.

[0273] Optionally, the generator model can output the first feature information and the motion information of the current frame. In this case, the generator model can be improved to enable it to perform temporal and spatial domain processing, changing from outputting only one type of feature information to outputting both the first feature information and the motion information.

[0274] Optionally, the generated feature information output by the generative model can be decomposed to obtain the first feature information and motion information of the current frame. This feature decomposition process can be performed using a spatiotemporal feature extractor (e.g., a Casual 3D Convolution network, 3D CNN, or other neural networks, without specific limitations) to process the generated feature information output by the generative model in both the temporal and spatial domains, thereby obtaining the first feature information and motion information of the current frame. In this case, further feature decomposition of the generated feature information output by the generative model allows for the acquisition of the first feature information and motion information without modifying the structure of the generative model, thus achieving compatibility with the generative model. Furthermore, further feature decomposition of the generated feature information output by the generative model yields motion information that can compensate for the first feature information, making it closer to the original content of the current frame. In AIGC, the reconstructed content is obtained from the enhanced feature information, improving the quality of the reconstructed content.

[0275] Step 602: Perform motion modeling based on the first feature information and motion information of the current frame to obtain the second feature information of the current frame.

[0276] Motion modeling is used to compensate for the primary feature information of the current frame based on motion information. For example, motion modeling establishes inter-frame correlations by analyzing motion information (e.g., optical flow, inter-frame differences, displacement fields, block matching information, depth information, etc.) between frames in video or other media. The core task of motion modeling is to compensate for the features of different frames using this motion information. Compensation methods can include transformations (e.g., warping) and fusion (e.g., weighted summation) to enhance inter-frame consistency, improve reconstruction quality, and optimize inter-frame visual performance. Motion modeling not only includes the analysis of inter-frame motion but also involves the process of enhancing the current frame based on this motion information: such as temporal compensation, spatial compensation, and multi-frame fusion. Therefore, motion modeling can enhance the primary feature information, making it closer to the original content of the current frame. This allows for better inter-frame reference in the encoding and decoding of media content such as audio, speech, images, and 2D / 3D video during the decoding process, thereby improving the quality of the decoder's output features and increasing temporal compression efficiency.

[0277] In one possible implementation, the feature information of the reference frame can be obtained first; then temporal compensation can be performed based on the motion information and the feature information of the reference frame to obtain the prediction information of the current frame; then the prediction information and the first feature information of the current frame can be fused to obtain the second feature information of the current frame.

[0278] In this application, the reference frame includes at least one of the following frames:

[0279] (1) The reference frame includes the previous n frames in the video sequence, where n ≥ 1. For example, if the current frame is the 3rd frame in the video sequence, its reference frame could be the 2nd frame in the video sequence; or, if the current frame is the 3rd frame in the video sequence, its reference frames could be the 1st and 2nd frames in the video sequence. It should be understood that n can also be other values, and this application does not specifically limit them.

[0280] (2) The reference frame includes the m frames following the current frame in the video sequence, where m ≥ 1. For example, if the current frame is the first frame in the video sequence, its reference frame could be the second frame in the video sequence; or, if the current frame is the third frame in the video sequence, its reference frames could be the second and third frames in the video sequence. It should be understood that m can also be other values, and this application does not specifically limit them.

[0281] (3) The reference frame includes a specified frame in the image frame group to which the current frame belongs (e.g., the first frame in the image frame group to which the current frame belongs, or the middle frame in the image frame group to which the current frame belongs, or the last frame in the image frame group to which the current frame belongs, etc.). The image frame grouping is obtained by grouping the image frames contained in the video sequence. For example, the video sequence contains 9 frames 1-9, which are divided into 3 image frame groups. The first group includes image frames 1-3, the second group includes image frames 4-6, and the third group includes image frames 7-9. In the first group, image frames 2-3 reference image frame 1, and image frame 1 references itself. In the second group, image frames 5-6 reference image frame 4, and image frame 4 references itself. In the third group, image frames 8-9 reference image frame 7, and image frame 7 references itself. It should be understood that the number of image frames contained in the image frame group can also be other values, and the number of image frame groups obtained from the video sequence can also be other values. Furthermore, the number of image frames contained in each image frame group can be the same or different. This application does not specifically limit the foregoing content.

[0282] (4) When the current frame is the first frame in the video sequence, the reference frame includes the current frame. For example, if the current frame is the first frame in the video sequence, its reference frame is itself. Optionally, the first frame can be copied as the 0th frame, in which case the reference frame of the first frame is the 0th frame.

[0283] (5) When the current frame is the last frame in the video sequence, the reference frame includes the current frame. For example, if the current frame is the 9th frame in the video sequence (the video sequence contains 9 frames 1-9), its reference frame is itself. Optionally, the 9th frame can be copied as the 10th frame, in which case the reference frame of the 9th frame is the 10th frame.

[0284] In this application, when AIGC generates an image, the reference frame for the current frame is the current frame itself.

[0285] It should be noted that the reference frame of the current frame in this application can be one of the above five types of reference frames. For example, if it is a forward single-frame reference, the reference frame of the current frame can be the first type (1), or if it is a backward single-frame reference, the reference frame of the current frame can be the second type (2). The reference frame of the current frame in this application can also be a combination of two or more of the above five types of reference frames. For example, if it is a bidirectional single-frame reference, the reference frame of the current frame can be a combination of the first type (1) and the second type (2). This application does not make any specific limitation on this.

[0286] In one possible implementation, when the current frame is the first or last frame of a video sequence, the current frame does not require a reference frame; that is, it does not need to reference itself or other frames, and motion modeling is unnecessary. It can be directly reconstructed. In this application, the temporal compensation method varies depending on the different motion information.

[0287] (1) When the motion information includes optical flow information, the temporal compensation performed includes warping operations based on the motion information and the feature information of the reference frame. For example, 2D Warp and DCN Warp introduced above.

[0288] (2) When the motion information includes inter-frame difference information, the temporal compensation performed includes the addition or weighted summation operation based on the motion information and the feature information of the reference frame. For example, the motion information and the feature information of the reference frame are added together.

[0289] It should be noted that since other forms of motion information can be used in this application, the time-domain compensation used accordingly can also include other methods, and no specific limitation is made.

[0290] In this application, the prediction information of the current frame and the first feature information of the current frame can be fused to generate the second feature information of the current frame. The aforementioned fusion method includes inputting the two types of information into a neural network for fusion, such as CNN, Transformer, etc. This application does not specifically limit this.

[0291] Optionally, the feature information of the reference frame includes the original feature information of the reference frame. This original feature information can correspond to the first feature information mentioned above, that is, the feature information that has not undergone motion modeling.

[0292] That is, in the process of obtaining the second feature information of the current frame, the first feature information of the reference frame can be used as a factor affecting the temporal compensation result (the prediction information of the current frame). In this way, the prediction information of the current frame will not be affected by the motion modeling process of the reference frame. Each frame in the video sequence can be reconstructed independently and in parallel, thereby improving the decoding efficiency.

[0293] The first feature information of the reference frame can be obtained by referring to the method for obtaining the first feature information of the current frame described above.

[0294] Optionally, the feature information of the reference frame includes feature information compensated for by adjusting the original feature information of the reference frame. This compensated feature information can correspond to the second feature information mentioned above, that is, the feature information after motion modeling.

[0295] That is, in the process of obtaining the second feature information of the current frame, the second feature information of the reference frame can be used as a factor affecting the temporal compensation result (the prediction information of the current frame). In this way, the prediction information of the current frame depends on the reconstructed features of the reference frame, and the high-quality feature information after temporal compensation can be passed to the next frame, thereby improving the decoding and reconstruction quality.

[0296] The second feature information of the reference frame can be obtained by referring to the method for obtaining the second feature information of the current frame described above.

[0297] Step 603: Obtain the reconstructed content of the current frame based on the second feature information of the current frame.

[0298] In one possible implementation, when the second feature information of the current frame corresponds to a pixel field, the reconstructed content of the current frame is the second feature information of the current frame.

[0299] The neural network used in the above fusion process can realize feature reconstruction and directly output the second feature information corresponding to the pixel domain. At this time, the feature value in the second feature information corresponds to the pixel value of the image. Therefore, the second feature information of the current frame can be directly used as the reconstruction content of the current frame.

[0300] In one possible implementation, when the second feature information of the current frame corresponds to the feature domain, feature reconstruction is performed on the second feature information of the current frame to obtain the reconstructed content of the current frame.

[0301] The neural network used in the above fusion process can focus solely on processing feature information, outputting second feature information corresponding to the feature domain. In this case, feature reconstruction is needed to obtain pixel domain information from the second feature information in order to obtain the reconstructed content of the current frame. For example, the second feature information can be input into a feature reconstruction network (e.g., a 2D convolutional network) to output the reconstructed content of the current frame. It should be noted that other networks can also be used for feature reconstruction, and this application does not specifically limit this.

[0302] As can be seen, whether or not the second feature information needs to be post-processed (i.e., feature reconstruction or feature transformation) in this application depends on the output of motion modeling, and this application does not make specific limitations on this.

[0303] In one possible implementation, motion modeling of multiple frames in the video sequence to which the current frame belongs, based on their respective first feature information and motion information, is processed in parallel. When processing the video sequence, the feature information based on the reference frame can be the original feature information of the reference frame. Therefore, there is no need to wait for the motion modeling of the reference frame to complete. In this case, motion modeling of multiple frames in the video sequence can be processed in parallel, synchronously based on their respective first feature information and motion information. That is, multiple frames in the video sequence synchronously acquire the feature information of their respective reference frames, then perform temporal compensation based on the feature information of their respective reference frames to obtain their respective prediction information, and then reconstruct the video.

[0304] In this embodiment, the first feature information and motion information of the current frame are obtained by generating feature information based on the output of the AIGC generation model. In the decoding process, the motion information is used to compensate for the first feature information, which can enhance the first feature information and make it closer to the original content of the current frame. On the one hand, it can better realize inter-frame reference in media content encoding and decoding and improve temporal compression efficiency. On the other hand, the reconstructed content is obtained by reconstructing from the enhanced feature information in AIGC, which can improve the quality of the reconstructed content.

[0305] Figure 7 is a flowchart of process 700 of the encoding method in AIGC generation provided in this application. Process 700 can be executed by the terminal device described above. Process 700 is described as a series of steps or operations. It should be understood that process 700 can be executed in various orders and / or occur simultaneously, and is not limited to the execution order shown in Figure 7. Process 700 may include:

[0306] Step 701: Obtain the first feature information and motion information of the current frame.

[0307] Motion information is used to characterize inter-frame correlation; that is, it describes the difference between the current frame and the reference frame. For example, motion information can be optical flow information, inter-frame difference information (also known as inter-frame difference information), etc. In addition, motion information can also include other information, without specific limitations.

[0308] In one possible implementation, the feature information can be obtained through a feature extraction network (e.g., a 2D convolutional network).

[0309] In one possible implementation, motion information can be obtained through a motion information extraction network (e.g., an optical flow estimation network).

[0310] It should be noted that the aforementioned neural network can also employ other AI technologies, and this application does not impose specific limitations on them.

[0311] Step 702: Perform motion modeling based on the first feature information and motion information of the current frame to obtain the second feature information of the current frame.

[0312] In this application, the motion modeling in the encoding process can be described with reference to step 602 of the embodiment shown in Figure 6, and will not be repeated here.

[0313] Step 703: Obtain the encoded feature information of the current frame based on the second feature information and motion information of the current frame. This encoded feature information is used as input to the AIGC generation model.

[0314] In this application, the second feature information and motion information of the current frame can be input into a spatiotemporal feature extractor (e.g., a Casual 3DConvolution network) to obtain the encoded feature information of the current frame. It should be noted that the spatiotemporal feature extractor can also use other networks, and this application does not specifically limit it.

[0315] In one possible implementation, motion modeling of multiple frames in the video sequence to which the current frame belongs, based on their respective first feature information and motion information, is processed in parallel. When processing the video sequence, the feature information based on the reference frame can be the original feature information of the reference frame. Therefore, there is no need to wait for the motion modeling of the reference frame to complete. In this case, motion modeling of multiple frames in the video sequence can be processed in parallel, synchronously based on their respective first feature information and motion information. That is, multiple frames in the video sequence synchronously acquire the feature information of their respective reference frames, then perform temporal compensation based on the feature information of their respective reference frames to obtain their respective prediction information, and then reconstruct the video.

[0316] In this embodiment, by using motion information to compensate for the first feature information during the encoding process of AIGC generation, the first feature information can be enhanced, making it closer to the original content of the current frame. This allows for better inter-frame reference in media content encoding and decoding, thereby improving the quality of the encoder output features and assisting in improving temporal compression efficiency.

[0317] The technical solutions of the method embodiments shown in Figures 6 and 7 will be described in detail below using several specific examples.

[0318] Figure 8 is a framework diagram of the encoder in the AIGC generation of this application. As shown in Figure 8, the modules in the dashed boxes correspond to optional steps. The encoder process includes the following steps:

[0319] 1. Extract feature information (first feature information) from the input medium through a feature extraction network.

[0320] 2. (Optional) Extract motion information from the input media using a motion information extraction network.

[0321] 3. (Optional) Perform motion modeling based on motion information and feature information and fuse them into new features (second feature information).

[0322] 4. (Optional) Motion information and new features are then used again by the feature extraction network to generate encoded feature information.

[0323] Figure 9 is a framework diagram of the decoder in the AIGC generation of this application. As shown in Figure 9, the modules in the dashed boxes correspond to optional steps. The decoder process includes the following steps:

[0324] 1. (Optional) The features generated by the generative model (generated feature information) are decomposed into motion information and feature information of the current frame (first feature information) through motion modeling preprocessing.

[0325] 2. Extract motion information and current frame feature information from the features generated by the generative model. Motion information can take many forms, such as optical flow information and inter-frame difference information, which are used to describe the relevant information between adjacent frames and correspond to different temporal compensation methods. For example, if the motion information is optical flow information, it corresponds to the Warp operation; or if the motion information is inter-frame difference information, it corresponds to the weighted summation operation.

[0326] 3. Motion modeling is performed based on motion information and the feature information of the current frame to obtain new generated features (secondary feature information). For example, temporal compensation is performed based on motion information and the feature information of the reference frame of the current frame to obtain the prediction information of the current frame; then, the aforementioned prediction information and the feature information of the current frame are fused to obtain new generated features. There can be various reference frames, such as the previous frame, the two previous frames, the previous and next frames, the first frame, etc. When AIGC generates an image, the reference frame of the current frame is the current frame. Motion modeling can be performed at the spatial size of the (downsampled) features or at the size of the original video, mainly depending on the upsampling and downsampling of the preprocessing and postprocessing, and there is no specific limitation on this.

[0327] 4. (Optional) Post-process the new generated features through motion modeling to generate the final media.

[0328] The networks used for preprocessing and postprocessing are not specifically limited and can be common neural networks, such as CNNs, multilayer perceptrons (MLPs), Transformers, or combinations thereof with activation functions.

[0329] Based on the encoder and decoder framework described above, and in conjunction with specific embodiments, the encoding and decoding in the AIGC generation of this application will be explained. The video sequence includes t (t>1) image frames, denoted as x. 1:t .

[0330] Figure 10a is a schematic diagram of the inter-frame reference method of this application. As shown in Figure 10a, the temporal reference method of this embodiment is forward single-frame reference, that is, the reference frame is the previous frame of the current frame in the video sequence. Each frame in the video sequence, except for the first frame, only references the frame with the earliest time sequence (black solid line), and the first frame references itself. Figure 10b is a flowchart of the encoding method in AIGC generation of this application. As shown in Figure 10b, the encoding process includes the following steps:

[0331] 1. First Frame Processing: The first frame is copied as the new first frame of the video sequence, thus the video sequence consists of t+1 image frames, denoted as x. 0:t .

[0332] 2. Generation of motion information: Video sequence x 0:t Motion information m between each frame and the previous frame is obtained through an optical flow estimation network. 1:t .

[0333] 3. Feature Domain Transformation: Transform the video sequence x 0:t The feature information is converted into feature domain features by a feature extraction network (e.g., a 2D convolutional network), denoted as f. 1:t ;

[0334] Motion information m 1:t The motion information is converted into feature domain by the feature extraction network and represented as mf 1:t .

[0335] 4. Temporal compensation: This is applied to feature information f. 1:t According to the motion information mf 0:t Feature information f from the previous reference frame 0:t-1 Perform temporal compensation (e.g., 2D Warp or DCN Warp) to generate prediction information p. 1:t .

[0336] 5. Feature fusion: Integrating the predicted information p 1:t Feature information f 1:t and motion information mf 1:t A common input spatiotemporal feature extractor (e.g., a Casual 3D Convolution network) generates encoded feature information z.

[0337] Figure 10c is a flowchart of the decoding method in the AIGC generation of this application. As shown in Figure 10c, the decoding process includes the following steps:

[0338] 1. Obtaining encoded features: Obtaining generated feature information through a generative model.

[0339] 2. (Preprocessing) Feature Decomposition: The generated feature information is processed using a spatiotemporal feature extractor (e.g., Casual 3D Convolution network). Perform temporal and spatial processing and decompose it into motion information. and feature information

[0340] 3. (Motion Modeling) First Frame Processing: Based on feature information, the first frame is copied as the new first frame of the video sequence. Therefore, the video sequence consists of t+1 image frames, represented as...

[0341] 4. (Motion Modeling) Temporal Compensation: In the feature domain, based on motion information... Feature information of the previous reference frame Perform temporal compensation (e.g., 2D Warp or DCN Warp) to generate prediction information.

[0342] 5. (Motion Modeling) Feature Fusion: Integrating Predictive Information and feature information Fusion generates new feature information r 1:t .

[0343] 6. (Post-processing) Generate the final video: This involves adding new feature information r... 1:t The final video output is generated through a feature reconstruction network (e.g., a 2D convolutional network).

[0344] Based on the embodiment shown in Figure 10c, the temporal reference method in this embodiment is forward single-frame reference, that is, the reference frame is the previous frame of the current frame in the video sequence. The difference is that the feature information of the reference frame in the decoding process refers to the new feature information generated in the feature fusion step.

[0345] Decoding process:

[0346] 1. Obtaining encoded features: Obtaining generated feature information through a generative model.

[0347] 2. (Preprocessing) Feature Decomposition: The generated feature information is processed using a spatiotemporal feature extractor (e.g., Casual 3D Convolution network). Perform temporal and spatial processing and decompose it into motion information. and feature information

[0348] 3. (Motion Modeling) First Frame Processing: Based on feature information, directly use the feature information. As r1.

[0349] 4. (Motion Modeling) Temporal Compensation: In the feature domain, based on motion information... Feature information r of the previous reference frame i-1 Perform temporal compensation (e.g., 2D Warp or DCN Warp) to generate prediction information. Initialize i = 2.

[0350] 5. (Motion Modeling) Feature Fusion: Integrating Predictive Information and feature information Fusion generates new feature information r i .

[0351] Repeat steps 4 and 5 until new feature information r is generated. t .

[0352] 6. (Post-processing) Generate the final video: This involves adding new feature information r... 1:t The final video output is generated through a feature reconstruction network (e.g., a 2D convolutional network).

[0353] Figure 11a is a schematic diagram of the inter-frame reference method of this application. As shown in Figure 11a, the temporal reference method of this embodiment is a bidirectional single-frame reference, that is, the reference frames are the previous frame and the next frame in the video sequence. Each frame in the video sequence, except for the first frame and the last frame, references the frame that is earlier in the time sequence (black solid line) and the frame that is later in the time sequence (light solid line). The first frame references itself and the next frame, and the last frame references the frame that is earlier in the time sequence and itself. Figure 11b is a flowchart of the encoding method in AIGC generation of this application. As shown in Figure 11b, the encoding process includes the following steps:

[0354] 1. First and Last Frame Processing: The first frame is copied as the new first frame of the video sequence, and the last frame is copied as the new last frame of the video sequence. Thus, the video sequence consists of t+2 image frames, denoted as x. 0:t+1 .

[0355] 2. Generation of motion information: Video sequence x 0:t+1 Motion information m between each frame and the previous frame is obtained through an optical flow estimation network. 1:t Motion information b between each frame and the next frame 1:t .

[0356] 3. Feature Domain Transformation: Transform the video sequence x 0:t+1 The feature information is converted into feature domain features by a feature extraction network (e.g., a 2D convolutional network), denoted as f. 1:t ;

[0357] Motion information m 1:t Through a feature extraction network, the motion information is converted into feature domain motion information, represented as mf 1:t ;

[0358] Motion information b 1:t Through a feature extraction network, the motion information is converted into feature domain motion information, represented as bf. 1:t .

[0359] 4. Temporal compensation: This is applied to feature information f. 1:t According to the motion information mf 1:t Feature information f from the previous reference frame 0:t-1Perform temporal compensation (e.g., 2D Warp or DCN Warp) to generate prediction information p. 1:t ;

[0360] For feature information f 1:t According to the motion information bf 1:t and the feature information f of the next reference frame 2:t+1 Perform temporal compensation (e.g., 2DWarp or DCN Warp) to generate prediction information p. b1:t .

[0361] 5. Feature fusion: Integrating the predicted information p 1:t p b1:t Feature information f 1:t and motion information mf 1:t bf 1:t A common input spatiotemporal feature extractor (e.g., a Casual 3D Convolution network) generates encoded feature information z.

[0362] Figure 11c is a flowchart of the decoding method in the AIGC generation of this application. As shown in Figure 11c, the decoding process includes the following steps:

[0363] 1. Obtaining encoded features: Obtaining generated feature information through a generative model.

[0364] 2. (Preprocessing) Feature Decomposition: The generated feature information is processed using a spatiotemporal feature extractor (e.g., Casual 3D Convolution network). Perform temporal and spatial processing and decompose it into motion information. and feature information

[0365] 3. (Motion Modeling) First Frame Processing: Based on feature information, the first frame is copied as the new first frame of the video sequence, and the last frame is copied as the new last frame of the video sequence. Thus, the video sequence consists of t+2 image frames, represented as follows:

[0366] 4. (Motion Modeling) Temporal Compensation: In the feature domain, based on motion information... Feature information of the previous reference frame Perform temporal compensation (e.g., 2D Warp or DCN Warp) to generate prediction information.

[0367] In the feature domain, based on motion information and the feature information of the next reference frame Perform temporal compensation (e.g., 2D Warp or DCN Warp) to generate prediction information.

[0368] 5. (Motion Modeling) Feature Fusion: Integrating Predictive Information and feature information Fusion generates new feature information r 1:t .

[0369] 6. (Post-processing) Generate the final video: This involves adding new feature information r... 1:t The final video output is generated through a feature reconstruction network (e.g., a 2D convolutional network).

[0370] Figure 12 is a schematic diagram of the inter-frame reference method of this application. As shown in Figure 12, the temporal reference method of this embodiment is a one-way multi-frame reference, that is, the reference frame is the first two frames of the current frame in the video sequence. Each frame in the video sequence except the first frame and the second first frame references the two frames that are earlier in the time sequence (black solid line and light dashed line). The first frame references itself, and the second first frame references the first frame.

[0371] Decoding process:

[0372] 1. Obtaining encoded features: Obtaining generated feature information through a generative model.

[0373] 2. (Preprocessing) Feature Decomposition: The generated feature information is processed using a spatiotemporal feature extractor (e.g., Casual 3D Convolution network). Perform temporal and spatial processing and decompose it into motion information. and feature information

[0374] 3. (Motion Modeling) First Frame Processing: Based on feature information, the first frame is copied as the new first frame of the video sequence. Therefore, the video sequence consists of t+1 image frames, represented as...

[0375] 4. (Motion Modeling) Temporal Compensation: In the feature domain, based on motion information... Feature information of the previous reference frame Perform temporal compensation (e.g., 2D Warp or DCN Warp) to generate prediction information.

[0376] In the feature domain, based on motion information Feature information of the previous two reference frames Perform motion compensation (e.g., 2D Warp or DCN Warp) to generate prediction information.

[0377] 5. (Motion Modeling) Feature Fusion: Integrating Predictive Information and feature information Fusion generates new feature information r 1:t .

[0378] 6. (Post-processing) Generate the final video: This involves adding new feature information r... 1:t The final video output is generated through a feature reconstruction network (e.g., a 2D convolutional network).

[0379] Figure 13a is a schematic diagram of the inter-frame reference method of this application. As shown in Figure 13a, the temporal reference method of this embodiment is the first frame reference within a group, that is, the reference frame is the first frame in the image frame group to which the current frame belongs. Each frame in each image frame group, except for the first frame within the group, only references the first frame within the group (black solid line), and the first frame within the group is not referenced. Figure 13b is a flowchart of the encoding method in AIGC generation of this application. As shown in Figure 13b, the encoding process includes the following steps:

[0380] 1. Generation of motion information: Video sequence x 1:t Motion information m between each frame and the first frame in the group is obtained through an optical flow estimation network. 2:t .

[0381] 2. Feature Domain Transformation: Transform the video sequence x 1:t The feature information is converted into feature domain features by a feature extraction network (e.g., a 2D convolutional network), denoted as f. 1:t ;

[0382] Motion information m 2:t The motion information is converted into feature domain by the feature extraction network and represented as mf 2:t .

[0383] 3. Temporal compensation: For feature information f 2:t According to the motion information mf 2:t Temporal compensation (e.g., 2D Warp or DCN Warp) is performed on the feature information f1 of the first frame within the group to generate prediction information p. 2:t And by feature information f 2:t and predictive information p 2:t Generate difference information res 2:t The difference information in this application can also be called feature information, which is a specific implementation of feature information.

[0384] 4. Feature compression: Combine the feature information f1 and difference information res of the first frame within the group. 2:t and motion information mf 2:tThe spatiotemporal feature compression network is input separately to generate encoded feature information z_f1 and z_res. 2:t z_mf 2:t .

[0385] Figure 13c is a flowchart of the decoding method in AIGC generation of this application. As shown in Figure 13c, the decoding process includes the following steps:

[0386] 1. Obtaining encoded features: Obtaining generated feature information through a generative model.

[0387] 2. (Preprocessing) Feature Decomposition: The three generated feature information are processed in both the temporal and spatial domains using a spatiotemporal feature extractor (e.g., Casual 3D Convolution network) to obtain the first frame information. Difference information and sports information

[0388] 3. (Motion Modeling) Temporal Compensation: In the feature domain, based on motion information... and the feature information of the first frame within the group Perform temporal compensation (e.g., 2D Warp or DCN Warp) to generate prediction information.

[0389] 4. (Motion Modeling) Feature Fusion: Integrating Predictive Information and difference information Fusion generates new feature information r 2:t First frame information This serves as the new feature information r1.

[0390] 5. (Post-processing) Generate the final video: This involves adding new feature information r... 1:t The final video output is generated through a feature reconstruction network (e.g., a 2D convolutional network).

[0391] Based on the embodiment shown in Figure 10b, Figure 14 is a flowchart of the encoding method in the AIGC generation of this application. As shown in Figure 14, the temporal reference method of this embodiment is forward single-frame reference, that is, the reference frame is the previous frame of the current frame in the video sequence. The difference is that there is no step of temporal compensation based on motion information in the encoding process. The feature information is directly input into the spatiotemporal feature extractor to generate encoded feature information z.

[0392] Optionally, the scale of motion modeling in the decoding process can be the original video scale or the feature scale (generally smaller than the original video), without specific limitations.

[0393] Based on the embodiment shown in Figure 10b, the difference in this embodiment is that the motion information is no longer optical flow information, but inter-frame difference information.

[0394] Decoding process:

[0395] 1. Obtaining encoded features: Obtaining generated feature information through a generative model.

[0396] 2. (Preprocessing) Feature Decomposition: The generated feature information is processed using a spatiotemporal feature extractor (e.g., Casual 3D Convolution network). Perform temporal and spatial processing and decompose it into motion information. and feature information

[0397] 3. (Motion Modeling) First Frame Processing: Based on feature information, the first frame is copied as the new first frame of the video sequence. Therefore, the video sequence consists of t+1 image frames, represented as...

[0398] 4. (Motion Modeling) Temporal Compensation: In the feature domain, based on motion information... Feature information of the previous reference frame Perform weighted summation to generate prediction information.

[0399] 5. (Motion Modeling) Feature Fusion: Integrating Predictive Information and feature information Fusion generates new feature information r 1:t .

[0400] 6. (Post-processing) Generate the final video: This involves adding new feature information r... 1:t The final video output is generated through a feature reconstruction network (e.g., a 2D convolutional network).

[0401] Figure 15 is a structural schematic diagram of the decoding device 1500 of this application. As shown in Figure 15, the decoding device 1500 of this embodiment can be applied to the aforementioned terminal device. The decoding device 1500 may include: an acquisition module 1501, a motion modeling module 1502, and a reconstruction module 1503.

[0402] The acquisition module 1501 is used to obtain the first feature information and motion information of the current frame based on the generated feature information output by the AIGC generation model, wherein the motion information is used to characterize the inter-frame correlation; the motion modeling module 1502 is used to perform motion modeling based on the first feature information and the motion information of the current frame to obtain the second feature information of the current frame, wherein the motion modeling is used to compensate the first feature information of the current frame based on the motion information; the reconstruction module 1503 is used to obtain the reconstructed content of the current frame based on the second feature information of the current frame.

[0403] In one possible implementation, the acquisition module 1501 is specifically used to perform feature decomposition on the generated feature information to obtain the first feature information of the current frame and the motion information.

[0404] In one possible implementation, the acquisition module 1501 is specifically used to output the first feature information and the motion information of the current frame based on the AIGC generation model.

[0405] In one possible implementation, the motion modeling module 1502 is specifically used to acquire feature information of a reference frame of the current frame; perform temporal compensation based on the motion information and the feature information of the reference frame to obtain prediction information of the current frame; and fuse the prediction information and the first feature information of the current frame to obtain the second feature information of the current frame.

[0406] In one possible implementation, the reference frame includes at least one of the following frames:

[0407] The reference frame includes the previous n frames of the current frame in the video sequence, where n ≥ 1; or,

[0408] The reference frame includes the m frames following the current frame in the video sequence, where m ≥ 1; or...

[0409] The reference frame includes a designated frame within the image frame group to which the current frame belongs, wherein the image frame grouping is obtained by grouping the image frames contained in the video sequence; or...

[0410] When the current frame is the first frame in the video sequence, the reference frame includes the current frame; or,

[0411] When the current frame is the last frame in the video sequence, the reference frame includes the current frame.

[0412] In one possible implementation, when the AIGC generates an image, the reference frame for the current frame is the current frame.

[0413] In one possible implementation, the feature information of the reference frame includes the original feature information of the reference frame; or, the feature information of the reference frame includes feature information after compensation of the original feature information of the reference frame.

[0414] In one possible implementation, when the motion information includes optical flow information, the temporal compensation includes a warping operation based on the motion information and feature information of the reference frame; or, when the motion information includes inter-frame difference information, the temporal compensation includes a weighted summation operation based on the motion information and feature information of the reference frame.

[0415] In one possible implementation, when the video sequence to which the current frame belongs comprises multiple frames, the motion modeling of the multiple frames based on their respective first feature information and motion information is performed in parallel.

[0416] In one possible implementation, the reconstruction module 1503 is specifically used to: when the second feature information of the current frame corresponds to a pixel domain, the reconstructed content of the current frame is the second feature information of the current frame; or, when the second feature information of the current frame corresponds to a feature domain, to perform feature reconstruction on the second feature information of the current frame to obtain the reconstructed content of the current frame.

[0417] The apparatus in this embodiment can be used to execute the technical solution of the method embodiment shown in FIG6. Its implementation principle and technical effect are similar, and will not be described again here.

[0418] Figure 16 is a schematic diagram of the structure of the encoding device 1600 of this application. As shown in Figure 16, the encoding device 1600 of this embodiment can be applied to the aforementioned terminal device. The encoding device 1600 may include: an acquisition module 1601 and a processing module 1602.

[0419] The acquisition module 1601 is used to acquire first feature information and motion information of the current frame, wherein the motion information is used to characterize inter-frame correlation; the processing module 1602 is used to perform motion modeling based on the first feature information and motion information of the current frame to obtain second feature information of the current frame, wherein the motion modeling is used to compensate the first feature information of the current frame based on the motion information; and to acquire encoded feature information of the current frame based on the second feature information and motion information, wherein the encoded feature information is used to input to the AIGC generation model.

[0420] In one possible implementation, the processing module 1602 is specifically used to obtain feature information of a reference frame of the current frame; perform temporal compensation based on the motion information and the feature information of the reference frame to obtain prediction information of the current frame; and fuse the prediction information and the first feature information of the current frame to obtain the second feature information of the current frame.

[0421] In one possible implementation, the reference frame includes at least one of the following frames:

[0422] The reference frame includes the previous n frames of the current frame in the video sequence, where n ≥ 1; or,

[0423] The reference frame includes the m frames following the current frame in the video sequence, where m ≥ 1; or...

[0424] The reference frame includes the first frame in the image frame group to which the current frame belongs, and the image frame grouping is obtained by grouping the image frames contained in the video sequence; or...

[0425] When the current frame is the first frame in the video sequence, the reference frame includes the current frame; or,

[0426] When the current frame is the last frame in the video sequence, the reference frame includes the current frame.

[0427] In one possible implementation, when the AIGC generates an image, the reference frame for the current frame is the current frame.

[0428] In one possible implementation, when the motion information includes optical flow information, the temporal compensation includes a warping operation based on the motion information and feature information of the reference frame; or, when the motion information includes inter-frame difference information, the temporal compensation includes a weighted summation operation based on the motion information and feature information of the reference frame.

[0429] In one possible implementation, when the video sequence to which the current frame belongs comprises multiple frames, the motion modeling of the multiple frames based on their respective first feature information and motion information is performed in parallel.

[0430] In one possible implementation, the feature information of the current frame is obtained through a feature extraction network.

[0431] In one possible implementation, the motion information is obtained through a motion information extraction network.

[0432] The apparatus in this embodiment can be used to execute the technical solution of the method embodiment shown in FIG7. Its implementation principle and technical effect are similar, and will not be described again here.

[0433] It is understood that, in order to achieve the above-mentioned functions, electronic devices include hardware and / or software modules that perform the respective functions. Based on the algorithmic steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0434] In one example, Figure 17 is a schematic block diagram of an apparatus 1700 according to an embodiment of this application. The apparatus 1700 may include a processor 1701. Optionally, it may also include a transceiver / transceiver pin 1702, and optionally, it may also include a memory 1703.

[0435] The various components of device 1700 are coupled together via bus 1704, which includes a data bus, a power bus, a control bus, and a status signal bus. However, for clarity, all buses are referred to as bus 1704 in the figure.

[0436] Optionally, the memory 1703 can be used for the instructions in the foregoing method embodiments. The processor 1701 can be used to execute the instructions in the memory 1703, control the receive pin to receive signals, and control the transmit pin to transmit signals.

[0437] Device 1700 may be a terminal device or a chip of a terminal device in the above method embodiments.

[0438] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0439] In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware encoding processor, or by a combination of hardware and software modules in the encoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0440] The memory mentioned in the above embodiments can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0441] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0442] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0443] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0444] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0445] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0446] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0447] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A decoding method in AIGC (Artificial Intelligence Generated Content) generation, characterized in that, include: The first feature information and motion information of the current frame are obtained based on the generated feature information output by the AIGC generation model. The motion information is used to characterize the inter-frame correlation. Motion modeling is performed based on the first feature information of the current frame and the motion information to obtain the second feature information of the current frame. The motion modeling is used to compensate the first feature information of the current frame based on the motion information. The reconstructed content of the current frame is obtained based on the second feature information of the current frame.

2. The method according to claim 1, characterized in that, The generated feature information based on the AIGC generation model outputs is used to obtain the first feature information and motion information of the current frame, including: The generated feature information is decomposed to obtain the first feature information and the motion information of the current frame.

3. The method according to claim 1, characterized in that, The generated feature information based on the AIGC generation model outputs is used to obtain the first feature information and motion information of the current frame, including: The AIGC generation model outputs the first feature information and the motion information of the current frame.

4. The method according to any one of claims 1-3, characterized in that, The step of performing motion modeling based on the first feature information of the current frame and the motion information to obtain the second feature information of the current frame includes: Obtain the feature information of the reference frame of the current frame; Temporal compensation is performed based on the motion information and the feature information of the reference frame to obtain the prediction information of the current frame; The predicted information and the first feature information of the current frame are fused to obtain the second feature information of the current frame.

5. The method according to claim 4, characterized in that, The reference frame includes at least one of the following frames: The reference frame includes the previous n frames of the current frame in the video sequence, where n ≥ 1; or, The reference frame includes the m frames following the current frame in the video sequence, where m ≥ 1; or, The reference frame includes a designated frame within the image frame group to which the current frame belongs, wherein the image frame grouping is obtained by grouping the image frames contained in the video sequence; or... When the current frame is the first frame in the video sequence, the reference frame includes the current frame; or, When the current frame is the last frame in the video sequence, the reference frame includes the current frame.

6. The method according to claim 4, characterized in that, When the AIGC generates an image, the reference frame for the current frame is the current frame.

7. The method according to any one of claims 4-6, characterized in that, The feature information of the reference frame includes the original feature information of the reference frame; or, The feature information of the reference frame includes feature information after compensation of the original feature information of the reference frame.

8. The method according to any one of claims 4-7, characterized in that, When the motion information includes optical flow information, the temporal compensation includes a warping operation based on the motion information and feature information of the reference frame; or... When the motion information includes inter-frame difference information, the temporal compensation includes a weighted summation operation based on the motion information and the feature information of the reference frame.

9. The method according to any one of claims 1-8, characterized in that, When the video sequence to which the current frame belongs includes multiple frames, the motion modeling of the multiple frames based on their respective first feature information and motion information is performed in parallel.

10. The method according to any one of claims 1-9, characterized in that, The step of obtaining the reconstructed content of the current frame based on the second feature information of the current frame includes: When the second feature information of the current frame corresponds to a pixel field, the reconstructed content of the current frame is the second feature information of the current frame; or, When the second feature information of the current frame corresponds to a feature domain, feature reconstruction is performed on the second feature information of the current frame to obtain the reconstructed content of the current frame.

11. An encoding method for generating AIGC (Artificial Intelligence Generated Content), characterized in that, include: Obtain the first feature information and motion information of the current frame, wherein the motion information is used to characterize the inter-frame correlation; Motion modeling is performed based on the first feature information of the current frame and the motion information to obtain the second feature information of the current frame. The motion modeling is used to compensate the first feature information of the current frame based on the motion information. The encoded feature information of the current frame is obtained based on the second feature information of the current frame and the motion information, and the encoded feature information is used as input to the AIGC generation model.

12. The method according to claim 11, characterized in that, The step of performing motion modeling based on the first feature information of the current frame and the motion information to obtain the second feature information of the current frame includes: Obtain the feature information of the reference frame of the current frame; Temporal compensation is performed based on the motion information and the feature information of the reference frame to obtain the prediction information of the current frame; The predicted information and the first feature information of the current frame are fused to obtain the second feature information of the current frame.

13. The method according to claim 12, characterized in that, The reference frame includes at least one of the following frames: The reference frame includes the previous n frames of the current frame in the video sequence, where n ≥ 1; or, The reference frame includes the m frames following the current frame in the video sequence, where m ≥ 1; or, The reference frame includes the first frame in the image frame group to which the current frame belongs, and the image frame grouping is obtained by grouping the image frames contained in the video sequence; or... When the current frame is the first frame in the video sequence, the reference frame includes the current frame; or, When the current frame is the last frame in the video sequence, the reference frame includes the current frame.

14. The method according to claim 12, characterized in that, When the AIGC generates an image, the reference frame for the current frame is the current frame.

15. The method according to any one of claims 12-14, characterized in that, When the motion information includes optical flow information, the temporal compensation includes a warping operation based on the motion information and feature information of the reference frame; or... When the motion information includes inter-frame difference information, the temporal compensation includes a weighted summation operation based on the motion information and the feature information of the reference frame.

16. The method according to any one of claims 11-15, characterized in that, When the video sequence to which the current frame belongs includes multiple frames, the motion modeling of the multiple frames based on their respective first feature information and motion information is performed in parallel.

17. The method according to any one of claims 11-16, characterized in that, The feature information of the current frame is obtained through a feature extraction network.

18. The method according to any one of claims 11-17, characterized in that, The motion information is obtained through a motion information extraction network.

19. A decoding device, characterized in that, include: The acquisition module is used to obtain the first feature information and motion information of the current frame based on the generated feature information output by the AIGC generation model, wherein the motion information is used to characterize the inter-frame correlation. A motion modeling module is used to perform motion modeling based on the first feature information of the current frame and the motion information to obtain the second feature information of the current frame. The motion modeling is used to compensate the first feature information of the current frame based on the motion information. The reconstruction module is used to obtain the reconstructed content of the current frame based on the second feature information of the current frame.

20. The apparatus according to claim 19, characterized in that, The acquisition module is specifically used to perform feature decomposition on the generated feature information to obtain the first feature information and the motion information of the current frame.

21. The apparatus according to claim 19, characterized in that, The acquisition module is specifically used to output the first feature information and the motion information of the current frame based on the AIGC generation model.

22. The apparatus according to any one of claims 19-21, characterized in that, The motion modeling module is specifically used to acquire feature information of a reference frame of the current frame; perform temporal compensation based on the motion information and the feature information of the reference frame to obtain prediction information of the current frame; and fuse the prediction information and the first feature information of the current frame to obtain the second feature information of the current frame.

23. The apparatus according to claim 22, characterized in that, The reference frame includes at least one of the following frames: The reference frame includes the previous n frames of the current frame in the video sequence, where n ≥ 1; or, The reference frame includes the m frames following the current frame in the video sequence, where m ≥ 1; or, The reference frame includes a designated frame within the image frame group to which the current frame belongs, wherein the image frame grouping is obtained by grouping the image frames contained in the video sequence; or... When the current frame is the first frame in the video sequence, the reference frame includes the current frame; or, When the current frame is the last frame in the video sequence, the reference frame includes the current frame.

24. The apparatus according to claim 22, characterized in that, When the AIGC generates an image, the reference frame for the current frame is the current frame.

25. The apparatus according to any one of claims 22-24, characterized in that, The feature information of the reference frame includes the original feature information of the reference frame; or, The feature information of the reference frame includes feature information after compensation of the original feature information of the reference frame.

26. The apparatus according to any one of claims 22-25, characterized in that, When the motion information includes optical flow information, the temporal compensation includes a warping operation based on the motion information and feature information of the reference frame; or... When the motion information includes inter-frame difference information, the temporal compensation includes a weighted summation operation based on the motion information and the feature information of the reference frame.

27. The apparatus according to any one of claims 19-26, characterized in that, When the video sequence to which the current frame belongs includes multiple frames, the motion modeling of the multiple frames based on their respective first feature information and motion information is performed in parallel.

28. The apparatus according to any one of claims 19-27, characterized in that, The reconstruction module is specifically used to: when the second feature information of the current frame corresponds to a pixel field, the reconstructed content of the current frame is the second feature information of the current frame; or, when the second feature information of the current frame corresponds to a feature field, to perform feature reconstruction on the second feature information of the current frame to obtain the reconstructed content of the current frame.

29. An encoding device, characterized in that, include: The acquisition module is used to acquire the first feature information and motion information of the current frame, wherein the motion information is used to characterize the inter-frame correlation. The processing module is used to perform motion modeling based on the first feature information and the motion information of the current frame to obtain the second feature information of the current frame. The motion modeling is used to compensate the first feature information of the current frame based on the motion information. The module is also used to obtain the encoded feature information of the current frame based on the second feature information and the motion information. The encoded feature information is used to input the encoded feature information into the AIGC generation model.

30. The apparatus according to claim 29, characterized in that, The processing module is specifically used to obtain feature information of a reference frame of the current frame; perform temporal compensation based on the motion information and the feature information of the reference frame to obtain prediction information of the current frame; and fuse the prediction information and the first feature information of the current frame to obtain the second feature information of the current frame.

31. The apparatus according to claim 30, characterized in that, The reference frame includes at least one of the following frames: The reference frame includes the previous n frames of the current frame in the video sequence, where n ≥ 1; or, The reference frame includes the m frames following the current frame in the video sequence, where m ≥ 1; or, The reference frame includes the first frame in the image frame group to which the current frame belongs, and the image frame grouping is obtained by grouping the image frames contained in the video sequence; or... When the current frame is the first frame in the video sequence, the reference frame includes the current frame; or, When the current frame is the last frame in the video sequence, the reference frame includes the current frame.

32. The apparatus according to claim 30, characterized in that, When the AIGC generates an image, the reference frame for the current frame is the current frame.

33. The apparatus according to any one of claims 30-32, characterized in that, When the motion information includes optical flow information, the temporal compensation includes a warping operation based on the motion information and feature information of the reference frame; or... When the motion information includes inter-frame difference information, the temporal compensation includes a weighted summation operation based on the motion information and the feature information of the reference frame.

34. The apparatus according to any one of claims 29-33, characterized in that, When the video sequence to which the current frame belongs includes multiple frames, the motion modeling of the multiple frames based on their respective first feature information and motion information is performed in parallel.

35. The apparatus according to any one of claims 29-34, characterized in that, The feature information of the current frame is obtained through a feature extraction network.

36. The apparatus according to any one of claims 29-35, characterized in that, The motion information is obtained through a motion information extraction network.

37. A decoder, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

38. An encoder, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 11-18.

39. A computer-readable storage medium, characterized in that, Includes a computer program, which, when executed on a computer, causes the computer to perform the method of any one of claims 1-18.

40. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the method of any one of claims 1-18.

41. An AIGC (Artificial Intelligence Generated Content) generation system, characterized in that, include: A decoder and an AIGC generation model; wherein the decoder employs the decoding apparatus of any one of claims 19 to 28, or the decoder of claim 37.

42. The AIGC generation system according to claim 41, characterized in that, Also includes: Encoder.

43. The AIGC generation system according to claim 42, characterized in that, The encoder is the encoding device according to any one of claims 29 to 36, or the encoder according to claim 38.

Images