Model training method, video processing method, electronic device and storage medium

Abstract

The present disclosure relates to the technical field of video processing and the technical field of large models. Disclosed are a model training method, a video processing method, an electronic device and a storage medium. The method comprises: acquiring training video data; using an initial video variational autoencoder model to perform video reconstruction processing on the training video data, so as to obtain a reconstruction result, wherein the initial video variational autoencoder model is used for performing spatial compression of key frames and overall temporal compression on a plurality of video frame groups in the training video data, respectively, so as to complete video reconstruction; and on the basis of the training video data and the reconstruction result, training the initial video variational autoencoder model, so as to generate a target video variational autoencoder model, wherein the target video variational autoencoder model is used for providing reconstructed video data for an initial video generation model, so as to generate a target video generation model by means of training. The present disclosure solves the technical problems in the related art of a video variational autoencoder model having low reconstruction result quality and the optimization of the video variational autoencoder model being slow.

Description

Model training methods, video processing methods, electronic devices and storage media Technical Field

[0001] This disclosure relates to video processing technology and large model technology, specifically to a model training method, a video processing method, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, video generation models have become an important research direction in the fields of computer vision and deep learning. Among them, video generation models based on latent diffusion have shown great potential in generating high-quality, high-resolution video content. These video generation models typically rely on video variational autoencoders (VAEs), which effectively reduce the learning difficulty of latent diffusion models by establishing a bidirectional mapping between high-dimensional pixel space and low-dimensional latent space, thereby improving the quality and efficiency of video generation.

[0003] Current video VAE models have shortcomings in model optimization, operator implementation, and structural design, resulting in slow optimization and low-quality reconstruction results.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This disclosure provides a model training method, a video processing method, an electronic device, and a storage medium to at least address the technical problems of low reconstruction quality and slow optimization of video variational autoencoder models in related technologies.

[0006] According to one aspect of the present disclosure, a model training method is provided, comprising: acquiring training video data; performing video reconstruction processing on the training video data using an initial video variational autoencoder model to obtain a reconstruction result, wherein the initial video variational autoencoder model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data to complete video reconstruction; training the initial video variational autoencoder model based on the training video data and the reconstruction result to generate a target video variational autoencoder model, wherein the target video variational autoencoder model is used to provide reconstructed video data for the initial video generation model to train and generate the target video generation model, and the target video generation model is used to perform video generation processing on input multimedia reference data to output a target video.

[0007] According to another aspect of the embodiments of this disclosure, a model training method is also provided, comprising: acquiring reconstructed video data, wherein the reconstructed video data is generated by a target video variational autoencoder model, the target video variational autoencoder model being obtained by performing any of the above-described model training methods; training an initial video generation model using the reconstructed video data to generate a target video generation model, wherein the target video generation model is used to perform video generation processing on input multimedia reference data to output a target video.

[0008] According to another aspect of the present disclosure, a video processing method is also provided, comprising: acquiring multimedia reference data; performing video generation processing on the multimedia reference data using a target video generation model to obtain a target video; wherein the target video generation model is obtained by training an initial video generation model with reconstructed video data, the reconstructed video data is generated by a target video variational autoencoder model, and the target video variational autoencoder model is obtained by performing any of the above-described model training methods.

[0009] According to another aspect of the embodiments of this disclosure, a video processing method is also provided, comprising: acquiring target product description data, wherein the target product description data includes at least one of the following: target product description text, target product video clips; performing video generation processing on the target product description data using a target video generation model to obtain a target product promotional video; wherein the target video generation model is obtained by training an initial video generation model with reconstructed video data, the reconstructed video data is generated by a target video variational autoencoder model, and the target video variational autoencoder model is obtained by performing any of the above-described model training methods.

[0010] According to another aspect of the embodiments of this disclosure, a video processing method is also provided, comprising: obtaining a video processing request through a first application programming interface, wherein the request data carried in the video processing request includes: multimedia reference data; and returning a video processing response through a second application programming interface, wherein the response data carried in the video processing response includes: a target video, wherein the target video is obtained by performing video generation processing on the multimedia reference data using a target video generation model, the target video generation model is obtained by training an initial video generation model using reconstructed video data, the reconstructed video data is generated by a target video variational autoencoder model, and the target video variational autoencoder model is obtained by executing any of the above-described model training methods.

[0011] According to another aspect of the embodiments of this disclosure, a video processing method is also provided, comprising: acquiring a currently input video processing dialogue request, wherein the request data carried in the video processing dialogue request includes: multimedia reference data; responding to the video processing dialogue request and returning a video processing dialogue response, wherein the information carried in the video processing dialogue response includes: a target video, wherein the target video is obtained by performing video generation processing on the multimedia reference data using a target video generation model, the target video generation model is obtained by training an initial video generation model using reconstructed video data, the reconstructed video data is generated by a target video variational autoencoder model, the target video variational autoencoder model is obtained by performing any of the above-described model training methods; and playing the target video within a graphical user interface.

[0012] According to another aspect of the present disclosure, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes any of the above-described model training method or video processing method during runtime.

[0013] According to another aspect of the embodiments of this disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform any of the above-described model training method or video processing method.

[0014] According to another aspect of the embodiments of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described model training methods or video processing methods.

[0015] In this embodiment, training video data is acquired, and then an initial video variational autoencoder (VAE) model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data, i.e., video reconstruction processing is performed on the training video data to obtain reconstruction results. Subsequently, the initial VAE model is trained based on the training video data and reconstruction results to generate a target VAE model. The target VAE model provides reconstructed video data to the initial video generation model for training and generation. The target video generation model processes the input multimedia reference data to output a target video, thereby achieving the goal of generating a high-quality reconstructed video. This achieves the technical effect of accelerating the convergence speed of video VAE model training, improving the performance of the video VAE model, and enhancing the video quality of the output reconstructed video through the proposed keyframe-based temporal compression architecture. This solves the technical problems of low reconstruction result quality and slow optimization of video variational autoencoder models in related technologies.

[0016] It is worth noting that the above general description and the following detailed description are merely for illustrative and explanatory purposes and do not constitute a limitation thereof. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0018] Figure 1 is a schematic diagram of an application scenario of a model training method according to an embodiment of the present disclosure;

[0019] Figure 2 is a network structure diagram of an all-around video compressor;

[0020] Figure 3 is a flowchart of a model training method according to an embodiment of the present disclosure;

[0021] Figure 4 is a schematic diagram of the framework of the model training method according to an embodiment of the present disclosure;

[0022] Figure 5 is a flowchart of a model training method according to an embodiment of the present disclosure;

[0023] Figure 6 is a flowchart of a video processing method according to an embodiment of the present disclosure;

[0024] Figure 7 is a flowchart of a video processing method according to an embodiment of the present disclosure;

[0025] Figure 8 is a flowchart of a video processing method according to an embodiment of the present disclosure;

[0026] Figure 9 is a flowchart of a video processing method according to an embodiment of the present disclosure;

[0027] Figure 10 is a schematic diagram of a model training device according to an embodiment of the present disclosure;

[0028] Figure 11 is a schematic diagram of another model training device according to an embodiment of the present disclosure;

[0029] Figure 12 is a schematic diagram of another video processing apparatus according to an embodiment of the present disclosure;

[0030] Figure 13 is a schematic diagram of another video processing apparatus according to an embodiment of the present disclosure;

[0031] Figure 14 is a schematic diagram of another video processing apparatus according to an embodiment of the present disclosure;

[0032] Figure 15 is a schematic diagram of another video processing apparatus according to an embodiment of the present disclosure;

[0033] Figure 16 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] The technical solution disclosed herein is primarily implemented using large-scale model technology. Here, "large-scale model" refers to a deep learning model with a massive number of parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large-scale models, also known as foundation models, are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. These models are adaptable to a wide range of downstream tasks and exhibit good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0037] It should be noted that, in practical applications, large models can be fine-tuned using a small number of samples to adapt them to different tasks. For example, large models can be widely used in Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios for large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this embodiment, the explanation is based on a video processing scenario where a target video generation model trained using reconstructed video data provided by a target video variational autoencoder model is used as an example.

[0038] First, some nouns or terms that appear in the description of the embodiments of this disclosure shall be interpreted as follows:

[0039] Video Variational Autoencoder (VAE): A deep learning model specifically designed for processing video data. Video VAEs map high-dimensional video data distributions to a low-dimensional latent space, accompanied by spatial and even temporal compression, to reduce the learning difficulty and computational cost of subsequent latent diffusion models.

[0040] Omni-dimensional Video Compressor for Improving Latent Video Diffusion Model (OD-VAE): A video compression method designed to improve the efficiency of Latent Video Diffusion Models (LVDMs). The core objective of OD-VAE is to compress video data as much as possible while maintaining video reconstruction quality, thereby reducing the computational resource consumption during video generation.

[0041] Latent Video Diffusion Models (LVDMs) are deep learning models primarily used for video generation and processing. LVDMs combine the advantages of latent variable models and diffusion models, enabling efficient generation and editing of high-dimensional video data.

[0042] Currently, the construction of video VAE models typically follows the same paradigm: 1) Using the UNet architecture with good pixel awareness, consistent with the classic image VAE series. 2) Initializing the weights of the video VAE with image VAE weights of the same potential number of channels, thus providing good spatial compression capabilities at the initial stage. 3) Employing three-dimensional (3D) causal convolution with only forward sequence correlation as the basic convolution operator.

[0043] For example, Figure 2 is a network structure diagram of an Omni-dimensional Video Compressor for Improving Latent Video Diffusion Model (OD-VAE). As shown in Figure 2, it can be understood as a schematic diagram of a video VAE network structure trained based on the above paradigm, which has a spatial compression of 8×8 times and a temporal compression of 4 times.

[0044] As can be seen, the input to OD-VAE is video data. OD-VAE receives the input video data and processes it into a frame sequence of a certain length, such as a 25-frame video segment. The resolution of the resulting frame sequence is also adjusted to a specific size, such as 256×256 pixels, though this is not explicitly limited. Thus, the input video data is encoded into a potential low-dimensional representation, involving compression in both spatial and temporal dimensions.

[0045] The output of OD-VAE is a compressed latent representation that can reconstruct the original video with high accuracy. Specifically, OD-VAE uses a 3D causal convolutional neural network (3D-causal-CNN) architecture. This means that a causal-transformation 3D encoder compresses the video into a concise latent representation in both time and space, and a causal-transformation 3D decoder decodes this latent representation to accurately reconstruct the video. This compressed representation can be used for subsequent video generation tasks, such as latent video diffusion models, to generate new videos or perform other video processing tasks. It is understood that the causal-transformation 3D encoder includes multiple downsampling blocks and intermediate blocks, and the causal-transformation 3D decoder includes multiple upsampling blocks and intermediate blocks.

[0046] However, the comprehensive video compressor technology has the following drawbacks.

[0047] Defect 1: The video VAE model has shortcomings in model optimization, operator implementation and structural design, which leads to slow optimization of the video VAE model and low quality of the reconstruction results output by the video VAE model.

[0048] To address the aforementioned deficiencies, no effective solution has been proposed prior to this disclosure.

[0049] According to embodiments of this disclosure, a model training method is provided. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0050] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the model training method provided in this disclosure can be applied to the application scenario shown in Figure 1, but is not limited thereto. In the application scenario shown in Figure 1, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to invoke the large model, thereby implementing the method provided in this disclosure.

[0051] In this embodiment of the disclosure, the system consisting of a client device and a server can perform the following steps: the client device performs steps such as acquiring training video data input by the user in the graphical user interface and sending the training video data to the server; the server performs video reconstruction processing on the acquired training video data using an initial video variational autoencoder model to obtain the reconstruction result, and trains the initial video variational autoencoder model based on the training video data and the reconstruction result to generate a target video variational autoencoder model, and then returns the target video variational autoencoder model to the client device. It should be noted that, provided that the operating resources of the client device can meet the deployment and operation conditions of a large model, this embodiment of the disclosure can be performed on the client device.

[0052] In the above operating environment, this disclosure provides a model training method as shown in Figure 3. Figure 3 is a flowchart of a model training method according to an embodiment of this disclosure. As shown in Figure 3, the method may include the following steps:

[0053] Step S31: Obtain training video data;

[0054] Step S32: The initial video variational autoencoder model is used to perform video reconstruction processing on the training video data to obtain the reconstruction result. The initial video variational autoencoder model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data to complete the video reconstruction.

[0055] Step S33: Train the initial video variational autoencoder model based on the training video data and reconstruction results to generate the target video variational autoencoder model. The target video variational autoencoder model is used to provide reconstructed video data for the initial video generation model to train and generate the target video generation model. The target video generation model is used to perform video generation processing on the input multimedia reference data to output the target video.

[0056] In this embodiment of the disclosure, the training video data can be understood as a video clip or a complete video, and is not limited thereto. It is understood that the training video data in this embodiment of the disclosure is used to train a video VAE model, so that the trained video VAE model can output reconstruction results that are close to or even completely identical to the training video data.

[0057] The initial video variational autoencoder model can be understood as the video VAE model before training. For example, the initial video variational autoencoder model can be a large model or other deep learning models, which is not limited here.

[0058] After acquiring the training video data, it is input into an initial video variational autoencoder (AVCA) model. This model then performs video reconstruction on the training video data, ultimately yielding the reconstruction result. Further, the AVCA model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups within the training video data, thereby completing the video reconstruction and obtaining the reconstruction result. The reconstruction result can be understood as the reconstructed video obtained by reconstructing the training video data using the AVCA model. It is understood that the loss functions of this reconstructed video, such as Mean Absolute Error (MAE) and Learned Perceptual Image Patch Similarity (LPIPS), can be used to update the model parameters of the video generation model, thereby optimizing its performance.

[0059] For example, performing keyframe spatial compression on multiple video frame groups in the training video data can be understood as selecting representative keyframes from each group and then using a deep learning model, such as a two-dimensional (2D) convolutional neural network (CNN), to compress the selected keyframes in both width and height dimensions, converting them into low-dimensional feature representations, thereby extracting image information from the training video data. This approach is not limited to specific methods. By performing keyframe spatial compression on multiple video frame groups in the training video data, the spatial complexity of the video can be effectively reduced while preserving its key information and dynamic characteristics.

[0060] Performing overall temporal compression on multiple video frame groups in the training video data can be understood as using techniques such as 3D convolution, recurrent neural networks (RNNs), long short-term memory (LSTM) networks, or grouped causal convolution to extract temporal features from consecutive video frames—that is, the dynamic change information between frames. Then, the extracted temporal features are compressed and converted into a low-dimensional representation. Simultaneously, techniques such as grouped causal convolution ensure that the performance of the current frame in video processing depends only on previous frames, maintaining the natural temporal logic of the video (this is not a limitation). By performing overall temporal compression on multiple video frame groups in the training video data, the temporal dimension of the video data can be processed, reducing temporal redundancy while preserving the dynamic features and coherence of the video.

[0061] As can be seen, by performing keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data respectively, this embodiment of the present disclosure can ensure that not only are key images accurately restored during video reconstruction, but the dynamic flow and coherence of the video are also maintained, thereby improving the visual quality and viewing experience of the video, that is, improving the video quality of the reconstructed video.

[0062] After obtaining the reconstruction results, the initial video variational autoencoder model is trained based on the training video data and the reconstruction results to generate a target video variational autoencoder model. The target video variational autoencoder model provides reconstructed video data to the initial video generation model for training. This can be understood as the reconstructed video output by the target video variational autoencoder model in this embodiment closely approximating the input video data. Therefore, after obtaining the target video variational autoencoder model, video reconstruction can be performed on the received video data to obtain reconstructed video data. The initial video generation model can then be trained based on this reconstructed video data, thereby training the target video generation model. This reduces computational resource consumption, improves training efficiency and target video generation model performance, and enhances the generalization ability and robustness of the target video generation model, enabling it to support higher quality and higher resolution video generation tasks.

[0063] The target video generation model is used to process input multimedia reference data to generate a target video. The multimedia reference data can be text prompts, image data, video data, audio data, etc., and is not limited here. It can be understood that the target video generation model can learn and imitate the characteristics of the input multimedia reference data to create a new target video that has some visual or content association with the multimedia reference data and meets specific requirements.

[0064] In this embodiment, training video data is acquired, and then an initial video variational autoencoder (VAE) model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data, i.e., video reconstruction processing is performed on the training video data to obtain reconstruction results. Subsequently, the initial VAE model is trained based on the training video data and the reconstruction results to generate a target VAE model. The target VAE model provides reconstructed video data for the initial video generation model to train and generate the target video generation model. The target video generation model processes the input multimedia reference data to output the target video. Therefore, by proposing a keyframe-based temporal compression architecture (i.e., keyframe spatial compression and overall temporal compression), the video VAE model can perform compression and reconstruction more efficiently when processing video data, providing more balanced and powerful temporal-spatial compression capabilities, accelerating the convergence speed of video VAE model training, and improving the performance and resource utilization efficiency of the video VAE model. In addition, it can ensure that not only key images are accurately restored during video reconstruction, but also the dynamic flow and coherence of the video are maintained, thereby improving the visual quality and viewing experience of the video, that is, improving the video quality of the reconstructed video.

[0065] The model training method provided in this disclosure can be applied, but is not limited to, to application scenarios involving the training of video variational autoencoder models or video generation models in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, video variational autoencoder model or video generation model training scenarios in the e-commerce service field, video variational autoencoder model or video generation model training scenarios in the education service field, and video variational autoencoder model or video generation model training scenarios in the legal service field, etc., without limitation.

[0066] By employing the embodiments of this disclosure, training video data is acquired, and then an initial video variational autoencoder (VAE) model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data, i.e., video reconstruction processing is performed on the training video data to obtain reconstruction results. Subsequently, the initial VAE model is trained based on the training video data and reconstruction results to generate a target VAE model. The target VAE model is used to provide reconstructed video data for the initial video generation model to train and generate the target video generation model. The target video generation model is used to perform video generation processing on the input multimedia reference data to output the target video. This achieves the goal of generating high-quality reconstructed video, thereby realizing the technical effect of accelerating the convergence speed of video VAE model training, improving the performance of video VAE model, and improving the video quality of the output reconstructed video through the proposed keyframe-based temporal compression architecture. This solves the technical problems of low reconstruction result quality and slow optimization of video variational autoencoder models in related technologies.

[0067] In one optional embodiment, the initial video variational autoencoder model is used to split the video sequence corresponding to the training video data into multiple video frame groups, wherein the video frames contained in the multiple video frame groups are non-repeating, and the compression process of each video frame group in the multiple video frame groups is decoupled into keyframe spatial compression and overall temporal compression, respectively.

[0068] In this embodiment, the initial video variational autoencoder model can split the input training video data into multiple video frame groups, and the video frames contained in the multiple video frame groups are non-repeating. Furthermore, when the initial video variational autoencoder model performs keyframe spatial compression and overall temporal compression on the multiple video frame groups respectively, the compression process of each video frame group is decoupled into keyframe spatial compression and overall temporal compression, that is, the compression process of each video frame group is decomposed into two independent but interconnected parts: keyframe spatial compression and overall temporal compression of the video frame group.

[0069] In an optional embodiment, in step S32, the training video data is reconstructed using an initial video variational autoencoder model to obtain the reconstruction result, including the following method steps:

[0070] Step S321: Perform keyframe spatial compression coding and overall temporal compression coding on the training video data to obtain the target compression coding result;

[0071] Step S322: Perform parallel dilated convolution on the target compression encoding result to obtain the target convolution result;

[0072] Step S323: Perform keyframe spatial compression decoding and overall temporal compression decoding on the target convolution result to obtain the reconstruction result.

[0073] In this embodiment, when using an initial video variational autoencoder model to reconstruct training video data and obtain the reconstruction result, keyframe spatial compression coding and overall temporal compression coding can be performed on the training video data to obtain the target compression coding result. Spatial compression coding of keyframes ensures that the model captures the most important visual features of the video, while overall temporal compression coding compresses the temporal correlation in the video sequence, improving coding efficiency and reducing computational resource consumption. Then, parallel dilated convolution is performed on the target compression coding result to obtain the target convolution result. Dilated convolution significantly increases the model's receptive field, helping the model better understand object motion and details, maintaining the high quality of the reconstructed video. Finally, keyframe spatial compression decoding and overall temporal compression decoding are performed on the target convolution result to obtain the reconstruction result. By restoring the compressed features to the original video resolution and frame rate, the relationships between frames in the video can be better recovered, ensuring that the reconstructed video has good coherence and smoothness, thereby improving the quality of the reconstructed video.

[0074] In an optional embodiment, the initial video variational autoencoder model includes: a keyframe-based temporal compression encoder. In step S321, keyframe spatial compression encoding and overall temporal compression encoding are performed on the training video data to obtain the target compression encoding result, including the following method steps:

[0075] Step S3211: Use a keyframe-based temporal compression encoder to perform keyframe spatial compression encoding on the training video data to obtain spatial compression encoding results, and perform overall temporal compression encoding on the training video data to obtain temporal compression encoding results.

[0076] Step S3212: Normalize the spatial compression coding result to obtain the first processing result, and normalize the temporal compression coding result to obtain the second processing result;

[0077] Step S3213: The first processing result and the second processing result are concatenated to obtain the target compressed encoding result.

[0078] In this embodiment of the disclosure, the initial video variational autoencoder model includes: a keyframe-based temporal compression encoder, which is used to efficiently compress keyframes in the training video data spatially and temporally to reduce data redundancy and extract feature representations of the video.

[0079] This disclosure presents an embodiment of a dual-branch architecture for keyframe-based motion compression (Kinematic Trajectory Compression, KTC). This KTC dual-branch architecture decouples the compression of a frame group into keyframe spatial compression (i.e., keyframe spatial compression coding) and overall temporal motion compression (i.e., overall temporal compression coding).

[0080] Furthermore, in this disclosure, when using a keyframe-based temporal compression encoder to perform keyframe spatial compression coding and overall temporal compression coding on the training video data to obtain the target compressed coding result, the following steps can be taken: First, the keyframe-based temporal compression encoder can be used to perform keyframe spatial compression coding on the training video data to obtain a spatial compression coding result, and then the overall temporal compression coding on the training video data can be used to obtain a temporal compression coding result. Then, the spatial compression coding result is normalized to obtain a first processing result, and the temporal compression coding result is normalized to obtain a second processing result. Finally, the first processing result and the second processing result are concatenated to obtain the target compressed coding result.

[0081] In other words, this disclosure inputs the training video data into two branches. One branch performs keyframe spatial compression coding on the training video data, extracting image information from the training video data to obtain the spatial compression coding result. The other branch performs overall temporal compression coding on the training video data, extracting the overall temporal motion information of the training video data to obtain the temporal compression coding result. The two output coding results are then normalized and concatenated as the final output. This dual-branch architecture ensures the integrity of keyframe information and the random access characteristics of the video, while also capturing the dynamic flow and coherence of the video, thus jointly promoting efficient video representation and high-quality reconstruction.

[0082] In an optional embodiment, the keyframe-based temporal compression encoder includes: a two-dimensional convolutional unit and a grouped causal convolutional unit. In step S3211, the keyframe-based temporal compression encoder is used to perform keyframe spatial compression encoding on the training video data to obtain a spatial compression encoding result, and the training video data is also subjected to overall temporal compression encoding to obtain a temporal compression encoding result. The method includes the following steps:

[0083] Step S32111: Use two-dimensional convolutional units to perform keyframe spatial compression coding on the training video data to obtain the spatial compression coding result;

[0084] Step S32112: Use grouped causal convolutional units to perform overall temporal compression coding on the training video data to obtain the temporal compression coding result.

[0085] In this embodiment of the disclosure, the temporal compression encoder for keyframes includes: a two-dimensional convolutional unit and a grouped causal convolutional unit. The two-dimensional convolutional unit (Conv 2D) is used to perform convolution operations on video frames using two-dimensional convolutional kernels. The grouped causal convolutional unit (GCConv 3D) divides the input video data into different groups, applies conventional 3D convolution within each group, and applies causal padding between groups to maintain the causal relationship of the time series, that is, future information does not affect the current or past state.

[0086] In this embodiment of the disclosure, when using a keyframe-based temporal compression encoder to perform keyframe spatial compression encoding on the training video data to obtain spatial compression encoding results, and when performing overall temporal compression encoding on the training video data to obtain temporal compression encoding results, a two-dimensional convolutional unit can be used to perform keyframe spatial compression encoding on the training video data to obtain spatial compression encoding results, and a grouped causal convolutional unit can be used to perform overall temporal compression encoding on the training video data to obtain temporal compression encoding results.

[0087] That is, the features of each layer are input into a 2D convolution to extract image information and a grouped causal 3D convolution to extract overall temporal motion information. Then, the features output by the two convolutions are normalized and concatenated together as the output of the unit.

[0088] In an optional embodiment, in step S32112, grouped causal convolutional units are used to perform overall temporal compression coding on the training video data to obtain the temporal compression coding result, including the following method steps:

[0089] The temporal compression coding result is obtained by using grouped causal convolution units to perform three-dimensional convolution on multiple video frames within the same video frame group, and causal padding between different video frame groups within the multiple video frame groups.

[0090] In this embodiment, considering that conventional causal convolution can flexibly model image and video data, but suffers from the problem of a single information flow direction, and that this single information flow direction hinders the interaction between video frames and exacerbates the imbalance in frame performance under time compression, this disclosure proposes a grouped causal convolution method for video VAE models based on time compression.

[0091] Furthermore, in this embodiment of the present disclosure, when performing overall temporal compression coding on the training video data using grouped causal convolutional units to obtain the temporal compression coding result, the grouped causal convolutional units can be used to perform 3D convolution on multiple video frames within the same video frame group. This can be understood as using conventional 3D convolution within the same frame group to make the features between frames mutually visible and interactive, thereby increasing the continuity of the reconstructed video and reducing flicker. Simultaneously, causal padding is performed between different video frame groups within the multiple video frame groups to obtain the temporal compression coding result. This can be understood as using causal padding between frame groups to ensure that the current frame group only depends on the previous frame group, thereby maintaining the causal logic unchanged.

[0092] As can be seen, the grouped causal convolution method disclosed herein groups the input frames according to the temporal compression ratio. Frames within the same frame group use conventional 3D convolution to make the features between frames visible and interactive. Causal padding is used between frame groups to ensure that the current frame group depends only on the previous frame group, thereby maintaining causal relationships between different frame groups and allowing the frame features within the same frame group to interact with each other.

[0093] In an optional embodiment, the model training method further includes the following steps:

[0094] Step S34: Pre-train the image variational autoencoder model to obtain the weights to be used;

[0095] Step S35: Initialize the keyframe-based temporal compression encoder and the keyframe-based temporal compression decoder using the weights to be used.

[0096] Understandably, the initial video variational autoencoder model may also include a keyframe-based temporal compression decoder, which is used to convert the low-dimensional latent representation generated by the encoder back to the high-resolution video, that is, to reconstruct the video content from the low-dimensional latent space as close as possible to the original video, while maintaining the natural temporal logic of the video.

[0097] In this embodiment of the disclosure, to enable the video VAE model to achieve better temporal-spatial compression capabilities during initialization, a method is proposed to use image VAE weights with low potential channel counts as pre-trained weights. Furthermore, during the initialization phase, this disclosure allows for pre-training of the image variational autoencoder model, i.e., training a separate image VAE model to obtain the weights to be used. These weights are then used to initialize the keyframe-based temporal compression encoder and the keyframe-based temporal compression decoder.

[0098] This can be understood as follows: before training, the video VAE model uses the weights of the image VAE model as the initial weights of the video VAE model to provide initial spatial compression capability. Then, in subsequent training, it learns the complete spatial-temporal compression capability. In this disclosure, during the initialization phase, the 2D and 3D branches of the neural network are initialized using the weights of the pre-trained image VAE model with a low latent channel count. That is, the two branches use the weights of the image VAE model as initialization, enabling the neural network to independently process two different temporal information within a frame group to achieve an initial 2x temporal compression while maintaining strong image compression capability. This results in a more balanced and better temporal-spatial compression capability at the initial stage.

[0099] In an optional embodiment, the initial video variational autoencoder model includes: a parallel dilated convolution module, in step S322, performing parallel dilated convolution on the target compression coding result to obtain the target convolution result, including the following method steps:

[0100] Step S3221: Using a parallel dilated convolution module, dilated convolution is performed on the target compression coding result under multiple parallel dilation rates to obtain multiple dilated convolution results;

[0101] Step S3222: Perform feature concatenation on multiple dilated convolution results to obtain the concatenated result;

[0102] Step S3223: Perform two-dimensional convolution on the spliced ​​result according to the preset convolution kernel to obtain the target convolution result.

[0103] In this embodiment of the disclosure, the initial video variational autoencoder model includes a parallel dilated convolution module, which is used to expand the receptive field of the video VAE model to better capture long-range dependencies in the training video data.

[0104] In this embodiment of the disclosure, considering that for the same video, as the video resolution increases, the pixel span caused by object movement or camera movement also increases, and due to the limitation of the receptive field, it becomes more difficult for the video VAE model to capture video motion patterns, this disclosure introduces two improvements to increase the receptive field of the video VAE model.

[0105] Furthermore, when using a parallel dilated convolution module to perform parallel dilated convolution on the target compressed encoding result to obtain the target convolution result, the parallel dilated convolution module can be used to perform dilated convolution on the target compressed encoding result at multiple parallel levels with different dilation rates, resulting in multiple dilated convolution results. Then, the features of the multiple dilated convolution results are concatenated to obtain a concatenated result. Finally, a two-dimensional convolution is performed on the concatenated result according to a preset convolution kernel to obtain the target convolution result. For example, the preset convolution kernel can be a convolution kernel with a kernel size of 1, which is not limited here.

[0106] This disclosure can be understood as follows: First, multiple parallel dilated convolutions with different dilation rates are introduced into the last layer of the encoder to expand the receptive field. Then, the features generated by different branches are concatenated together. Finally, the concatenated result is processed by a convolution with a kernel size of 1 to adjust the number of channels. Thus, the receptive field of the network is expanded based on the parallel dilated convolution module, thereby promoting high-resolution temporal compression learning.

[0107] In addition, this disclosure increases the number of attention modules, and all attention modules are implemented with complete temporal and spatial compression, thereby reducing computational cost. This allows the global receptive field of the attention modules to effectively enhance the network's perception of temporal actions, especially for high-resolution, fast-moving videos.

[0108] It is understandable that when performing keyframe spatial compression decoding and overall temporal compression decoding on the target convolution result to obtain the reconstruction result, the introduction of parallel dilated convolution enables the decoder to better handle high-resolution videos and fast-moving scenes, thereby improving the quality of the reconstructed video.

[0109] In an optional embodiment, in step S33, the initial video variational autoencoder model is trained based on the training video data and the reconstruction results to generate the target video variational autoencoder model, including the following method steps:

[0110] Step S331: Determine the target loss based on the training video data and reconstruction results;

[0111] Step S332: Update the model parameters of the initial video variational autoencoder model based on the target loss to obtain the target video variational autoencoder model.

[0112] In this embodiment of the disclosure, when training an initial video variational autoencoder model based on training video data and reconstruction results to generate a target video variational autoencoder model, the target loss can be determined based on the training video data and reconstruction results, and then the model parameters of the initial video variational autoencoder model can be updated according to the target loss to obtain the target video variational autoencoder model.

[0113] This disclosure can be understood as follows: by designing a target loss function based on training video data and reconstruction results, and then training an initial video variational autoencoder model based on the obtained target loss, a target video variational autoencoder model is obtained. This ensures the learning effect of the video generation model, improves the quality of video reconstruction, significantly improves the efficiency and results of video generation tasks, and enables the target video variational autoencoder model to be applicable to various application scenarios, such as video compression, enhancement, restoration and style transfer.

[0114] Figure 4 is a schematic diagram of the model training method according to an embodiment of the present disclosure. As shown in Figure 4, the present disclosure mainly includes three basic modules, namely (a), (b), and (c) in Figure 4. Figure 4(a) represents the grouped causal convolution module, used to maintain causal relationships between different frame groups while allowing frame features within the same frame group to interact. Figure 4(b) represents the keyframe-based temporal compression unit, used to split the network into two dual-branch structures, 2D and 3D, to learn spatial compression of keyframes and overall temporal compression respectively. Figure 4(c) represents the parallel dilated convolution module, used to expand the network's receptive field to promote high-resolution temporal compression learning.

[0115] Furthermore, as shown in Figure 4(a), the time feature map F i Taking 0-8 as an example, the first step is to determine the time compression rate t. compress For the input frame (i.e., the temporal feature map F) i The frames are grouped, for example, into groups 0, 1-4, and 2-8. Then, causal logic padding is performed between frame groups, and regular 3D convolution is used on frames within the same group to make features visible and interactive between frames, thereby increasing the continuity of the reconstructed video and reducing flicker, resulting in a temporal feature map F. i+1 In other words, the grouped causal convolution module of this disclosure will maintain the causal logic by using causal padding between frame groups to ensure that the current frame group depends only on the previous frame group.

[0116] As shown in Figure 4(b), this disclosure constructs a dual-branch architecture for KTC, decoupling the compression of a frame group into spatial compression of keyframes and overall temporal motion compression. It can be seen that for input C... in The features input from each layer are fed into a 2D convolution (Conv2D) to extract image information and a grouped causal 3D convolution (GCConv3D) to extract overall temporal motion information. The features output from the two convolutions (i.e., C...) out / 2) After being normalized separately (e.g., Root Mean Square Normalization (RMSNorm)), they are concatenated together, and after passing through an activation function (e.g., SiLU activation function), they are used as the output C of the unit.out In Figure (b), “GCConv 3D” is the same as Figure (a).

[0117] It is important to note that in the initialization phase, the 2D and 3D branches of the network are initialized using VAE weights from pre-trained images with low latent channel counts. This allows the network to independently process two different temporal information within a frame group to achieve an initial doubling of temporal compression, while also possessing strong image compression capabilities. This initialization method achieves a more balanced and better temporal-spatial compression capability at the initial stage.

[0118] As shown in Figure 4(c), this disclosure first introduces multiple parallel dilated convolutions with different dilation rates in the last layer of the encoder to expand the receptive field. Features generated by different branches are concatenated and then subjected to a convolution with a kernel size of 1 to adjust the number of channels. Furthermore, this disclosure increases the number of attention modules from 2 to 7, and all attention modules are implemented with complete temporal and spatial compression to reduce computational cost. The global receptive field of the attention modules can effectively enhance the network's perception of temporal actions, especially for high-resolution, fast-moving videos.

[0119] As can be understood, the overall framework of the model training method proposed in this disclosure is shown in Figure 4(d). The network is based on the UNet architecture and uses the three basic modules proposed above. Specifically, the KTC module in Figure 4(d) is the KCT unit in Figure 4(b), and the PAC module in Figure 4(d) is the parallel dilated convolution in Figure 4(c). It can be seen that the input to the overall framework is a video segment (F, consisting of frames 0-8), and the output is the reconstructed video segment, i.e., the reconstruction result. The goal is for the reconstructed result to be as close as possible to, or even completely identical to, the input video segment. This disclosure uses MAE and LPIPS as training losses, and then updates the model parameters of the entire architecture based on these losses.

[0120] Furthermore, in Figure 4(d), the network decomposes the complete input video into spatial compression of keyframes and overall temporal compression. Keyframes are the darker-colored frames in the video sequence of the network graph, while the remaining frames are lighter-colored. The KTC unit has two branches that process these two parts respectively: 2D convolution learns the spatial compression of keyframes to obtain keyframes, and 3D convolution learns the overall temporal compression to obtain the temporal information of each frame group. First, a keyframe-based temporal compression encoder performs keyframe spatial compression encoding and overall temporal compression encoding on the input video to obtain the compressed encoding result. Then, a parallel dilated convolution module performs parallel dilated convolution on the target compressed encoding result to obtain the convolution result. Finally, a keyframe-based temporal compression decoder performs keyframe spatial compression decoding and overall temporal compression decoding on the target convolution result to obtain the reconstruction result. The final output reconstruction result is a combination of the results from both branches.

[0121] It is easy to understand that the beneficial effects of the model training method provided in this disclosure include the following points.

[0122] Beneficial effects (1) This disclosure proposes a keyframe-based temporal compression architecture that provides more balanced and powerful temporal-spatial compression capabilities during initialization, thereby accelerating the training convergence speed of video VAE models.

[0123] Beneficial effect (2): This disclosure proposes grouped causal convolution to replace conventional causal convolution in order to achieve better and more balanced performance between frames while maintaining causal logic.

[0124] Beneficial effect (3): This disclosure introduces dilated convolution and increases the number of attention modules to expand the receptive field of the model and promote temporal compression performance at high resolution.

[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0126] Furthermore, it should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solutions of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0128] According to an embodiment of this disclosure, a model training method is also provided, as shown in FIG5. FIG5 is a flowchart of a model training method according to an embodiment of this disclosure. As shown in FIG5, the method includes:

[0129] Step S51: Obtain reconstructed video data, wherein the reconstructed video data is generated by a target video variational autoencoder model, and the target video variational autoencoder model is obtained by performing any of the above-mentioned model training methods;

[0130] Step S52: The initial video generation model is trained using reconstructed video data to generate a target video generation model. The target video generation model is used to process the input multimedia reference data to output the target video.

[0131] In this embodiment of the disclosure, the reconstructed video data is the reconstructed video generated by the target video variational autoencoder model, which can be understood as the reconstruction result in the above embodiments. The target video variational autoencoder model is obtained by performing any of the above-described model training methods, which will not be elaborated here.

[0132] After acquiring the reconstructed video data, the initial video generation model is trained using the reconstructed video data to obtain the target video generation model. This target video generation model is used to process the input multimedia reference data to generate the target video. The multimedia reference data can be text prompts, image data, video data, audio data, etc., and is not limited here.

[0133] This can be understood as follows: the target video generation model can perform video generation processing on the input multimedia reference data, and by learning and imitating the characteristics of the multimedia reference data, create new target videos that have some visual or content association with the multimedia reference data and meet specific needs.

[0134] In this embodiment of the disclosure, reconstructed video data is acquired, wherein the reconstructed video data is generated by a target video variational autoencoder model, which is obtained by performing any of the model training methods described above. Then, the reconstructed video data is used to train an initial video generation model to generate a target video generation model. This target video generation model is used to perform video generation processing on the input multimedia reference data to output a target video. Therefore, a high-quality reconstructed video can be generated using the keyframe-based temporal compression architecture proposed in this disclosure, and a high-quality target video generation model can be generated based on this reconstructed video to train the initial video generation model.

[0135] The model training method provided in this disclosure can be applied, but is not limited to, to application scenarios involving the training of video variational autoencoder models or video generation models in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, video variational autoencoder model or video generation model training scenarios in the e-commerce service field, video variational autoencoder model or video generation model training scenarios in the education service field, and video variational autoencoder model or video generation model training scenarios in the legal service field, etc., without limitation.

[0136] By employing embodiments of this disclosure, reconstructed video data is acquired, which is generated by a target video variational autoencoder (VAE) model. The target VAE model is obtained by performing any of the aforementioned model training methods. Then, the reconstructed video data is used to train an initial video generation model to generate a target video generation model. The target video generation model is used to perform video generation processing on the input multimedia reference data to output a target video. This achieves the goal of generating high-quality reconstructed videos and training a high-quality video generation model based on the reconstructed videos. This realizes the proposed keyframe-based temporal compression architecture, thereby accelerating the convergence speed of video VAE model training, improving the performance of the video VAE model, and enhancing the video quality of the output reconstructed video. Furthermore, it achieves the technical effect of training a high-quality video generation model based on the reconstructed videos, thus solving the technical problems of low reconstruction result quality and slow optimization of video variational autoencoder models in related technologies.

[0137] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0138] According to an embodiment of this disclosure, a video processing method as shown in FIG6 is also provided. FIG6 is a flowchart of a video processing method according to an embodiment of this disclosure. As shown in FIG6, the method includes:

[0139] Step S61: Obtain multimedia reference data;

[0140] Step S62: The target video is generated by using a target video generation model to process the multimedia reference data to obtain the target video. The target video generation model is obtained by training the initial video generation model with reconstructed video data. The reconstructed video data is generated by the target video variational autoencoder model. The target video variational autoencoder model is obtained by performing any of the above-mentioned model training methods.

[0141] In this embodiment of the disclosure, the multimedia reference data may be text prompts, image data, video data, audio data, etc., and there is no limitation herein.

[0142] A target video generation model is used to process multimedia reference data to generate a target video, which is a video highly correlated with the multimedia reference data. The target video generation model is obtained by training an initial video generation model with reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder model, which is obtained by performing any of the model training methods described above. For details, please refer to the description of the foregoing embodiments; further elaboration is not provided here.

[0143] In this embodiment, multimedia reference data is acquired, and then a target video generation model is used to process the multimedia reference data to generate a target video. The target video generation model is obtained by training an initial video generation model with reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder model, which is obtained by executing any of the model training methods described above. Therefore, a high-quality reconstructed video can be generated using the keyframe-based temporal compression architecture proposed in this disclosure. Based on this reconstructed video, the initial video generation model is trained to generate a high-quality target video generation model. Furthermore, the multimedia reference data is processed using the generated target video generation model to obtain a high-quality target video.

[0144] The video processing method provided in this disclosure can be applied, but is not limited to, to video processing application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, video processing for e-commerce services, video processing for education services, video processing for legal services, etc., are not limited here.

[0145] By employing the embodiments of this disclosure, multimedia reference data is acquired, and then a target video generation model is used to process the multimedia reference data to generate a target video. The target video generation model is obtained by training an initial video generation model with reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder (VAE) model, which is obtained by executing any of the aforementioned model training methods. This achieves the goal of generating a high-quality target video based on the video generation model. Furthermore, the proposed keyframe-based temporal compression architecture accelerates the convergence speed of video VAE model training, improves the performance of the video VAE model, and enhances the video quality of the output reconstructed video. The high-quality video generation model is then trained based on the reconstructed video, and a high-quality target video is generated based on the video generation model. This solves the technical problems of low reconstruction quality and slow optimization of video variational autoencoder models in related technologies.

[0146] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0147] According to an embodiment of this disclosure, a video processing method as shown in FIG7 is also provided. FIG7 is a flowchart of a video processing method according to an embodiment of this disclosure. As shown in FIG7, the method includes:

[0148] Step S71: Obtain target product description data, wherein the target product description data includes at least one of the following: target product description text, target product video clip;

[0149] Step S72: The target product description data is processed by the target video generation model to generate a video, thereby obtaining a promotional video for the target product. The target video generation model is obtained by training the initial video generation model with reconstructed video data. The reconstructed video data is generated by the target video variational autoencoder model, which is obtained by performing any of the above-mentioned model training methods.

[0150] In this embodiment of the disclosure, the target product description data can be understood as data that describes the target product in multiple aspects. The target product description data includes at least one of the following: target product description text, target product video clips, and may also include target product description images, target product description audio, etc., which are not limited here.

[0151] A target video generation model is used to process the target product description data into a video, resulting in a promotional video for the target product. This promotional video is highly correlated with the target product description data. The target video generation model is obtained by training an initial video generation model with reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder model, which is obtained by performing any of the model training methods described above. For details, please refer to the description of the foregoing embodiments; further elaboration is omitted here.

[0152] In this embodiment, target product description data is acquired, and then a target video generation model is used to process the target product description data into a video to obtain a target product promotional video. The target video generation model is obtained by training an initial video generation model with reconstructed video data. The reconstructed video data is generated using a target video variational autoencoder model, which is obtained by executing any of the aforementioned model training methods. Therefore, a high-quality reconstructed video can be generated using the keyframe-based temporal compression architecture proposed in this disclosure. Based on this reconstructed video, the initial video generation model is trained to generate a high-quality target video generation model. Furthermore, the target product description data is processed using the generated target video generation model to obtain a high-quality target product promotional video.

[0153] The video processing method provided in this disclosure can be applied, but is not limited to, to video processing application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, video processing for e-commerce services, video processing for education services, video processing for legal services, etc., are not limited here.

[0154] By employing the embodiments of this disclosure, target product description data is obtained, and then a target video generation model is used to process the target product description data into a video, resulting in a target product promotional video. The target video generation model is obtained by training an initial video generation model with reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder (VAE) model, which is obtained by executing any of the aforementioned model training methods. This achieves the goal of generating a high-quality target product promotional video based on the video generation model. Furthermore, the proposed keyframe-based temporal compression architecture accelerates the convergence speed of the video VAE model training, improves the performance of the video VAE model, and enhances the video quality of the output reconstructed video. The high-quality video generation model is then trained based on the reconstructed video, and a high-quality target product promotional video is generated based on the video generation model. This solves the technical problems of low reconstruction quality and slow optimization of video variational autoencoder models in related technologies.

[0155] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0156] According to embodiments of this disclosure, a video processing method is also provided, as shown in FIG8. FIG8 is a flowchart of a video processing method according to an embodiment of this disclosure. As shown in FIG8, the method includes:

[0157] Step S81: Obtain a video processing request through the first application programming interface, wherein the request data carried in the video processing request includes: multimedia reference data;

[0158] Step S82: Return a video processing response through the second application programming interface. The response data carried in the video processing response includes: a target video, which is obtained by performing video generation processing on the multimedia reference data using a target video generation model. The target video generation model is obtained by training an initial video generation model using reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder model. The target video variational autoencoder model is obtained by executing any of the above-mentioned model training methods.

[0159] The first and second application programming interfaces (APIs) mentioned above can be the same or different APIs. In one optional embodiment, the interface parameters in the first and second APIs may include, but are not limited to: a global interface identifier, an interface signing key, an interface timestamp, an interface request identifier, and a system call credential identifier. The first API can use GET or POST as the interface request method to obtain the file processing request. The second API can use JSON format to return the file processing response.

[0160] In this embodiment of the disclosure, a video processing request can be understood as a request to generate a corresponding target video based on the multimedia reference data carried, and a video processing response can be understood as a response to the video processing request, wherein the response data carried in the video processing response includes: the target video.

[0161] The target video is obtained by processing the multimedia reference data using a target video generation model. The target video generation model is obtained by training the initial video generation model using reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder model. The target video variational autoencoder model is obtained by performing any of the above-mentioned model training methods, which will not be elaborated here.

[0162] The video processing method provided in this disclosure can be applied, but is not limited to, to video processing application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, video processing for e-commerce services, video processing for education services, video processing for legal services, etc., are not limited here.

[0163] In this embodiment, a video processing request is obtained through a first application programming interface (API), wherein the request data carried in the video processing request includes multimedia reference data. Then, a video processing response is returned through a second API, wherein the response data carried in the video processing response includes a target video. The target video is obtained by performing video generation processing on the multimedia reference data using a target video generation model. The target video generation model is obtained by training an initial video generation model using reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder (VAE) model, which is obtained by executing any of the above-described model training methods. This achieves the goal of generating high-quality target product promotional videos based on a video generation model. It realizes the proposed keyframe-based temporal compression architecture, thereby accelerating the convergence speed of video VAE model training, improving the performance of the video VAE model, and enhancing the video quality of the output reconstructed video. Furthermore, it trains a high-quality video generation model based on the reconstructed video, and then generates high-quality target product promotional videos based on the video generation model. This solves the technical problems of low reconstruction result quality and slow optimization of video variational autoencoder models in related technologies.

[0164] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0165] According to an embodiment of this disclosure, a video processing method is also provided, as shown in FIG9. FIG9 is a flowchart of a video processing method according to an embodiment of this disclosure. As shown in FIG9, the method includes:

[0166] Step S91: Obtain the currently input video processing dialogue request, wherein the request data carried in the video processing dialogue request includes: multimedia reference data;

[0167] Step S92, in response to the video processing dialogue request, return a video processing dialogue response, wherein the information carried in the video processing dialogue response includes: the target video, which is obtained by performing video generation processing on the multimedia reference data using a target video generation model, the target video generation model is obtained by training an initial video generation model using reconstructed video data, the reconstructed video data is generated by a target video variational autoencoder model, and the target video variational autoencoder model is obtained by performing any of the above-mentioned model training methods;

[0168] Step S93: Play the target video within the graphical user interface.

[0169] In this embodiment of the disclosure, the video processing dialogue request can be understood as a dialogue request initiated by a user when engaging in dialogue with artificial intelligence. This video processing dialogue request is used to request video processing based on carried multimedia reference data to obtain the corresponding target video. The video processing dialogue response is the content replying to the video processing dialogue request, and the information carried in the video processing dialogue response includes: the target video.

[0170] The target video is obtained by processing the multimedia reference data using a target video generation model. The target video generation model is obtained by training the initial video generation model using reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder model. The target video variational autoencoder model is obtained by performing any of the above-mentioned model training methods, which will not be elaborated here.

[0171] After returning to the target video, it can be played within the graphical user interface to provide feedback to the user.

[0172] The video processing method provided in this disclosure can be applied, but is not limited to, to video processing application scenarios in fields such as e-commerce services, education services, legal services, medical services, conference services, social networking services, financial product services, logistics services, and navigation services. For example, video processing for e-commerce services, video processing for education services, video processing for legal services, etc., are not limited here.

[0173] By employing the embodiments of this disclosure, a video processing dialogue request is obtained from the current input, wherein the request data carried in the video processing dialogue request includes multimedia reference data. Then, in response to the video processing dialogue request, a video processing dialogue response is returned, wherein the information carried in the video processing dialogue response includes: a target video. The target video is obtained by performing video generation processing on the multimedia reference data using a target video generation model. The target video generation model is obtained by training an initial video generation model using reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder model, which is obtained by executing any of the above-described model training methods. Finally, the target video is played within a graphical user interface. This achieves the goal of generating a high-quality target product promotional video based on a video generation model. It also realizes the proposed keyframe-based temporal compression architecture, thereby accelerating the convergence speed of video VAE model training, improving the performance of the video VAE model, and improving the video quality of the output reconstructed video. Furthermore, it trains a high-quality video generation model based on the reconstructed video, and then generates a high-quality target product promotional video based on the video generation model. This solves the technical problems of low reconstruction result quality and slow optimization of video variational autoencoder models in related technologies.

[0174] It should be noted that the preferred implementation of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0175] According to an embodiment of this disclosure, an apparatus embodiment for implementing the above-described model training method is also provided. FIG10 is a schematic structural diagram of a model training apparatus according to an embodiment of this disclosure. As shown in FIG10, the apparatus includes:

[0176] The first acquisition module 1001 is configured to acquire training video data;

[0177] The first processing module 1002 is configured to use an initial video variational autoencoder model to perform video reconstruction processing on the training video data to obtain the reconstruction result. The initial video variational autoencoder model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data to complete the video reconstruction.

[0178] The first training module 1003 is configured to train the initial video variational autoencoder model based on the training video data and reconstruction results to generate a target video variational autoencoder model. The target video variational autoencoder model is used to provide reconstructed video data for the initial video generation model to train and generate the target video generation model. The target video generation model is used to perform video generation processing on the input multimedia reference data to output the target video.

[0179] Optionally, the first processing module 1002 is further configured to: perform keyframe spatial compression coding and overall temporal compression coding on the training video data to obtain the target compression coding result; perform parallel dilated convolution on the target compression coding result to obtain the target convolution result; and perform keyframe spatial compression decoding and overall temporal compression decoding on the target convolution result to obtain the reconstruction result.

[0180] Optionally, the initial video variational autoencoder model includes a keyframe-based temporal compression encoder. The first processing module 1002 is further configured to: perform keyframe spatial compression encoding on the training video data using the keyframe-based temporal compression encoder to obtain a spatial compression encoding result; perform overall temporal compression encoding on the training video data to obtain a temporal compression encoding result; normalize the spatial compression encoding result to obtain a first processing result; normalize the temporal compression encoding result to obtain a second processing result; and concatenate the first processing result and the second processing result to obtain the target compression encoding result.

[0181] Optionally, the keyframe-based temporal compression encoder includes a two-dimensional convolutional unit and a grouped causal convolutional unit. The first processing module 1002 is further configured to: use the two-dimensional convolutional unit to perform keyframe spatial compression encoding on the training video data to obtain a spatial compression encoding result; and use the grouped causal convolutional unit to perform overall temporal compression encoding on the training video data to obtain a temporal compression encoding result.

[0182] Optionally, the first processing module 1002 is further configured to: perform three-dimensional convolution on multiple video frames located in the same video frame group using a grouped causal convolution unit, and perform causal padding between different video frame groups in the multiple video frame groups to obtain a temporal compression coding result.

[0183] Optionally, the device further includes: an initialization module configured to pre-train an image variational autoencoder model to obtain weights to be used; and to initialize a keyframe-based temporal compression encoder and a keyframe-based temporal compression decoder using the weights to be used.

[0184] Optionally, the initial video variational autoencoder model includes a parallel dilated convolution module. The first processing module 1002 is further configured to: use the parallel dilated convolution module to perform dilated convolution on the target compression coding result at multiple parallel dilation rates to obtain multiple dilated convolution results; perform feature concatenation on the multiple dilated convolution results to obtain a concatenation result; and perform two-dimensional convolution on the concatenation result according to a preset convolution kernel to obtain the target convolution result.

[0185] Optionally, the first training module 1003 is further configured to: determine the target loss based on the training video data and reconstruction results; update the model parameters of the initial video variational autoencoder model according to the target loss to obtain the target video variational autoencoder model.

[0186] Optionally, the initial video variational autoencoder model is used to split the video sequence corresponding to the training video data into multiple video frame groups, wherein the video frames contained in the multiple video frame groups are non-repeating, and the compression process of each video frame group in the multiple video frame groups is decoupled into keyframe spatial compression and overall temporal compression, respectively.

[0187] By employing the embodiments of this disclosure, training video data is acquired, and then an initial video variational autoencoder (VAE) model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data, i.e., video reconstruction processing is performed on the training video data to obtain reconstruction results. Subsequently, the initial VAE model is trained based on the training video data and reconstruction results to generate a target VAE model. The target VAE model is used to provide reconstructed video data for the initial video generation model to train and generate the target video generation model. The target video generation model is used to perform video generation processing on the input multimedia reference data to output the target video. This achieves the goal of generating high-quality reconstructed video, thereby realizing the technical effect of accelerating the convergence speed of video VAE model training, improving the performance of video VAE model, and improving the video quality of the output reconstructed video through the proposed keyframe-based temporal compression architecture. This solves the technical problems of low reconstruction result quality and slow optimization of video variational autoencoder models in related technologies.

[0188] It should be noted that the first acquisition module 1001, the first processing module 1002, and the first training module 1003 mentioned above correspond to steps S31 to S33 in the embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units may be hardware or software components stored in memory and processed by one or more processors, and the above modules may also run in the server 10 provided in the embodiments.

[0189] According to embodiments of this disclosure, another apparatus embodiment for implementing the above-described model training method is also provided. Figure 11 is a schematic structural diagram of another model training apparatus according to an embodiment of this disclosure. As shown in Figure 11, the apparatus includes:

[0190] The second acquisition module 1101 is configured to acquire reconstructed video data, wherein the reconstructed video data is generated by a target video variational autoencoder model, and the target video variational autoencoder model is obtained by performing any of the above-mentioned model training methods.

[0191] The second training module 1102 is configured to train the initial video generation model using reconstructed video data to generate a target video generation model, wherein the target video generation model is used to perform video generation processing on the input multimedia reference data to output the target video.

[0192] By employing embodiments of this disclosure, reconstructed video data is acquired, which is generated by a target video variational autoencoder (VAE) model. The target VAE model is obtained by performing any of the aforementioned model training methods. Then, the reconstructed video data is used to train an initial video generation model to generate a target video generation model. The target video generation model is used to perform video generation processing on the input multimedia reference data to output a target video. This achieves the goal of generating high-quality reconstructed videos and training a high-quality video generation model based on the reconstructed videos. This realizes the proposed keyframe-based temporal compression architecture, thereby accelerating the convergence speed of video VAE model training, improving the performance of the video VAE model, and enhancing the video quality of the output reconstructed video. Furthermore, it achieves the technical effect of training a high-quality video generation model based on the reconstructed videos, thus solving the technical problems of low reconstruction result quality and slow optimization of video variational autoencoder models in related technologies.

[0193] It should be noted that the second acquisition module 1101 and the second training module 1102 mentioned above correspond to steps S51 and S52 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0194] According to embodiments of this disclosure, an apparatus embodiment for implementing the above-described video processing method is also provided. FIG12 is a schematic structural diagram of another video processing apparatus according to an embodiment of this disclosure. As shown in FIG12, the apparatus includes:

[0195] The third acquisition module 1201 is configured to acquire multimedia reference data;

[0196] The second processing module 1202 is configured to use a target video generation model to perform video generation processing on the multimedia reference data to obtain a target video; wherein, the target video generation model is obtained by training an initial video generation model with reconstructed video data, the reconstructed video data is generated by a target video variational autoencoder model, and the target video variational autoencoder model is obtained by executing any of the above-mentioned model training methods.

[0197] By employing the embodiments of this disclosure, multimedia reference data is acquired, and then a target video generation model is used to process the multimedia reference data to generate a target video. The target video generation model is obtained by training an initial video generation model with reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder (VAE) model, which is obtained by executing any of the aforementioned model training methods. This achieves the goal of generating a high-quality target video based on the video generation model. Furthermore, the proposed keyframe-based temporal compression architecture accelerates the convergence speed of video VAE model training, improves the performance of the video VAE model, and enhances the video quality of the output reconstructed video. The high-quality video generation model is then trained based on the reconstructed video, and a high-quality target video is generated based on the video generation model. This solves the technical problems of low reconstruction quality and slow optimization of video variational autoencoder models in related technologies.

[0198] It should be noted that the third acquisition module 1201 and the second processing module 1202 mentioned above correspond to steps S61 and S62 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0199] According to embodiments of this disclosure, another apparatus embodiment for implementing the above-described video processing method is also provided. FIG13 is a schematic structural diagram of another video processing apparatus according to an embodiment of this disclosure. As shown in FIG13, the apparatus includes:

[0200] The fourth acquisition module 1301 is configured to acquire target product description data, wherein the target product description data includes at least one of the following: target product description text, target product video clip;

[0201] The third processing module 1302 is configured to use a target video generation model to process the target product description data into a video to obtain a target product promotional video. The target video generation model is obtained by training an initial video generation model with reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder model, which is obtained by executing any of the above-mentioned model training methods.

[0202] By employing the embodiments of this disclosure, target product description data is obtained, and then a target video generation model is used to process the target product description data into a video, resulting in a target product promotional video. The target video generation model is obtained by training an initial video generation model with reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder (VAE) model, which is obtained by executing any of the aforementioned model training methods. This achieves the goal of generating a high-quality target product promotional video based on the video generation model. Furthermore, the proposed keyframe-based temporal compression architecture accelerates the convergence speed of the video VAE model training, improves the performance of the video VAE model, and enhances the video quality of the output reconstructed video. The high-quality video generation model is then trained based on the reconstructed video, and a high-quality target product promotional video is generated based on the video generation model. This solves the technical problems of low reconstruction quality and slow optimization of video variational autoencoder models in related technologies.

[0203] It should be noted that the fourth acquisition module 1301 and the third processing module 1302 mentioned above correspond to steps S71 and S72 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0204] According to embodiments of this disclosure, another apparatus embodiment for implementing the above-described video processing method is also provided. FIG14 is a schematic structural diagram of another video processing apparatus according to an embodiment of this disclosure. As shown in FIG14, the apparatus includes:

[0205] The fifth acquisition module 1401 is configured to acquire a video processing request through the first application programming interface, wherein the request data carried in the video processing request includes: multimedia reference data;

[0206] The first return module 1402 is configured to return a video processing response via a second application programming interface. The response data carried in the video processing response includes: a target video, which is obtained by performing video generation processing on multimedia reference data using a target video generation model. The target video generation model is obtained by training an initial video generation model using reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder model, which is obtained by executing any of the above-mentioned model training methods.

[0207] In this embodiment, a video processing request is obtained through a first application programming interface (API), wherein the request data carried in the video processing request includes multimedia reference data. Then, a video processing response is returned through a second API, wherein the response data carried in the video processing response includes a target video. The target video is obtained by performing video generation processing on the multimedia reference data using a target video generation model. The target video generation model is obtained by training an initial video generation model using reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder (VAE) model, which is obtained by executing any of the above-described model training methods. This achieves the goal of generating high-quality target product promotional videos based on a video generation model. It realizes the proposed keyframe-based temporal compression architecture, thereby accelerating the convergence speed of video VAE model training, improving the performance of the video VAE model, and enhancing the video quality of the output reconstructed video. Furthermore, it trains a high-quality video generation model based on the reconstructed video, and then generates high-quality target product promotional videos based on the video generation model. This solves the technical problems of low reconstruction result quality and slow optimization of video variational autoencoder models in related technologies.

[0208] It should be noted that the fifth acquisition module 1401 and the first return module 1402 mentioned above correspond to steps S81 and S82 in the embodiments. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0209] According to embodiments of this disclosure, another apparatus embodiment for implementing the above-described video processing method is also provided. FIG15 is a schematic structural diagram of another video processing apparatus according to an embodiment of this disclosure. As shown in FIG15, the apparatus includes:

[0210] The sixth acquisition module 1501 is configured to acquire the currently input video processing dialogue request, wherein the request data carried in the video processing dialogue request includes: multimedia reference data;

[0211] The second return module 1502 is configured to return a video processing dialogue response in response to a video processing dialogue request. The information carried in the video processing dialogue response includes: a target video, which is obtained by performing video generation processing on multimedia reference data using a target video generation model, the target video generation model is obtained by training an initial video generation model using reconstructed video data, the reconstructed video data is generated by a target video variational autoencoder model, and the target video variational autoencoder model is obtained by performing any of the above-mentioned model training methods.

[0212] Playback module 1503 is configured to play the target video within the graphical user interface.

[0213] By employing the embodiments of this disclosure, a video processing dialogue request is obtained from the current input, wherein the request data carried in the video processing dialogue request includes multimedia reference data. Then, in response to the video processing dialogue request, a video processing dialogue response is returned, wherein the information carried in the video processing dialogue response includes: a target video. The target video is obtained by performing video generation processing on the multimedia reference data using a target video generation model. The target video generation model is obtained by training an initial video generation model using reconstructed video data. The reconstructed video data is generated by a target video variational autoencoder model, which is obtained by executing any of the above-described model training methods. Finally, the target video is played within a graphical user interface. This achieves the goal of generating a high-quality target product promotional video based on a video generation model. It also realizes the proposed keyframe-based temporal compression architecture, thereby accelerating the convergence speed of video VAE model training, improving the performance of the video VAE model, and improving the video quality of the output reconstructed video. Furthermore, it trains a high-quality video generation model based on the reconstructed video, and then generates a high-quality target product promotional video based on the video generation model. This solves the technical problems of low reconstruction result quality and slow optimization of video variational autoencoder models in related technologies.

[0214] It should be noted that the sixth acquisition module 1501, the second return module 1502, and the playback module 1503 mentioned above correspond to steps S91 to S93 in the embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also run in the server 10 provided in the embodiments.

[0215] It should be noted that the preferred implementation schemes involved in the above embodiments of this disclosure are the same as the schemes, application scenarios and implementation processes provided in the embodiments, but are not limited to the schemes provided in the embodiments.

[0216] Embodiments of this disclosure can provide an electronic device, which can be any one of a group of electronic devices. Optionally, in this embodiment, the electronic device can also be replaced by a terminal device such as a mobile terminal. Optionally, in this embodiment, the electronic device can be located in at least one of a plurality of network devices in a computer network.

[0217] In this embodiment, the aforementioned electronic device can execute the program code for the following steps in the model training method: acquiring training video data; performing video reconstruction processing on the training video data using an initial video variational autoencoder model to obtain reconstruction results, wherein the initial video variational autoencoder model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data to complete video reconstruction; training the initial video variational autoencoder model based on the training video data and reconstruction results to generate a target video variational autoencoder model, wherein the target video variational autoencoder model is used to provide reconstructed video data for the initial video generation model to train and generate the target video generation model, and the target video generation model is used to perform video generation processing on the input multimedia reference data to output the target video.

[0218] Optionally, FIG16 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG16, taking electronic device A as an example, electronic device A may include: one or more (only one is shown in the figure) processors 1602, memory 1604, memory controller, and peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0219] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the model training method and apparatus in this embodiment. The processor executes various functional applications and data processing by running the stored software programs and modules, thereby realizing the aforementioned model training method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to electronic device A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0220] The processor can access information and applications stored in memory via a transmission device to perform the following steps: acquiring training video data; performing video reconstruction processing on the training video data using an initial video variational autoencoder model to obtain reconstruction results, wherein the initial video variational autoencoder model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data to complete video reconstruction; training the initial video variational autoencoder model based on the training video data and reconstruction results to generate a target video variational autoencoder model, wherein the target video variational autoencoder model is used to provide reconstructed video data for the initial video generation model to train and generate the target video generation model, and the target video generation model is used to perform video generation processing on the input multimedia reference data to output the target video.

[0221] By employing the embodiments of this disclosure, training video data is acquired, and then an initial video variational autoencoder (VAE) model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data, i.e., video reconstruction processing is performed on the training video data to obtain reconstruction results. Subsequently, the initial VAE model is trained based on the training video data and reconstruction results to generate a target VAE model. The target VAE model is used to provide reconstructed video data for the initial video generation model to train and generate the target video generation model. The target video generation model is used to perform video generation processing on the input multimedia reference data to output the target video. This achieves the goal of generating high-quality reconstructed video, thereby realizing the technical effect of accelerating the convergence speed of video VAE model training, improving the performance of video VAE model, and improving the video quality of the output reconstructed video through the proposed keyframe-based temporal compression architecture. This solves the technical problems of low reconstruction result quality and slow optimization of video variational autoencoder models in related technologies.

[0222] Those skilled in the art will understand that the structure shown in Figure 16 is merely illustrative, and electronic device A can also be a smartphone, tablet computer, PDA, mobile internet device (MID), PAD, or other terminal device. Figure 16 does not limit the structure of the aforementioned electronic device. For example, electronic device A may include more or fewer components (such as network interfaces, display devices, etc.) than shown in Figure 16, or have a different configuration than shown in Figure 16.

[0223] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0224] Embodiments of this disclosure also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the model training method provided in Embodiment 1.

[0225] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0226] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring training video data; performing video reconstruction processing on the training video data using an initial video variational autoencoder model to obtain a reconstruction result, wherein the initial video variational autoencoder model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data to complete video reconstruction; training the initial video variational autoencoder model based on the training video data and the reconstruction result to generate a target video variational autoencoder model, wherein the target video variational autoencoder model is used to provide reconstructed video data for the initial video generation model to train and generate the target video generation model, and the target video generation model is used to perform video generation processing on the input multimedia reference data to output a target video.

[0227] Embodiments of this disclosure also provide a computer program product comprising a computer program that, when executed by a processor, implements any of the above-described model training methods.

[0228] Optionally, in this embodiment, when the computer program product is executed by the processor, the program code is used to perform the following steps: acquiring training video data; performing video reconstruction processing on the training video data using an initial video variational autoencoder model to obtain a reconstruction result, wherein the initial video variational autoencoder model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data to complete video reconstruction; training the initial video variational autoencoder model based on the training video data and the reconstruction result to generate a target video variational autoencoder model, wherein the target video variational autoencoder model is used to provide reconstructed video data for the initial video generation model to train and generate the target video generation model, and the target video generation model is used to perform video generation processing on the input multimedia reference data to output a target video.

[0229] The sequence numbers of the embodiments disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0230] In the above embodiments of this disclosure, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0231] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0232] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0233] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0234] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0235] The above description is only a preferred embodiment of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of this disclosure, and these improvements and modifications should also be considered within the scope of protection of this disclosure.

Claims

1. A model training method, comprising: Acquire training video data; The training video data is processed by an initial video variational autoencoder model to obtain the reconstruction result. The initial video variational autoencoder model is used to perform keyframe spatial compression and overall temporal compression on multiple video frame groups in the training video data to complete the video reconstruction. The initial video variational autoencoder model is trained based on the training video data and the reconstruction results to generate a target video variational autoencoder model. The target video variational autoencoder model is used to provide reconstructed video data for the initial video generation model to train and generate the target video generation model. The target video generation model is used to perform video generation processing on the input multimedia reference data to output the target video.

2. The model training method according to claim 1, wherein, The initial video variational autoencoder model is used to perform video reconstruction processing on the training video data, and the reconstruction results include: The training video data is subjected to keyframe spatial compression coding and overall temporal compression coding to obtain the target compression coding result; Parallel dilated convolution is performed on the target compression encoding result to obtain the target convolution result; The target convolution result is subjected to keyframe spatial compression decoding and overall temporal compression decoding to obtain the reconstruction result.

3. The model training method according to claim 2, wherein, The initial video variational autoencoder model includes: a keyframe-based temporal compression encoder, which performs keyframe spatial compression coding and overall temporal compression coding on the training video data to obtain the target compression coding result, including: The training video data is spatially compressed using the keyframe-based temporal compression encoder to obtain a spatial compression encoding result, and the training video data is temporally compressed using the overall temporal compression encoding to obtain a temporal compression encoding result. The spatial compression coding result is normalized to obtain a first processing result, and the temporal compression coding result is normalized to obtain a second processing result. The first processing result and the second processing result are concatenated to obtain the target compression encoding result.

4. The model training method according to claim 3, wherein, The keyframe-based temporal compression encoder includes: a two-dimensional convolutional unit and a grouped causal convolutional unit. The keyframe-based temporal compression encoder is used to perform keyframe spatial compression encoding on the training video data to obtain the spatial compression encoding result, and to perform overall temporal compression encoding on the training video data to obtain the temporal compression encoding result, including: The training video data is spatially compressed and encoded using the two-dimensional convolutional unit to obtain the spatial compression encoding result. The training video data is subjected to overall temporal compression coding using the grouped causal convolutional unit to obtain the temporal compression coding result.

5. The model training method according to claim 4, wherein, The training video data is subjected to overall temporal compression coding using the grouped causal convolutional unit, and the temporal compression coding result includes: The grouped causal convolution unit is used to perform three-dimensional convolution on multiple video frames located in the same video frame group, and causal padding is performed between different video frame groups in the multiple video frame groups to obtain the temporal compression coding result.

6. The model training method according to claim 2, wherein, The model training method also includes: The image variational autoencoder model is pre-trained to obtain the weights to be used; The keyframe-based temporal compression encoder and keyframe-based temporal compression decoder are initialized using the weights to be used.

7. The model training method according to claim 2, wherein, The initial video variational autoencoder model includes a parallel dilated convolution module, which performs parallel dilated convolution on the target compression coding result to obtain the target convolution result, including: The parallel dilated convolution module is used to perform dilated convolution on the target compression coding result under multiple parallel dilation rates to obtain multiple dilated convolution results; The multiple dilated convolution results are concatenated to obtain a concatenated result. The spliced ​​result is subjected to two-dimensional convolution according to a preset convolution kernel to obtain the target convolution result.

8. The model training method according to any one of claims 1-7, wherein, Training the initial video variational autoencoder model based on the training video data and the reconstruction results, and generating the target video variational autoencoder model includes: The target loss is determined based on the training video data and the reconstruction results; The model parameters of the initial video variational autoencoder model are updated based on the target loss to obtain the target video variational autoencoder model.

9. The model training method according to any one of claims 1-7, wherein, The initial video variational autoencoder model is used to split the video sequence corresponding to the training video data into multiple video frame groups, wherein the video frames contained in the multiple video frame groups are non-repeating, and the compression process of each video frame group in the multiple video frame groups is decoupled into keyframe spatial compression and overall temporal compression.

10. A model training method, comprising: Acquire reconstructed video data, wherein the reconstructed video data is generated by a target video variational autoencoder model, and the target video variational autoencoder model is obtained by performing the model training method according to any one of claims 1 to 9; The reconstructed video data is used to train the initial video generation model to generate a target video generation model, wherein the target video generation model is used to perform video generation processing on the input multimedia reference data to output the target video.

11. A video processing method, comprising: Obtain multimedia reference data; The multimedia reference data is processed using a target video generation model to generate the target video. The target video generation model is obtained by training the initial video generation model with reconstructed video data. The reconstructed video data is generated by the target video variational autoencoder model, which is obtained by performing the model training method described in any one of claims 1 to 9.

12. A video processing method, comprising: Obtain target product description data, wherein the target product description data includes at least one of the following: target product description text, target product video clip; The target product description data is processed using a target video generation model to generate a promotional video for the target product. The target video generation model is obtained by training the initial video generation model with reconstructed video data. The reconstructed video data is generated by the target video variational autoencoder model, which is obtained by performing the model training method described in any one of claims 1 to 8.

13. A video processing method, comprising: A video processing request is obtained through a first application programming interface, wherein the request data carried in the video processing request includes: multimedia reference data; A video processing response is returned through a second application programming interface, wherein the response data carried in the video processing response includes: a target video, which is obtained by performing video generation processing on the multimedia reference data using a target video generation model, the target video generation model being obtained by training an initial video generation model using reconstructed video data, the reconstructed video data being generated by a target video variational autoencoder model, and the target video variational autoencoder model being obtained by executing the model training method described in any one of claims 1 to 9.

14. A video processing method, comprising: Obtain the currently input video processing dialogue request, wherein the request data carried in the video processing dialogue request includes: multimedia reference data; In response to the video processing dialogue request, a video processing dialogue response is returned, wherein the information carried in the video processing dialogue response includes: a target video, which is obtained by performing video generation processing on the multimedia reference data using a target video generation model, the target video generation model is obtained by training an initial video generation model using reconstructed video data, the reconstructed video data is generated by a target video variational autoencoder model, and the target video variational autoencoder model is obtained by performing the model training method according to any one of claims 1 to 9; Play the target video within the graphical user interface.

15. An electronic device comprising: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the model training method according to any one of claims 1 to 10 or the video processing method according to any one of claims 11 to 14.

16. A computer-readable storage medium comprising a stored executable program, wherein, When the executable program is executed, it controls the device containing the computer-readable storage medium to perform the model training method according to any one of claims 1 to 10 or the video processing method according to any one of claims 11 to 14.

17. A computer program product comprising a computer program that, when executed by a processor, implements the model training method of any one of claims 1 to 10 or the video processing method of any one of claims 11 to 14.

Images