Data generation method and apparatus, chip, device, and computer program product

Abstract

Embodiments of the present application relate to a data generation method and apparatus, a chip, a device, and a computer program product. The method comprises: acquiring a compression ratio for a set of input vectors, and determining a plurality of sets of similar vectors in the set of input vectors on the basis of the compression ratio. The method further comprises: generating a first intermediate vector for a first set of similar vectors among the plurality of sets of similar vectors. Thus, a set of fewer intermediate vectors adapted to the compression ratio can be generated, thereby substantially compressing the data amount of the input vectors, and helping to increase the execution speed of subsequent operations.

Description

Data generation methods, apparatuses, chips, devices, and computer program products

[0001] This application claims priority to Chinese Patent Application No. 202411732068.9, filed on November 27, 2024, entitled “Data Generation Method, Apparatus, Chip, Device and Computer Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of this application generally relate to processing techniques, and more specifically, to data generation methods, apparatus, chips, devices, and computer program products. Background Technology

[0003] By integrating and processing data from diverse sources (such as text, images, and audio), multimodal large models provide generative AI tools with powerful content creation and understanding capabilities. This enables them to generate more effective, diverse, and high-quality AI-generated content (AIGC), thereby pushing the boundaries of creative and personalized content generation. Among these technologies, text-to-image or video generation (also known as text-to-image or video technology) expands human creative expression by transforming textual descriptions into visual images or videos, revolutionizing multiple fields such as art, education, and entertainment, while also promoting the development of human-computer interaction and the digital content industry.

[0004] With advancements in natural language processing and computer vision technologies, text-based image and video generation techniques have achieved significant breakthroughs. Various models, including Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Diffusion Models (DMs), have been applied in this field. Currently, DM has become one of the mainstream paradigms for text-based image and video generation due to its ability to effectively generate high-quality, high-resolution images or videos from text descriptions, while also offering diversity and controllability. Summary of the Invention

[0005] The embodiments of this application provide a data generation scheme.

[0006] In a first aspect, a data generation method is provided. The method includes: obtaining a compression ratio for a set of input vectors; determining multiple sets of similar vectors within the set of input vectors based on the compression ratio; and generating a first intermediate vector for the first set of similar vectors among the multiple sets of similar vectors. Thus, a smaller set of intermediate vectors adapted to the compression ratio can be generated, thereby substantially compressing the amount of data in the input vectors and helping to improve the execution speed of subsequent operations.

[0007] In some embodiments of the first aspect, a set of input vectors is the input to the attention layer of the generative model at the corresponding time step of execution of the generative model, and obtaining the compression ratio includes: determining the number of vectors in the set of input vectors that make a threshold contribution to the output of the attention layer; and determining the ratio of this number to the total number of the set of input vectors as the compression ratio. When compressing the input to the attention layer, the compression ratio can be adaptively adjusted in a denoising-aware manner, thereby improving execution speed while ensuring output accuracy.

[0008] In some embodiments of the first aspect, determining the number of vectors in a set of input vectors that contribute a threshold to the output of the attention layer includes determining this number based on the attention score heatmap of the attention layer at the corresponding time step. Thus, the minimum number of vectors that retain the required amount of key information during compression can be determined. Performing compression on a set of input vectors based on the compression ratio determined therein enables the final output generated by subsequent operations to achieve the desired accuracy.

[0009] In some embodiments of the first aspect, determining multiple sets of similar vectors includes: obtaining downsampled vectors from multiple layers by downsampling a set of input vectors multiple times; and determining multiple sets of similar vectors based on the downsampled vectors from multiple layers and a similarity threshold, wherein the similarity threshold is determined based on the compression ratio. This provides a fast way to determine and label input vectors, significantly reducing computational overhead and accelerating model inference compared to similar data determination methods that require traversing all data.

[0010] In some embodiments of the first aspect, determining multiple sets of similar vectors based on multiple layers and a similarity threshold includes: based on the similarity threshold, for a first vector in a set of input vectors, determining a set of vectors in the last layer of multiple layers that are similar to the first vector. Thus, the range of vectors similar to the vector to be matched can be determined at the layer with the lowest precision, serving as the basis for progressively narrowing the search range for similar vectors.

[0011] In some embodiments of the first aspect, determining multiple sets of similar vectors based on multiple layers and a similarity threshold further includes: for each layer except the last layer, obtaining a first set of vectors similar to the first vector in the next layer of that layer; determining a second set of vectors corresponding to the first set of vectors in that layer; and determining a third set of vectors similar to the first vector in the second set of vectors based on the similarity threshold. Thus, the search range of vectors similar to the vector to be matched can be passed layer by layer and further narrowed in the previous layer, thereby avoiding traversing all original input vectors and improving the matching speed of similar vectors.

[0012] In some embodiments of the first aspect, determining the third set of vectors includes: determining the second set of vectors as the third set of vectors in response to a resolution corresponding to the layer being greater than a resolution threshold. This can, under certain conditions, further improve execution speed and avoid offsetting the benefits by avoiding the computational overhead of similarity determination.

[0013] In some embodiments of the first aspect, a vector in a set of input vectors corresponds to a set of patches in an image sequence to be generated by an image generator, and obtaining the downsampled vectors includes downsampling the set of input vectors in both the temporal and spatial domains. This makes the embodiments of this application applicable to spatiotemporal image data.

[0014] In some embodiments of the first aspect, the method further includes: generating a first set of output vectors for the first set of similar vectors based on the first intermediate vector and grouping information for multiple sets of similar vectors. Thus, by performing the operation to be performed on the input vector only once on the intermediate vector, the operation result for the corresponding set of similar vectors can be determined, thereby reducing the computational load.

[0015] In some embodiments of the first aspect, generating the first set of output vectors includes: generating a mask matrix indicating grouping information, wherein the positions of the first set of similar vectors in a set of input vectors are indicated by a set of identical elements in the mask matrix; and generating the first set of output vectors based on a set of intermediate vectors and the mask matrix. This provides an implementation of the grouping information.

[0016] In a second aspect, a data generation apparatus is provided, comprising: an acquisition module configured to acquire a compression ratio for a set of input vectors; a determination module configured to determine multiple sets of similar vectors from the set of input vectors based on the compression ratio; and a generation module configured to generate a first intermediate vector for a first set of similar vectors from the multiple sets of similar vectors.

[0017] In some embodiments of the second aspect, a set of input vectors is the input to the attention layer of the generative model at the corresponding time step of executing the generative model, and the acquisition module includes: a number determination module configured to determine the number of vectors in the set of input vectors that make a threshold contribution to the output of the attention layer; and a compression ratio determination module configured to determine the compression ratio as the ratio of the number to the total number of the set of input vectors.

[0018] In some embodiments of the second aspect, the number determination module includes: a heatmap-based determination module configured to determine the number based on the attention score heatmap of the attention layer at the corresponding time step.

[0019] In some embodiments of the second aspect, the determining module includes: a downsampling module configured to obtain downsampled vectors of multiple layers by downsampling a set of input vectors multiple times; and a similar vector determining module configured to determine multiple sets of similar vectors based on the downsampled vectors of multiple layers and a similarity threshold, wherein the similarity threshold is determined based on the compression ratio.

[0020] In some embodiments of the second aspect, the similarity vector determination module includes: a first determination module configured to, based on a similarity threshold, determine a set of vectors in the last layer of a plurality of layers that are similar to the first vector, for a first vector in a set of input vectors.

[0021] In some embodiments of the second aspect, the similarity vector determination module further includes: a next-layer similarity acquisition module, configured to acquire, for a layer other than the last layer among multiple layers, a first group of vectors in the next layer of that layer that are similar to the first vector; a second determination module, configured to determine a second group of vectors in that layer that correspond to the first group of vectors; and a third determination module, configured to determine a third group of vectors in the second group of vectors that are similar to the first vector based on a similarity threshold.

[0022] In some embodiments of the second aspect, the third determining module includes: a fourth determining module configured to determine the second set of vectors as the third set of vectors in response to a resolution corresponding to the layer being greater than a resolution threshold.

[0023] In some embodiments of the second aspect, a vector in a set of input vectors corresponds to a set of patches in an image sequence to be generated by an image generator, and the downsampling module includes a spatiotemporal downsampling module configured to downsample a set of input vectors in the temporal and spatial domains.

[0024] In some embodiments of the second aspect, the apparatus further includes: an output generation module configured to generate a first set of output vectors for the first set of similar vectors based on a first intermediate vector and grouping information for multiple sets of similar vectors.

[0025] In some embodiments of the second aspect, the output generation module includes: a mask generation module configured to generate a mask matrix indicating grouping information, wherein the positions of the first group of similar vectors in a set of input vectors are indicated by a set of identical elements in the mask matrix; and a vector generation module configured to generate a first group of output vectors based on a set of intermediate vectors and the mask matrix.

[0026] In a third aspect, a chip is provided, including processing circuitry and an interface, the processing circuitry being configured to perform actions according to the method of the first aspect above or any embodiment thereof.

[0027] In a fourth aspect, an electronic device is provided, including a processor and a memory storing computer instructions that, when executed by the processor, cause the electronic device to perform actions according to the method of the first aspect or any embodiment thereof.

[0028] In a fifth aspect, a computer program product is provided, the computer program product including instructions that, when executed by an electronic device, cause the electronic device to perform actions according to the method of the first aspect above or any embodiment thereof.

[0029] In a sixth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions that, when executed by an electronic device, cause the electronic device to perform actions according to the method of the first aspect or any embodiment thereof. Attached Figure Description

[0030] Figure 1 shows a schematic diagram of an example environment in which various embodiments of this application can be implemented;

[0031] Figure 2 shows a flowchart of an example data generation method according to some embodiments of this application;

[0032] Figure 3 illustrates an example process for determining similar vectors in a set of input vectors according to some embodiments of this application;

[0033] Figure 4 illustrates a schematic diagram of determining a set of similarity vectors according to some embodiments of this application;

[0034] Figure 5 illustrates an example process for generating a set of output vectors from a set of input vectors according to some embodiments of this application;

[0035] Figure 6 illustrates a schematic diagram of improvements to the attention layer of a generative model according to some embodiments of this application;

[0036] Figure 7 shows a schematic diagram of the structure of a data generation system according to some embodiments of this application;

[0037] Figure 8 shows a schematic block diagram of an example data generation apparatus according to some embodiments of this application; and

[0038] Figure 9 shows a schematic block diagram of an example device that can be used to implement embodiments of this application. Detailed Implementation

[0039] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0040] In the description of embodiments of this application, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0041] While multimodal diffusion models (DM) excel in visual content generation, their inference phase requires multiple forward computations, significantly increasing computational cost and memory usage, leading to increased image or video generation latency and limiting their practicality in various applications. Taking video generation as an example, as the frame rate and resolution of videos increase, the number of tokens input to the diffusion model grows exponentially. These tokens, after being mapped to a low-dimensional latent space through the video coding network, form extremely long input vector sequences. The diffusion model contains numerous self-attention and cross-attention layers. Since the computational complexity and memory usage of attention are quadratic with the length of the input sequence, the inference speed of the DM model significantly decreases for long token sequences. Maintaining both generation quality and efficiency remains a challenge for multimodal diffusion models in image and video generation tasks.

[0042] To address the aforementioned and other issues, embodiments of this application provide a data generation scheme. This scheme obtains a compression ratio for a set of input vectors (e.g., a set of feature vectors corresponding to the input symbol sequence of a diffusion model), and determines multiple sets of similar vectors within that set of input vectors based on the compression ratio. Then, the scheme generates an intermediate vector for each of the multiple sets of similar vectors. In this way, a smaller set of intermediate vectors adapted to the compression ratio can be generated, thereby substantially compressing the amount of data in the input vectors.

[0043] Using these generated intermediate vectors for subsequent calculations can reduce the computational load of the model while maintaining its output accuracy. For example, in a diffusion model that iterates through multiple time steps, the denoising level varies at each time step, and the number of inputs required to achieve the desired accuracy at each time step also differs. The embodiments of this application can obtain a compression ratio adapted to the corresponding time step and determine multiple sets of similar vectors in the input of that time step accordingly, thereby ensuring the generation of an appropriate number of intermediate vectors to achieve the desired accuracy in computation.

[0044] In some embodiments of this application, when determining multiple sets of similar vectors, the input vector can be downsampled multiple times to obtain downsampled vectors from multiple layers. Then, multiple sets of similar vectors in the input vector can be determined based on the downsampled vectors from multiple layers and a similarity threshold determined according to the compression ratio. In this way, the criterion for whether vectors are similar can be determined by the obtained compression ratio, making the number of obtained sets of similar vectors (i.e., the number of subsequently generated intermediate vectors) compatible with the desired compression ratio.

[0045] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. For ease of explanation, embodiments of this application will be described herein in conjunction with the input and output of the attention layer of a generative model (e.g., a diffusion model). However, it should be understood that embodiments of this application can also be used to generate a smaller set of intermediate vectors for a different set of input vectors.

[0046] Referring first to Figure 1, Figure 1 illustrates a schematic diagram of an example environment 100 in which various embodiments of this application can be implemented. Example environment 100 includes a computing device 110. The computing device 110 can be implemented as any device with the required computing capabilities. Examples include, but are not limited to, servers of service (e.g., image or video generation services) providers, desktop computers, smartphones, tablets, laptops, or any combination thereof.

[0047] It should be understood that although shown as a single device, computing device 110 can also be implemented in any other form suitable for performing the corresponding functions. For example, in some embodiments, the functionality of the components of computing device 110 can be implemented as a computing cluster or multiple computing machines communicatively connected to each other. In other embodiments, some or all of the components of computing device 110 can also be arranged in the form of a cloud computing architecture, wherein these components can be remotely deployed and can work together to achieve the functions described in this application.

[0048] The computing device 110 is equipped with a content generator 120 based on a generative model. For example, the computing device 110 can use the content generator 120 to generate content data, such as data 105. In some embodiments, the content generator 120 can generate output images or videos (e.g., as image sequences) based on text input. The content generator 120 includes an implementation of a generative model 130. As an example, the generative model 130 can be a diffusion model (e.g., a U-Net-based diffusion model, Diffusion Transformers (DiT), etc.). A diffusion model is a machine learning model primarily used to generate new data. The core idea of ​​a diffusion model originates from the diffusion process in physics and is applied in reverse in deep learning to generate data.

[0049] The diffusion model begins its computation from the final state of the diffusion process (e.g., a noisy data sample at the final time T). The model first initializes this noisy sample and then performs the reverse process of diffusion step by step. At each time step, the output of the previous time step is passed through the various layers of the model to remove noise to a certain extent, gradually revealing the details hidden beneath the noise. Finally, through a series of reverse steps, the model is able to generate data corresponding to the initial state of the diffusion process. In this way, the diffusion model can generate high-quality data, such as images, text, or other types of data, from noisy samples.

[0050] It should be understood that the environment 100 shown in Figure 1 is merely exemplary and should not constitute any limitation on the functionality and scope of the implementation described in this application. Other devices, systems, or components, not shown, may also be present in the example environment 100. For example, in a scenario where the computing device 110 is a server for a video generation service, environment 110 may also include a client (not shown). This client may send a request to the computing device 110 to generate desired content.

[0051] Referring now to FIG2. FIG2 illustrates a flowchart of an example data generation method 200 according to some embodiments of this application. Example method 200 can be performed, for example, by a computing device 110 as shown in FIG1. ​​It should be understood that method 200 may also include additional actions not shown, and the scope of this application is not limited in this respect. Method 200 will now be described in detail with reference to the example setting 100 of FIG1.

[0052] At 210, the compression ratio for a set of input vectors is obtained. For example, computing device 110 can obtain the compression ratio for a set of input vectors. Generative model 130 can encode input symbols into corresponding feature vectors and compute the feature vectors of a set of input symbols through various layers. In some embodiments, the set of input vectors can be the input passed by computing device 110 to the attention layer of the generative model at a corresponding time step in the execution of generative model 130.

[0053] In machine learning, attention layers enable models to selectively focus on certain parts of the input data. They automatically learn which elements in the input sequence are more critical to the task the model is performing. Attention layers determine attention weights by computing the interaction between the query vector and the key and value vectors. These attention weights are then applied to the value vectors to highlight the parts the model should focus on. As those skilled in the art will understand, attention layers can include self-attention layers and cross-attention layers.

[0054] In some embodiments concerning the input to the attention layer, computing device 110 can determine the number n of vectors in a set of input vectors that contribute a threshold percentage k to the output of the attention layer. The ratio n / m of this number n to the total number m of the set of input vectors is determined as the compression ratio. In some embodiments, computing device 110 can determine the number for a given time step based on the attention score heatmap of the attention layer at that time step. The acquisition of the compression ratio will be described in more detail later with reference to Figure 5.

[0055] At 220, multiple sets of similar vectors are determined from a set of input vectors based on the compression ratio. For example, computing device 110 may determine multiple sets of similar vectors from a set of input vectors based on the compression ratio. In some embodiments, computing device 110 may determine multiple sets of similar vectors based on the compression ratio such that the ratio a / b between the number of such sets a and the number of input vectors is adapted to the compression ratio, for example, the same as the compression ratio or differing from it within a tolerance range.

[0056] In some embodiments, the computing device 110 can determine multiple groups of similar vectors based on a similarity threshold, which is determined based on a compression ratio. For example, the computing device 110 can determine vectors whose similarity reaches the threshold as sufficiently similar to be grouped together. For example, the smaller the compression ratio n / m, the smaller the similarity threshold can be adjusted, resulting in a smaller number of similar vector groups, so that fewer intermediate vectors are subsequently generated. The determination of similar vectors will be described in more detail later with reference to Figures 3 and 4.

[0057] At 230, a first intermediate vector is generated for the first group of similar vectors in a plurality of similar vectors. For example, computing device 110 may generate an intermediate vector for each group of similar vectors in a plurality of similar vectors. In this way, computing device 110 can generate a compressed set of intermediate vectors from a set of output vectors.

[0058] In some embodiments, the computing device 110 may randomly output one vector from a set of similar vectors as the intermediate vector corresponding to that set of similar vectors. In some embodiments, the computing device 110 may generate the mean vector of a set of output vectors as the corresponding intermediate vector. In some embodiments, the computing device 110 may generate the centroid vector of a set of output vectors in the vector space as the corresponding intermediate vector. Other suitable generation methods may also be used, and the embodiments of this application do not limit this.

[0059] Using method 200, a smaller number of intermediate vectors corresponding to the input vector can be generated for subsequent calculations, so that the data to be calculated is compressed in a way that fits the desired compression ratio. This allows for a reduction in model computation while still maintaining the model's desired output accuracy.

[0060] When compressing input data, some methods determine the importance of each input symbol and remove unimportant inputs. These methods are either limited to traditional classification tasks or require additional training for specific models, resulting in poor generalization ability. On the other hand, generative models such as diffusion models involve a large amount of repetitive inter-layer computation during image generation, where similar input vectors produce similar outputs. Some methods use K-means to cluster similar vectors in the input and merge similar vectors. However, these methods introduce significant additional computation, limiting the overall running speed and making it difficult to accelerate applications of the model.

[0061] In contrast, in some embodiments of this application, when determining multiple sets of similar vectors from a set of input vectors based on the aforementioned compression ratio, the computing device 110 can obtain downsampled vectors from multiple layers by downsampling a set of input vectors multiple times. Then, based on the downsampled vectors from multiple layers and a similarity threshold, the computing device 110 can determine multiple sets of similar vectors. As described above in conjunction with 220, the similarity threshold is determined based on the compression ratio. In this process, the computing device 110 does not need to traverse the complete sequence of input vectors, thereby improving execution speed.

[0062] Referring now to FIG3, an example process 300 for determining similar vectors in a set of input vectors according to some embodiments of this application is illustrated. Process 300 may be an example implementation of 220 in method 200 and may be implemented by computing device 110 in FIG1. ​​It should be understood that some actions in process 300 may be omitted, and process 300 may also include additional actions not shown, and the scope of this application is not limited in this respect. Process 300 is described in detail below with reference to the example environment 100 of FIG1.

[0063] In step 310, computing device 110 can obtain downsampled vectors for multiple layers by downsampling a set of input vectors multiple times. Computing device 110 can employ any suitable downsampling method, such as random pooling, mean pooling, or region pooling; the embodiments of this application do not specifically limit the downsampling method. The resolution of each layer in the downsampled vectors of multiple layers decreases progressively; for example, computing device 110 can perform multiple downsampling operations at 2*2, 5*5, or other appropriate multiples each time. For ease of explanation, the following description of operations for multiple layers also includes the original layer containing the set of input vectors.

[0064] At 320, computing device 110 can determine multiple sets of similar vectors based on downsampled vectors from multiple layers and the similarity thresholds described above. In operation, computing device 110 can use any appropriate similarity metric, such as mean-square error (MSE), cosine similarity, and normalized similarity coefficient, etc., and the embodiments of this application are not limited thereto.

[0065] Specifically, computing device 110 can operate on input vectors within a set of input vectors. At 330, computing device 110 can determine a group of vectors similar to the input vector in the last layer (i.e., the layer with the lowest resolution) among multiple layers, based on a similarity threshold. Computing device 110 can calculate the similarity between each vector in the last layer and the input vector, and include vectors with similarity reaching the threshold in the group of vectors similar to the input vector. Thus, computing device 110 can obtain a smaller scan / search range from a set of input vectors and pass this range to the previous layer to scan / search for vectors similar to the currently compared input vector within that range of the previous layer.

[0066] At 340, computing device 110 can move up one layer and operate on that layer (e.g., from the last layer to the penultimate layer with higher resolution). At 340-1, computing device 110 can obtain vectors similar to the input vector in the next layer of that layer. For example, for the penultimate layer, computing device 110 can obtain a set of vectors similar to the last layer obtained at 330 as the first set of vectors.

[0067] At 340-2, computing device 110 can acquire the second set of vectors corresponding to the first set of vectors in the current layer (during downsampling, the second set of vectors is downsampled to the first set of vectors). At 340-3, computing device 110 can determine a third set of vectors in the second set of vectors that are similar to the input vector, based on a similarity threshold. Computing device 110 can calculate the similarity between each vector in the second set of vectors and the input vector, and include vectors whose similarity reaches the threshold in the third set of vectors similar to the input vector. In this way, computing device 110 can obtain a further narrowed range, which can be used as a determination result or passed to the previous layer based on the judgment at 350.

[0068] During the upward shift, if the layer resolution is greater than a certain threshold, the computational overhead of continuing to determine similarity within the search range may offset or exceed the benefits of further compressing a set of input vectors. Therefore, in some embodiments, in response to the resolution corresponding to the current layer being greater than a resolution threshold (e.g., 768*768), the computing device 110 can adaptively and directly determine the second set of vectors as the third set of vectors to improve execution speed.

[0069] At 350, computing device 110 can determine whether the current layer is the original layer. If it is, computing device 360 ​​determines the third set of vectors as a set of vectors similar to the input vectors. If the current layer is not the original layer, process 300 returns to 340, and computing device 110 continues to move up one layer and perform operations on that layer until the original layer is reached.

[0070] In some embodiments, after determining the current input vector and similar vectors in the original layer, the computing device 110 can group them into the same group of similar vectors. The operation 320 continues to be performed on the remaining input vectors that have not yet been grouped until all input vectors are grouped.

[0071] Example procedure 300 provides a multi-scale symbol matching method. This method can quickly narrow down the range of vectors in a set of input vectors for searching for vectors similar to the vector to be matched, without traversing all input vectors, thereby improving the efficiency of determining multiple sets of similar data in the input data.

[0072] In some embodiments, a set of input vectors corresponds to a set of tiles in an image or image sequence to be generated by the image generator. The image generator may be implemented based on a generative model. At a corresponding time step during the execution of the image generator, the set of input vectors is fed into the image generator as a batch. Alternatively, the set of input vectors is the input to a layer of the image generator, such as an attention layer. During the execution of the image generator, the image sequence is denoised step-by-step to ultimately generate the output image sequence. The computing device 110 may downsample the set of input vectors multiple times in the temporal and spatial domains (e.g., each sample resolution is divided by 2 in the temporal domain and by 2 in the spatial domain) to obtain downsampled vectors for multiple layers.

[0073] Figure 4 illustrates a schematic diagram 400 for determining a set of similar vectors according to some embodiments of this application. Figure 4 is described using a set of input vectors corresponding to an image sequence as an example. As shown in Figure 4, each set of input vectors at a corresponding time step corresponds to a set of patches in the image sequence 410 (in other words, a region of the image sequence in the temporal and spatial domains), as shown in layer 430 of the original input vectors.

[0074] The computing device 110 can downsample the set of input vectors multiple times to obtain multiple downsampled layers, including layer 420. The vectors in each layer correspond to a region of the image sequence 410. For example, the filter used for downsampling can be a Gaussian filter. For example, for a set of input vectors related to the attention layer, the computing device 110 can downsample the matrix of the original vectors multiple times, or it can downsample the converted K, V, or Q matrices multiple times to obtain multiple downsampled layers for determining similar vectors.

[0075] The computing device 110 can use, for example, process 300 to determine multiple sets of similar vectors. For a given input vector, the computing device can determine the range of locations of vectors similar to it in the last downsampling layer, as shown by reference numeral 415. This matching location is then passed up to the next layer so that the computing device 110 can perform a more refined matching of vectors within that location range in the previous layer with the input vector. Ultimately, a portion of that location range in the original layer is determined to be similar to the input vector, as shown by reference numeral 425. These similar vectors can be grouped together with the input vector. The computing device 110 can then generate an intermediate vector for each group of input vectors for subsequent computation. As described above, this will not be repeated here.

[0076] In some embodiments, the computing device 110 can generate a set of output vectors for a set of similar vectors corresponding to an intermediate vector, based on each intermediate vector and grouping information for multiple sets of similar vectors. In some embodiments, the computing device 110 can generate a mask matrix indicating the grouping information for multiple sets of similar vectors. The relative positions of a set of similar vectors in a set of input vectors are indicated by a set of identical elements in the mask matrix. For example, a set of similar vectors corresponding to reference numeral 425 can be indicated by the same value in the mask matrix.

[0077] Figure 5 illustrates an example process 500 for generating a set of output vectors from a set of input vectors according to some embodiments of this application. It should be understood that some actions in process 500 may be omitted, and process 500 may also include additional actions not shown; the scope of this application is not limited in this respect. The example process 500 will continue to be described below with reference to Figure 1. In the example of Figure 5, a set of input vectors can be used to compute the inputs fed to the attention layer of the generative model 130 by device 110 at the corresponding time step during the execution of the generative model 130.

[0078] At 510, computing device 110 can determine a corresponding compression ratio for each time step of executing generative model 130. As generative model 130 completes training, the degree of denoising at each sampled time step varies, and the number of vectors required to achieve the desired accuracy output also differs. Computing device 110 can therefore employ a method according to embodiments of this disclosure for denoising-aware adaptive compression ratio adjustment.

[0079] In some embodiments, computing device 110 can acquire an attention score heatmap of the attention layer at a corresponding time step, which is fed from the input vector. The attention score heatmap reflects the degree of importance the attention layer attaches to each input. Based on this, computing device 110 can employ a threshold k algorithm to determine the compression ratio for that time step.

[0080] Specifically, the computing device 110 can determine, based on the attention score heatmap, the number K of input vectors that contribute a threshold (e.g., 95%) to the output of the attention layer (in other words, contribute threshold precision to the output), reflecting the minimum number of vectors required to retain the amount of key information. The computing device 110 can then determine the compression ratio r for that time step as the ratio of K to the total number of input vectors N. The larger the value of K / N, the less the input vectors can be compressed.

[0081] Furthermore, at 520, based on the compression ratio, computing device 110 can determine a similarity threshold f(r) for identifying similar vectors at 530. Taking MSE as an example of similarity measurement, for vectors token1 and token2 corresponding to two symbols, computing device 110 can determine that the two vectors are similar when MSE(token1,token2) < f(r). The larger r is, the larger f(r) is, and vice versa. In an example implementation, computing device 110 can determine the similarity threshold f(r) based on the following formula: f(r) = e -ar Where 0 < f(r) < 1, α is the similarity decay rate, and α > 0. Other similarity measures and corresponding similarity threshold determination algorithms can also be used, as long as they meet the required correlation.

[0082] The computing device 110 can pre-determine the compression ratio {r0, r1, ..., r} for each time step offline. n Alternatively, a predetermined compression ratio can be obtained (e.g., as a configuration or execution parameter), and the corresponding compression ratio can be directly applied at each time step of executing the generative model. In some embodiments, the computing device 110 may also predetermine a similarity threshold f(r) or obtain a predetermined f(r) (e.g., as a configuration or execution parameter).

[0083] At 530, computing device 110 can determine multiple sets of similar vectors from a set of input vectors for a given time step by performing multi-scale vector matching based on a similarity threshold, for example, following process 300. At 540, computing device can determine the corresponding intermediate vector for each set of input vectors. The actions at 530 and 540 have been described in detail above in conjunction with Figures 2 to 4, and will not be repeated here.

[0084] Through steps 510 to 540 described above, computing device 110 can obtain a smaller set of intermediate vectors corresponding to a set of input vectors (in this context, such a set of intermediate vectors is also referred to as a set of compressed vectors). Then, at 550, based on the set of intermediate vectors and similarity information, computing device 110 can determine a set of output vectors corresponding to a set of input vectors. Specifically, based on each intermediate vector and corresponding grouping information, computing device 110 can generate a subgroup (e.g., a first set of output vectors) of the set of output vectors for that intermediate vector (e.g., a first intermediate vector), which corresponds to a set of similar vectors corresponding to that intermediate vector.

[0085] In some embodiments, for a vector in a set of intermediate vectors, computing device 110 can perform the operation to be performed on a vector in a set of input vectors to obtain a corresponding intermediate output. Then, computing device 110 can copy this output as a final output of a set of similar vectors corresponding to the intermediate vector, based on similarity information. In some embodiments, computing device 110 can add fluctuations to each output during copying.

[0086] For example, for each intermediate vector input to the attention layer, its attention can be computed as n*exp(q*k). T )=exp(q*k T +log(n)). As mentioned above, the actual number of vectors calculated differs between different batches and different sampling steps. To achieve batch processing, the computing device 110 needs to restore the final output vectors to the number corresponding to the input vectors, i.e., the number before compression. For example, in an implementation using a mask matrix, the computing device 110 can determine a set of similar vectors corresponding to the intermediate output based on the values ​​in the mask matrix, and copy the intermediate output to the final output corresponding to these similar vectors.

[0087] Process 500 provides an improved approach to enhance the efficiency of computing devices in executing generative models without requiring additional model training. Generative models, such as diffusion models, involve numerous inter-layer computations in generating content like images, where similar inputs produce similar outputs. By compressing a set of similar vectors into an intermediate vector and performing a single computation on it, this approach reduces redundant computations during execution. Furthermore, the approach adaptively adjusts the compression ratio of the input vectors at different time steps, thereby improving execution speed while maintaining output accuracy. Further, in determining similar vectors, the approach provides a fast method for identifying and labeling similar vectors, significantly reducing computational overhead compared to methods that require traversing all data for similarity determination, thus accelerating model inference.

[0088] Figure 6 illustrates a schematic diagram 600 of an example improvement to the attention layer of a generative model according to some embodiments of this application. Such an improvement can be applied, for example, to the generative model 130 of Figure 1. The examples in Figure 6 are described in detail below with reference to the example environment 100 of Figure 1.

[0089] As shown in Figure 6, the attention layer 610 may include cross-attention 611, self-attention 612, and a feed-forward network (FFN) 613 that performs a nonlinear transformation on the output of the attention computation. In this example, at a certain time step of executing the generative model, a set of input vectors 601 is compressed into a set of intermediate vectors (not shown) when passed to the attention layer 610. The computing device 110 can generate a set of intermediate vectors from a set of input vectors 601 based on the manner described above in conjunction with Figures 2 to 5. At the corresponding time step, the compression ratio of the input vectors is determined based on the attention score heatmap 620 of the attention layer for that time step, as described above in conjunction with Figure 5, and will not be repeated here.

[0090] Attention layer 610 performs attention calculations on the set of intermediate vectors and outputs a set of intermediate output vectors that is fewer in number than the expected output, for example, as shown by reference numeral 602. To achieve batch processing, computing device 110 also needs to recover missing vectors from the final output vectors (e.g., as shown by reference numeral 603). Therefore, the set of intermediate output vectors is recovered by recovery component 630. During the recovery process in this example, recovery component 630 can copy the values ​​of intermediate output vectors to the output positions corresponding to their associated similar vector groups, based on the grouping information of similar vectors determined when compressing the input vectors, for example, as described by reference numeral 604.

[0091] Figure 7 shows a schematic diagram of the structure of a data generation system 700 according to some embodiments of this application. The data generation system 700 can be implemented, for example, as an example of the content generator 120 in Figure 1. It should be understood that some modules in the system 700 can be omitted, and the system 700 may also include additional modules not shown, and the scope of this application is not limited in this respect.

[0092] The data generation system 700 may include a diffusion transformer 710, which includes a DiT block 720 for processing the input symbol sequence. For clarity, other modules of the diffusion transformer 710 are omitted in FIG. 7. As known to those skilled in the art, the DiT block 720 includes an attention module 730 that performs attention calculations. For clarity, other sub-modules of the attention module 730 are omitted in FIG. 7.

[0093] According to some embodiments of this disclosure, the attention module 730 includes a compression module and a restoration module at the computational layer of each attention. For example, the attention module 730 may include a compression module 760-1 and a restoration module 770-1 for self-attention 740, and a compression module 760-2 and a restoration module 770-1 for cross-attention 750.

[0094] Each compression module compresses a set of input vectors for the corresponding attention calculation into a set of intermediate vectors for attention calculation, as described above in conjunction with Figures 2 to 6. During compression, a set of similar vectors is merged into a single intermediate vector. The grouping information of similar vectors is recorded in the form of mask information 705-1 and mask information 705-2, where elements corresponding to similar elements have the same value.

[0095] Take a computation request for one attention head in a multi-head attention problem as an example. Before compression, the partial output out1 of the attention computation can be calculated as follows, where k m and v n Let out1 represent the key vector and value vector of the corresponding input symbol, respectively: out1=(exp(qk1)v1+exp(qk2)v2+exp(qk3)v3) / Z,

[0096] Where Z = exp(qk1) + exp(qk2) + exp(qk3).

[0097] In this example, by downsampling the query vector matrix multiple times, k1 and k2 are determined to be similar, meaning their corresponding input symbols are similar. Therefore, k1 and k2 can be merged / compressed into their corresponding vector k. 12 As mentioned earlier, k 12 It can be one of k1 and k2, the mean vector of both, or the centroid vector of both, etc. Similarly, v1 and v2 can be combined into the corresponding vector v. 12 Furthermore, in this example, assuming that the outputs of attention calculations for similar vectors will also be equal, the compressed output out2 can be calculated as follows: out2=(exp(qk 12 )v1+exp(qk 12 )v2+exp(qk3)v3) / Z =(2*exp(qk 12 )v 12 +1*exp(qk3)v3) / Z,

[0098] Where Z = exp(qk) 12 )+exp(qk 12 )+exp(qk3)

[0099] =2*exp(qk) 12 )+1*exp(qk3).

[0100] In this way, after merging, the multiplication calculation for the input vectors corresponding to numbers 1 and 2 only needs to be performed once, thereby reducing computational complexity and improving the output generation speed.

[0101] Furthermore, the corresponding attention calculation generates intermediate results for the corresponding set of intermediate vectors. Then, based on the corresponding mask information 705-1 or 705-2, the corresponding restoration module restores the intermediate results into the corresponding set of output vectors to be passed to the subsequent computation layers, as described above in conjunction with Figures 5 and 6. During the execution of the diffusion transformer 710, the compression ratio 715 for each time step is used to determine multiple sets of similar elements in the input vector, as described above in conjunction with Figures 2 to 4, and will not be repeated here.

[0102] The embodiments of this application can be applied to the field of content generation, such as image or video generation, especially in scenarios requiring the rapid generation of high-quality images or videos. Through the accelerated diffusion model, the time and computational resources required to generate images or videos can be significantly reduced, making real-time or near real-time image or video generation possible. Furthermore, the embodiments of this application can also support large-scale content generation tasks, such as data augmentation, style transfer, image editing, and synthetic data generation. In a test example for text-to-image generation, compared to traditional methods, generating image data according to the embodiments of this application can improve GPU end-to-end performance by 32%, NPU end-to-end performance by 21%, and the CLIP score accuracy loss of the generated result is less than 0.3%.

[0103] Figure 8 shows a schematic block diagram of a data generation apparatus 800 according to some embodiments of this application. The apparatus 800 may be implemented as or included in the computing device 110 of Figure 1, for example. The apparatus 800 may include multiple modules for performing corresponding actions in, for example, method 200, process 300 and / or process 500.

[0104] As shown in Figure 8, the device 800 includes an acquisition module 810, a determination module 820, and a generation module 830. The acquisition module 810 is configured to acquire the compression ratio for a set of input vectors. The determination module 820 is configured to determine multiple sets of similar vectors from the set of input vectors based on the compression ratio. The generation module 830 is configured to generate a first intermediate vector for a first set of similar vectors from the multiple sets of similar vectors.

[0105] In some embodiments, a set of input vectors is the input to the attention layer of the generative model at the corresponding time step of executing the generative model, and the acquisition module 810 includes: a number determination module configured to determine the number of vectors in the set of input vectors that make a threshold contribution to the output of the attention layer; and a compression ratio determination module configured to determine the compression ratio as the ratio of the number to the total number of the set of input vectors.

[0106] In some embodiments, the number determination module includes a heatmap-based determination module configured to determine the number based on the attention score heatmap of the attention layer at the corresponding time step.

[0107] In some embodiments, the determining module 820 includes: a downsampling module configured to obtain downsampled vectors of multiple layers by downsampling a set of input vectors multiple times; and a similar vector determining module configured to determine multiple sets of similar vectors based on the downsampled vectors of multiple layers and a similarity threshold, wherein the similarity threshold is determined based on the compression ratio.

[0108] In some embodiments, the similarity vector determination module includes: a first determination module configured to determine, based on a similarity threshold, a set of vectors in the last layer of a plurality of layers that are similar to the first vector, for a first vector in a set of input vectors.

[0109] In some embodiments, the similarity vector determination module further includes: a next-layer similarity acquisition module, configured to acquire, for a layer other than the last layer among multiple layers, a first group of vectors similar to the first vector in the next layer of that layer; a second determination module, configured to determine a second group of vectors in that layer corresponding to the first group of vectors; and a third determination module, configured to determine a third group of vectors in the second group of vectors similar to the first vector based on a similarity threshold.

[0110] In some embodiments, the third determining module includes a fourth determining module configured to determine the second set of vectors as the third set of vectors in response to a resolution corresponding to the layer being greater than a resolution threshold.

[0111] In some embodiments, a vector in a set of input vectors corresponds to a set of tiles in an image sequence to be generated by an image generator, and the downsampling module includes a spatiotemporal downsampling module configured to downsample a set of input vectors in the temporal and spatial domains.

[0112] In some embodiments, the apparatus 800 further includes an output generation module configured to generate a first set of output vectors for the first set of similar vectors based on a first intermediate vector and grouping information for multiple sets of similar vectors.

[0113] In some embodiments, the output generation module includes: a mask generation module configured to generate a mask matrix indicating grouping information, wherein the positions of the first group of similar vectors in a set of input vectors are indicated by a set of identical elements in the mask matrix; and a vector generation module configured to generate a first group of output vectors based on a set of intermediate vectors and the mask matrix.

[0114] Figure 9 shows a schematic block diagram of an example device 900 that can be used to implement embodiments of this application. Device 900 can be used to implement the functions of the computing device 110 shown in Figure 1. As shown, device 900 includes a processing unit 901, which can perform various appropriate actions and processes according to computer program instructions stored in random access memory (RAM) 903 and / or read-only memory (ROM) 902, or computer program instructions loaded from storage unit 908 into RAM 903 and / or ROM 902. Various programs and data required for the operation of device 900 may also be stored in RAM 903 and / or ROM 902. Processing unit 901 and RAM 903 and / or ROM 902 are connected to each other via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0115] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0116] Processing unit 901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processing unit 901 include, but are not limited to, CPUs, GPUs, various special-purpose AI computing chips, various processing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processors, controllers, microcontrollers, etc. Processing unit 901 performs the various methods and processes described above, such as method 200 and / or method 400. For example, in some embodiments, method 400 and / or method 400 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via RAM and / or ROM and / or communication unit 909. When the computer program is loaded into RAM and / or ROM and executed by processing unit 901, one or more actions of the various methods and processes described above may be performed. Alternatively, in other embodiments, the processing unit 901 may be configured by any other suitable means (e.g., by means of firmware) to perform one or more actions of the methods and processes described above.

[0117] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0118] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. As used herein, computer-readable storage media is not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0119] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to computer-readable storage media in the respective computing / processing device.

[0120] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some exemplary implementations, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of this disclosure.

[0121] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products according to exemplary implementations of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0122] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0123] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various exemplary embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0125] Embodiments of this disclosure have been described. The above description is exemplary and not exhaustive, and is not limited to the various embodiments disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or improvements to the technology in the market, or to enable others skilled in the art to understand the various embodiments disclosed herein.

Claims

1. A data generation method, characterized in that, include: Obtain the compression ratio for a set of input vectors; Based on the compression ratio, multiple sets of similar vectors are determined from the set of input vectors; as well as For the first group of similar vectors among the multiple groups of similar vectors, a first intermediate vector is generated.

2. The method according to claim 1, characterized in that, The set of input vectors is the input to the attention layer of the generative model at the corresponding time step during the execution of the generative model, and obtaining the compression ratio includes: Determine the number of vectors in the set of input vectors that contribute a threshold to the output of the attention layer; and The compression ratio is determined by the ratio of the number to the total number of the set of input vectors.

3. The method according to claim 2, characterized in that, Determining the number includes: The number is determined based on the attention score heatmap of the attention layer at the corresponding time step.

4. The method according to claim 1, characterized in that, Determining the multiple sets of similar vectors includes: By downsampling the set of input vectors multiple times, downsampled vectors from multiple layers are obtained; and The multiple sets of similar vectors are determined based on the downsampled vectors of the multiple layers and a similarity threshold, wherein the similarity threshold is determined based on the compression ratio.

5. The method according to claim 4, characterized in that, Determining the multiple sets of similar vectors based on the multiple layers and the similarity threshold includes: Based on the similarity threshold, for the first vector in the set of input vectors, a set of vectors in the last layer of the plurality of layers that are similar to the first vector are determined.

6. The method according to claim 5, characterized in that, Determining the multiple sets of similar vectors based on the multiple layers and the similarity threshold further includes: For each of the multiple layers except the last layer, obtain a first group of vectors in the next layer that are similar to the first vector; Determine the second set of vectors in the layer that correspond to the first set of vectors; and Based on the similarity threshold, a third group of vectors similar to the first vector is determined in the second group of vectors.

7. The method according to claim 6, characterized in that, Determining the third set of vectors includes: In response to the resolution corresponding to the layer being greater than a resolution threshold, the second set of vectors is determined as the third set of vectors.

8. The method according to claim 4, characterized in that, The vectors in the set of input vectors correspond to a set of patches in the image sequence to be generated by the image generator, and obtaining the downsampled vectors includes: The input vectors are downsampled in both the time and spatial domains.

9. The method according to claim 1, characterized in that, Also includes: Based on the first intermediate vector and the grouping information for the multiple sets of similar vectors, a first set of output vectors is generated for the first set of similar vectors.

10. The method according to claim 9, characterized in that, Generating the first set of output vectors includes: Generate a mask matrix indicating the grouping information, wherein the position of the first group of similar vectors within the group of input vectors is indicated by a set of identical elements in the mask matrix; and The first set of output vectors is generated based on the set of intermediate vectors and the mask matrix.

11. A data generation apparatus, characterized in that, include: The acquisition module is configured to acquire the compression ratio for a set of input vectors; The determination module is configured to determine multiple sets of similar vectors in the set of input vectors based on the compression ratio; as well as The generation module is configured to generate a first intermediate vector for the first group of similar vectors among the multiple groups of similar vectors.

12. The apparatus according to claim 11, characterized in that, The set of input vectors is the input to the attention layer of the generative model at the corresponding time step during the execution of the generative model, and the acquisition module includes: A number determination module is configured to determine the number of vectors in the set of input vectors that contribute a threshold to the output of the attention layer; and A compression ratio determination module is configured to determine the compression ratio by the ratio of the number to the total number of the set of input vectors.

13. The apparatus according to claim 12, characterized in that, The number determination module includes: The heatmap-based determination module is configured to determine the number based on the attention score heatmap of the attention layer at the corresponding time step.

14. The apparatus according to claim 11, characterized in that, The determining module includes: The downsampling module is configured to obtain downsampled vectors for multiple layers by downsampling the set of input vectors multiple times; and A similarity vector determination module is configured to determine the multiple sets of similar vectors based on downsampled vectors from the multiple layers and a similarity threshold, wherein the similarity threshold is determined based on the compression ratio.

15. The apparatus according to claim 14, characterized in that, The similarity vector determination module includes: The first determining module is configured to, based on the similarity threshold, determine a set of vectors in the last layer of the plurality of layers that are similar to the first vector, for the first vector in the set of input vectors.

16. The apparatus according to claim 15, characterized in that, The similarity vector determination module further includes: The next-layer similarity acquisition module is configured to acquire, for each of the multiple layers except the last layer, a first group of vectors in the next layer of that layer that are similar to the first vector; The second determining module is configured to determine a second set of vectors in the layer corresponding to the first set of vectors; and The third determining module is configured to determine a third group of vectors in the second group of vectors that are similar to the first vector, based on the similarity threshold.

17. The apparatus according to claim 16, characterized in that, The third determination module includes: The fourth determining module is configured to determine the second set of vectors as the third set of vectors in response to a resolution corresponding to the layer being greater than a resolution threshold.

18. The apparatus according to claim 14, characterized in that, The vectors in the set of input vectors correspond to a set of patches in the image sequence to be generated by the image generator, and the downsampling module includes: The spatiotemporal downsampling module is configured to downsample the set of input vectors in both the temporal and spatial domains.

19. The apparatus according to claim 11, characterized in that, Also includes: The output generation module is configured to generate a first set of output vectors for the first set of similar vectors based on the first intermediate vector and the grouping information for the multiple sets of similar vectors.

20. The apparatus according to claim 19, characterized in that, The output generation module includes: A mask generation module is configured to generate a mask matrix indicating the grouping information, wherein the positions of the first group of similar vectors within the set of input vectors are indicated by a set of identical elements in the mask matrix; and The vector generation module is configured to generate the first set of output vectors based on the set of intermediate vectors and the mask matrix.

21. A chip, characterized in that, It includes processing circuitry and an interface, the processing circuitry being configured to perform the method according to any one of claims 1 to 10.

22. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing computer instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1 to 10.

23. A computer program product, characterized in that, The computer program product includes instructions that, when executed by an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 10.

Images