Data processing method and apparatus

Abstract

Provided in the embodiments of the present application are a data processing method and apparatus, which are used for calculating an attention matrix on the basis of an adaptive attention window size during attention calculation, so that attention is calculated on the basis of a part having a higher degree of attention, thereby achieving a better output effect while reducing the amount of calculation. The method comprises: acquiring an input sequence, wherein the input sequence may be obtained on the basis of input data, and the input data may comprise, but is not limited to, images, audio data, video data, etc.; and then inputting the input sequence into a large model to obtain an output sequence, wherein the large model comprises a plurality of attention modules, each attention module is configured to determine, on the basis of model structural parameters, the size of an attention window from among a plurality of sizes, and the attention window comprises an input window size when a current region in the input sequence is calculated using an attention mechanism.

Description

A data processing method and apparatus

[0001] This application claims priority to Chinese Patent Application No. 202510129114.4, filed on January 27, 2025, entitled “A Data Processing Method and Apparatus”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and more particularly to a data processing method and apparatus. Background Technology

[0003] With the continuous development of artificial intelligence technology, text and image generation and understanding are gradually maturing, and people are moving towards the more complex field of video generation and understanding. Currently, the mainstream generative model is the Transformer-based diffusion transformer (DiT) model, which has attracted widespread attention due to its excellent scalability. However, its computational cost is proportional to the square of the input length. Videos, compared to images and text, have much longer inputs (e.g., the size changes from H*W to T*H*W), so training DiT models for videos often requires significant computational overhead. Existing solutions typically use spatiotemporal full attention mechanisms, requiring the entire sequence (tokens) of the video as input, resulting in very high computational costs. Alternatively, local attention mechanisms discard the direct correlation of spatiotemporal information, leading to reduced final performance. Therefore, how to achieve better modeling results with lower computational cost has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a data processing method and apparatus for calculating an attention matrix based on an adaptive attention window size when calculating attention, thereby calculating attention based on the parts with higher attention, reducing the amount of computation while achieving better output results.

[0005] In view of this, in a first aspect, this application provides a data processing method, comprising: acquiring an input sequence, which may be obtained based on input data, including but not limited to image data, audio data, video data, etc.; then inputting the input sequence into a large model to obtain an output sequence, wherein the large model includes multiple attention modules, each attention module being used to determine the size of an attention window from multiple sizes according to model structure parameters, the attention window including the input window size when calculating the current region in the input sequence using an attention mechanism.

[0006] In this context, the attention magnitude between the region within the attention window and the current region is typically greater than that between other regions in the input sequence. This means the correlation between the current region and the region within the attention window is stronger than that between other regions. Therefore, the output sequence after attention calculation better represents the features of the current region. Thus, in this embodiment, when calculating attention, the attention matrix can be calculated based on an adaptive attention window size. This allows attention to be calculated only based on the more attention-grabbing parts, eliminating the need to perform attention calculations on both the current region and all regions, reducing computational load without compromising output quality.

[0007] In one possible implementation, the aforementioned attention window includes the range of values ​​in the temporal and / or spatial domains and the number of regions when calculating attention for the current region in the input sequence using an attention mechanism. In this application, the attention window may include an adaptive range in the temporal and / or spatial domains. The size of the adaptive attention window can be determined according to different model structures, thereby enabling the focus on regions more relevant to the current region for different model structures, thus encoding more accurate data representing the features in the input sequence from both the temporal and spatial domains.

[0008] In one possible implementation, the aforementioned plurality of attention modules includes a first attention module, which is specifically configured to: receive a first input subsequence, the first input subsequence being the sequence output by the previous module; determine the size of the current attention window from a variety of sizes based on model structure parameters; calculate an attention matrix based on the size of the current attention window; and output a first output subsequence after post-processing the attention matrix. In this embodiment, multiple window sizes can be preset, and an appropriate size can be selected from multiple window sizes according to various model structures. This allows for the identification of regions more relevant to each region, thereby encoding data with a higher degree of correlation to the current region, making the output sequence more representative of the features of the current region.

[0009] In one possible implementation, the aforementioned attention window is a bidirectional attention window or a unidirectional attention window. A bidirectional attention window is a window that performs attention calculations by taking values ​​in the forward and backward directions in the time and / or spatial dimensions, respectively. A unidirectional attention window is a window that performs attention calculations by taking values ​​in the forward or backward directions in the time and / or spatial dimensions. The forward and backward directions can be opposite directions of value taking. For example, the time and spatial dimensions can be calculated in the order from front to back. The forward direction is the direction of taking values ​​from back to front in the dimension, and the backward direction is the direction of taking values ​​from front to back.

[0010] In one possible implementation, the aforementioned method further includes: obtaining a lookup table containing multiple sizes, wherein each model structure parameter corresponds to a window size, the sizes included in the lookup table may be preset or learned using historical data; each attention module may be used to query the size of the attention window from the lookup table according to the model structure parameters.

[0011] In this embodiment, the size of the attention window can be adaptively determined by looking up a table, thereby adaptively determining the area to be focused on when calculating attention.

[0012] In one possible implementation, the aforementioned acquisition of the lookup table includes: acquiring training data; inputting the training data into a large model, dividing the sequence input into each attention module into multiple regions, and determining the attention magnitude between each region in each attention module and other regions; filtering out the range and / or number of regions associated with each region in each attention module based on the attention magnitude between each region in each attention module and other regions; determining the attention window corresponding to each attention module based on the range and / or number of regions associated with each region in each attention module; and constructing a lookup table based on the model structure parameters of each attention module and the attention window corresponding to each attention module.

[0013] In this embodiment, a small amount of data is input into a large model without setting the size of the attention window. This allows the attention magnitude between each region and other regions to be obtained. Based on the attention magnitude between each region and other regions, regions more relevant to the current region are selected. For example, regions with attention magnitudes greater than a threshold are considered as regions relevant to the current region. This determines the size of the region more relevant to the current region, and this region size is used as the attention window size for the current region. This allows for the extraction of information with higher relevance to each region during subsequent training, reducing the amount of attention computation for each region and achieving better output results.

[0014] In one possible implementation, the aforementioned input sequence includes a sequence obtained based on at least one of text, images, or video. In embodiments of this application,

[0015] Secondly, this application provides a data processing apparatus, comprising:

[0016] The acquisition module is used to acquire the input sequence;

[0017] The processing module is used to input the input sequence into the large model and obtain the output sequence. The large model includes multiple attention modules. Each attention module is used to determine the size of the attention window from multiple sizes according to the model structure parameters. The attention window includes the input window size when the current region in the input sequence is calculated using the attention mechanism.

[0018] In one possible implementation, the aforementioned attention window includes the range of values ​​in the temporal and / or spatial domains of the input and the number of regions when the attention mechanism is used to calculate the attention for the current region in the input sequence.

[0019] The effects of the second aspect or any alternative implementation of the second aspect can be found in the description of the first aspect or any alternative implementation of the first aspect, and will not be repeated here.

[0020] In one possible implementation, the aforementioned plurality of attention modules include a first attention module, which is specifically configured to: receive a first input subsequence, the first input subsequence being the sequence output by the previous module; determine the size of the current attention window from a variety of sizes based on model structure parameters; calculate the attention matrix based on the size of the current attention window; and output the first output subsequence after post-processing the attention matrix.

[0021] In one possible implementation, the aforementioned attention window is a bidirectional attention window or a unidirectional attention window. A bidirectional attention window is a window that performs attention calculations by taking values ​​in the forward and backward directions in the time and / or spatial dimensions, respectively. A unidirectional attention window is a window that performs attention calculations by taking values ​​in the forward or backward directions in the time and / or spatial dimensions.

[0022] In one possible implementation, the aforementioned processing module is further configured to obtain a query table, which includes multiple sizes, wherein each model structure parameter corresponds to a window size;

[0023] Each attention module is specifically used to query the size of the attention window from the query table based on the model structure parameters.

[0024] In one possible implementation, the aforementioned processing module is specifically used for: acquiring training data; inputting the training data into a large model, dividing the sequence input into each attention module into multiple regions, and determining the attention magnitude between each region in each attention module and other regions; filtering out the range and / or number of regions associated with each region in each attention module based on the attention magnitude between each region in each attention module and other regions; determining the attention window corresponding to each attention module based on the range and / or number of regions associated with each region in each attention module; and constructing a lookup table based on the model structure parameters of each attention module and the attention window corresponding to each attention module.

[0025] In one possible implementation, the aforementioned input sequence includes a sequence obtained based on at least one of text, images, or videos.

[0026] Thirdly, this application provides a data processing apparatus, including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to execute processing-related functions in the data processing method shown in any of the first aspects above. Optionally, the data processing apparatus may be a chip.

[0027] Fourthly, embodiments of this application provide a digital processing chip or chip, the chip including a processing unit and a communication interface, the processing unit obtains program instructions through the communication interface, the program instructions are executed by the processing unit, and the processing unit is used to perform processing-related functions as described in the first aspect or any optional embodiment of the first aspect.

[0028] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any optional implementation thereof.

[0029] In a sixth aspect, embodiments of this application provide a computer program product comprising a computer program / instructions, which, when executed by a processor, causes the processor to perform the method described in the first aspect or any optional implementation thereof. Attached Figure Description

[0030] Figure 1 is a schematic diagram of the architecture of a cloud service system provided in an embodiment of this application;

[0031] Figure 2 is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0032] Figure 3 is a schematic diagram of a model architecture applied in an embodiment of this application;

[0033] Figure 4 is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0034] Figure 5 is a schematic diagram of an attention window size applied in an embodiment of this application;

[0035] Figure 6 is a schematic diagram of another attention window size applied in the embodiments of this application;

[0036] Figure 7 is a schematic diagram of another attention window size applied in the embodiments of this application;

[0037] Figure 8 is a schematic diagram of another attention window size applied in the embodiments of this application;

[0038] Figure 9 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0039] Figure 10 is a schematic diagram of another data processing device provided in an embodiment of this application. Detailed Implementation

[0040] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0041] First, for ease of understanding, some terms or concepts involved in the embodiments of this application will be introduced.

[0042] (1) Neural Network

[0043] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can be represented as follows:

[0044] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x sThe weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer; the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together; that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0045] (2) Deep Neural Networks

[0046] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple intermediate layers. Based on the position of these layers, the internal neural network of a DNN can be divided into three categories: input layer, intermediate layers, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are considered intermediate layers, or hidden layers.

[0047] Although DNNs appear complex, each layer can be represented as a linear relational expression: in, It is the input vector. It is the output vector. is the offset vector, also known as the bias parameter; w is the weight matrix (also called coefficients); and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also quite large. The definitions of these parameters in DNNs are as follows: Taking the coefficient w as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0048] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0049] It's important to note that the input layer does not have a W parameter. In deep neural networks, more intermediate layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0050] (3) Large Model

[0051] Large models are large-scale models. The "large" in large models can be reflected in many aspects, such as large data scale, large-scale parallel computing capabilities, and larger model structures.

[0052] (4) Language model (LM)

[0053] Language models play a crucial role in Natural Language Processing (NLP), where their task is to predict the probability of a sentence occurring in a language. For example, a language model is typically constructed as a probability distribution p(s) of a string s, where p(s) attempts to reflect the frequency of string s as a sentence. It can be applied to scenarios such as text recognition or machine translation. In the embodiments of this application, the NLP models mentioned below include language models.

[0054] (5) Large Language Model (LLM)

[0055] A large language model (LLM) refers to a language model containing hundreds of billions (or more) parameters trained on massive amounts of text data. It is a natural language processing model based on deep learning. These models can process large amounts of text data to learn the grammatical and semantic rules of natural language. LLMs can be applied to text generation, machine translation, question answering systems, text summarization, and sentiment analysis, offering advantages such as strong generative capabilities, high adaptability, accurate prediction, and strong scalability. For example, in movie recommendation scenarios, a large language model can generate descriptions of movie scenes, including genre, main actors, and plot, enabling the system to better recommend similar movies. Large language models can also generate recommendation reasons; for example, e-commerce websites can use large language models to generate reasons for recommending products, such as product quality, price, and features, allowing users to better understand the value of the product.

[0056] (6) transformer

[0057] A transformer architecture is a feature extraction network that includes both an encoder and a decoder (classified as a convolutional neural network). Of course, in some cases, a transformer architecture may not include an encoder but may include a decoder. The aforementioned language models or large language models can be models based on a transformer architecture.

[0058] Encoder: Learns features, such as pixel features, within the global receptive field using self-attention.

[0059] Decoder: Learns the features of the desired module, such as the features of the output box, through self-attention and cross-attention.

[0060] For example, a Transformer layer structure may include an attention network and a feedforward network module. Taking natural language processing as an example, the attention network obtains corresponding weight values ​​by calculating the relevance between words based on the attention mechanism, thus obtaining context-related word representations, which is the core part of the Transformer structure. The feedforward network further transforms the obtained representations to obtain the final output of the Transformer layer. In addition to these two important components, residual layers (ADD) and linear normalization (Norm) are also stacked on these two components to optimize the output of the Transformer layer.

[0061] (7) Tokens

[0062] The input to the transformer model is widely referred to in the industry as tokens, which is a set of fixed-length one-dimensional vectors.

[0063] (8) Attention mechanism

[0064] One computational approach, given three input tensors Q, K, and V (typically all n*d, where d is the vector dimension and n is the number of vectors), first calculates the attention matrix between tensors Q and K. Then, it performs a matrix-vector multiplication operation with the third tensor V to obtain the final output. The main computational cost of this approach comes from calculating the attention matrix, which can be expressed as O(n^2). 2 ).

[0065] Attention mechanisms can quickly extract important features from sparse data. They provide an effective modeling approach for capturing global contextual information through QKV (Queries, Keys, Values). Assuming the input is Q(query), and the context is stored as key-value pairs (K, V), then the attention mechanism is essentially a mapping function from the query to a series of key-value pairs. The essence of the attention function can be described as a mapping from a query to a series of (key-value) pairs. Attention essentially assigns a weight coefficient to each element in the sequence, which can also be understood as soft addressing. If each element in the sequence is stored in (K, V) form, then attention performs addressing by calculating the similarity between Q and K. The similarity calculated between Q and K reflects the importance of the extracted V values, i.e., the weights, and then a weighted sum is obtained to obtain the final feature value.

[0066] Attention calculation mainly consists of three steps. The first step is to calculate the similarity between the query and each key to obtain weights. Common similarity functions include dot product, concatenation, and perceptron. The second step typically uses a softmax function (which can normalize the weights, resulting in a probability distribution where the sum of all weight coefficients is 1, and also highlights the weights of important elements) to normalize these weights. Finally, the weights and their corresponding key values ​​are weighted and summed to obtain the final feature value. The specific calculation formula is as follows:

[0067] Where d is the dimension of matrix Q,K.

[0068] Furthermore, attention includes self-attention and cross-attention. Self-attention can be understood as a special type of attention where the inputs to the QKV features are consistent. Cross-attention, on the other hand, involves inconsistent inputs to the QKV features. Attention integrates the queried features as updated values ​​for the current features using the similarity between features (e.g., inner product) as weights. Self-attention is attention extracted based on the attention drawn from the feature map itself.

[0069] For convolutional networks, the kernel size limits the receptive field, often requiring multiple layers to focus on the entire feature map. Self-attention, on the other hand, has the advantage of global focus; it can obtain global spatial information about the feature map through simple lookups and assignments.

[0070] Secondly, the method provided in this application can be deployed on server clusters, cloud platforms, or other devices with computing capabilities.

[0071] For example, in one scenario, the method provided in this application can be deployed on a cloud platform to provide cloud services to users through a client.

[0072] For example, Figure 1 shows a schematic diagram of the structure of a cloud service system provided in this application. As shown in Figure 1, the cloud service system 10 may include a data center 11, a cloud platform 12, and a client 13.

[0073] The cloud platform 12 may specifically include a server cluster or a cloud platform, or the computing device may be other devices with computing capabilities. Optionally, the cloud platform 12 can work with other computing devices, such as data storage, routers, load balancers, etc. The cloud platform 12 can use data from the data storage system or call program code in the data storage system to implement the method steps provided in the embodiments of this application.

[0074] Data center 11 can be used to store data for cloud platform 12 to query or write data, etc.

[0075] The cloud platform 12 can provide services to users in the form of a client. Users can operate on the client 13 to interact with the cloud platform 12 on data or to request services from the cloud platform 12. This client can be deployed on personal computers, computer workstations, smartphones, tablets, laptops, and smart cars, etc.

[0076] In one possible implementation, the cloud platform 12 is used to implement the method provided in the embodiments of this application. That is, a large model is deployed in the cloud platform 12, the data processing method provided in the embodiments of this application is executed based on the large model, output data is obtained based on the data input by the client, and the output data is sent to the client.

[0077] It should be noted that client 13 is an optional device. For example, the inference process can be executed directly in the cloud platform 12 using the data stored in the data center 11, without the need for client 13.

[0078] With the development of artificial intelligence, the generation and understanding of text, images, and videos are evolving towards more complex video generation. Diffusion transformers (DiT) based on Transformers are the mainstream generative models, possessing excellent scalability. However, their computational cost is quadratic with the length of the input. Videos, compared to images and text, have much longer inputs (H*W becomes T*H*W), so training DiT models for videos often requires significant computational overhead. For example, an existing DiT inference model requires 6144 H100 cards and can only generate video clips with a maximum size of 16fps 10s and a spatial resolution of 768*768. For high-resolution, long-temporal videos with even longer input lengths, the computational cost of training Transformer-based diffusion models is prohibitive.

[0079] Specifically, for example, regarding the three dimensions of video (T*H*W), the key to existing solutions lies in minimizing the number of tokens input to the Transformer model in either the spatial or temporal dimensions. This solution can include the following variations:

[0080] 1. Spatiotemporal full attention mechanism: This means that for each region, attention is calculated between each region and all regions. While the number of input tokens remains unchanged, it implies very high computational overhead, expressed as O(n^2). 2 ),n=THW;

[0081] 2. By performing attention operations only on the spatial dimension to obtain the attention matrix, the number of tokens in the temporal dimension is reduced. The number of input tokens decreases from T*H*W to H*W, and the computational cost is reduced to that of spatiotemporal full attention.

[0082] 3. In addition to performing attention operations only on space, add tokens with some time or space dimensions.

[0083] However, modeling using a spatiotemporal full attention mechanism requires all tokens of the video as input, resulting in very high computational overhead; modeling using a local attention mechanism discards the direct correlation of spatiotemporal information, leading to a decrease in the final effect.

[0084] Therefore, this application provides a data processing method that can be used for modeling based on an adaptive attention window, adaptively adjusting the actual size of the area that needs attention for each region, reducing the amount of computation required to perform modeling, and outputting better data modeling results based on content with higher attention.

[0085] Referring to Figure 2, a flowchart of a data processing method provided in an embodiment of this application is shown below.

[0086] 201. Obtain the input sequence.

[0087] The input sequence can be a one-dimensional or multi-dimensional vector, which can be used to represent the input corpus.

[0088] Specifically, the input sequence can be the output sequence after processing input data such as text, images, audio, or video through a neural network, or it can be a sequence directly input by the user.

[0089] For example, in one possible scenario, if the input data is text, the text can be directly converted into a representation vector using the embedding layer. That is, the vector corresponding to each word in the text can be queried from the embedding vocabulary to obtain the input sequence.

[0090] For example, in one possible scenario, if the input data is an image or video, a visual model can be used to extract features from the image or video data and encode them into vectors to obtain the input sequence.

[0091] For example, in one possible scenario, if the input data is audio data, the audio can be converted into text, and then the text can be converted into a representation vector using an embedding layer to obtain the input sequence.

[0092] For example, in one possible scenario, the large model in this application embodiment can be a multimodal large model, and the input data can include multimodal data. Using the multimodal model, the multimodal data is mapped to the same sequence to obtain the input sequence.

[0093] 202. Input the input sequence into the large model to obtain the output sequence.

[0094] The large model includes multiple attention modules. Each attention module determines the size of the attention window from various options based on the model's structural parameters. The attention window is the size of the input window used to compute attention to the current region in the input sequence. For example, a region can be understood as one or more units in the sequence, typically the unit region for attention calculation by the attention module. Generally, the regions included within this attention window have a higher correlation with the current region than with other regions in the input sequence. Therefore, when computing attention for the current region, more attention is paid to regions more relevant to the current region, resulting in an output sequence more relevant to the current region.

[0095] The output sequence can be used in downstream tasks such as text, image, or video generation or understanding.

[0096] In this embodiment, multiple window sizes can be pre-set, allowing the appropriate attention window size to be determined from among these sizes based on model parameters. Each attention window can represent the portion of the input sequence that requires attention during processing. Therefore, when calculating the attention matrix, it is unnecessary to calculate all regions; only the regions of primary interest need to be calculated, significantly reducing the computational load of model inference.

[0097] Furthermore, the attention window can specifically include the range of values ​​in the temporal and / or spatial domains of the input and the number of regions when calculating attention for the current region in the input sequence using the attention mechanism. The region is the granularity of the attention matrix calculation, such as one or more elements in the sequence. The window size can be the range of regions with high correlation to the current region. Therefore, when calculating the attention matrix, only the part that needs to be paid attention to needs to be calculated, without having to calculate the attention for the part with low attention, which greatly reduces the amount of computation and can maintain the output performance of the model.

[0098] Taking the execution flow of one of the attention modules as an example, the module receives a first input subsequence, which includes the sequence output by the previous module. Then, based on the model structure parameters, the size of the current attention window is determined from multiple sizes. These model structure parameters are the structure parameters of the current attention module, typically parameters determined during training. An attention matrix is ​​calculated based on the size of the current attention window, that is, values ​​are taken from the input sequence according to the attention window size, and the attention score with the current region is calculated, outputting the attention matrix for the current region. After post-processing the attention matrix, a first output subsequence is output. This post-processing may include, but is not limited to, standardization or fusion with the input sequence, depending on the specific application scenario. Therefore, in this embodiment, during inference, the attention module can select an adaptive attention window size based on model parameters to perform attention calculations on regions requiring more attention, extracting the correlation between these regions and the current region.

[0099] When the attention window is a window that includes both temporal and spatial dimensions, the method of this application can combine the correlation within the sequence in the temporal dimension and the correlation within the sequence in the spatial dimension to obtain the output sequence. Therefore, features that are more relevant to each region can be extracted, thereby improving the accuracy of downstream tasks that process the output sequence.

[0100] Optionally, the attention module can determine the size of the attention window using a lookup table, which can include the aforementioned various sizes. The lookup table can be manually set or calculated using training data.

[0101] Optionally, training data is acquired, which may include one or more of text, audio, image, or video data; the training data is input into a large model, and a lookup table is determined based on the output data of the large model; specifically, based on the output results of the large model, the performance results of attention calculation between each region in each attention module and multiple regions in the input sequence are determined; based on the performance results, the regions associated with each region are determined from the multiple regions to obtain the attention window of each attention module; and a lookup table is constructed based on the model structure parameters of each attention module and the attention window.

[0102] For example, when determining the lookup table, the size of the attention window can be unlimited. The attention score between each region and all regions can be calculated, and regions with attention scores higher than a preset score or the k highest scores can be assigned to the corresponding windows. Thus, the lookup table is generated based on the window size of each region. In this embodiment, the region requiring more attention can be selected as the attention window size for each region based on its attention performance with all regions. The attention window size of the attention module is then determined based on the attention window sizes of all regions. For example, the attention window sizes of all regions can be fused to output the attention window size of the attention module. This attention window can cover the attention window size of each region within the module, or it can cover the attention window sizes of most regions. Therefore, during inference, attention calculation can be performed only on the regions requiring more attention, without needing to focus on all regions. This reduces computational load while maintaining the model's output performance.

[0103] The foregoing has described the method flow provided by the embodiments of this application. The method flow provided by the embodiments of this application will be further described below in conjunction with specific application scenarios.

[0104] The method provided in this application can be applied to large models, which may include multiple attention modules, i.e., modules based on attention mechanisms. For example, a possible large model structure is shown in Figure 3, which includes multiple heads, taking the (K-1)th head to the Kth head structure as an example. Each head may include multiple blocks, such as the Nth block to the N+1th block in Figure 3. The output of the previous block serves as the input of the next block, and the structures within the blocks are the same or similar. Using this structure, different data and training loss functions can be designed during the training phase to obtain models for performing tasks such as video generation or video understanding.

[0105] For example, the execution flow of one of the blocks can be shown in Figure 4, where each block may include a preprocessing unit, a spatiotemporal dynamic window calculation unit, an attention matrix calculation unit, and a postprocessing unit.

[0106] The preprocessing module can be used to preprocess the input token, which may include standardization or other preprocessing steps.

[0107] The preprocessed sequence is input into the spatiotemporal dynamic window calculation unit. The spatiotemporal dynamic window calculation unit queries the attention window size according to the model structure parameters and inputs the attention window size into the attention matrix calculation unit.

[0108] The attention matrix calculation unit can be used to perform attention calculations based on the attention window and the input token, and output the attention matrix.

[0109] The post-processing unit can perform post-processing on the input sequence, including standardization or matrix multiplication and vector operations with the sequence output by the pre-processing module.

[0110] For example, taking video data processing as an example, the original video is input into the preprocessing module in the form of a token sequence, and the output is the tensor required to calculate attention. Then, the spatiotemporal dynamic window calculation unit obtains the window size corresponding to the time and space dimensions using the current model's structural parameters and the input. Subsequently, the region where the attention matrix needs to be calculated is determined using the window size in the time and space dimensions, and the calculation process is performed to obtain the calculation result. The final output token sequence is obtained through the token sequence postprocessing unit.

[0111] Furthermore, the specific methods by which the spatiotemporal dynamic window calculation unit and the attention matrix calculation unit are executed can be categorized into several types.

[0112] In terms of dimensions, the attention window can include regions with temporal and / or spatial dimensions. For example, in video generation or video data processing, there is data with temporal and spatial dimensions; when performing attention operations, video generation or processing can be performed from either the temporal or spatial dimensions, thereby obtaining output results that perform better in both dimensions.

[0113] Furthermore, based on the scope covered by the attention window, the attention window can be divided into one-way windows and two-way windows. One-way windows include windows that take values ​​from the forward or backward directions in the time and / or spatial dimensions for attention calculation, while two-way windows include windows that take values ​​from both the forward and backward directions in the time and / or spatial dimensions for attention calculation.

[0114] The following section uses the execution of video-related tasks as an example to introduce the processing flow of the attention window.

[0115] I. Two-way attention window

[0116] The term "bidirectional" here refers to the fact that when calculating attention, attention is calculated both forward and backward based on the current region in terms of time and / or space. This allows attention to be calculated on all parts that may be related to the current region in both forward and backward directions, so as to obtain information that is more relevant to the current region.

[0117] Referring to Figure 4 above, taking the calculation process of the attention matrix in one of the blocks as an example, each spatiotemporal dynamic window calculation unit receives the model structure parameters at the current position and outputs the dynamic window size in the time and space dimensions, representing the bidirectional attention matrix window size.

[0118] Specifically, the model structure parameters of the current region can be determined based on the Block number and Head number of the dynamic window computation unit in the model, and the attention window size can be output based on the input model structure parameters, which can include the time dimension window size and the spatial dimension window size.

[0119] Based on the temporal and spatial window sizes corresponding to each Head, the bidirectional window positions for which the attention matrix needs to be calculated are determined, and the attention score for the corresponding position is obtained through the formula for calculating the attention matrix.

[0120] Optionally, a lookup table can be used to find the attention window size for each head. For example, no window size can be set during training. Different blocks or heads have certain patterns, and the attention performance of different blocks or heads varies in different parts, such as different attention scores. A more suitable window size can be selected based on the attention scores.

[0121] For example, as shown in Table 1, two blocks are listed, each containing three dynamically sized output heads. For instance, the time dimension window size of block 1 in head1 is 3, and the spatial dimension window size is 3; the time dimension window size of block 1 in head2 is 5, and the spatial dimension window size is 3, and so on.

[0122] Table 1

[0123] Accordingly, the attention matrix with a bidirectional dynamic window size of 3 in the time dimension can be shown in Figure 5. Each region or block represents a frame in the time dimension of the video. Taking frame F2 as an example, F1 is the frame before F2 and F3 is the frame after F2. When calculating the attention of F2, attention can be calculated between F1, F2 and F3 and F2 respectively. That is, the three blocks in the time dimension with F2 as the intermediate point are used as the attention window.

[0124] Furthermore, the attention window is expanded spatially. Figure 6 shows a schematic diagram of an attention matrix with a spatially bidirectional dynamic window size of 3. The left image represents the overall token in the spatial dimension of a video frame, and the right image is a schematic diagram of the spatially bidirectional dynamic window. "Bidirectional" refers to the output of the dynamic window calculation unit representing the bidirectional window size, which is reflected in the calculation of the attention matrix by only calculating all positions within the window size before and after the current position (i.e., forward and backward in the spatial dimension). For example, as shown in the right image of Figure 6, during the attention calculation of region 6 in space, attention calculations are performed on the adjacent forward and backward directions of region 6 using windows of size 3.

[0125] After calculating the attention based on the corresponding attention windows for the time and space dimensions respectively, the overall attention matrix can be obtained.

[0126] After traversing all the blocks within the head, the final token can be output. This token can then be used as the output of downstream tasks.

[0127] Therefore, in the embodiments of this application, attention calculation can be performed bidirectionally, both forward and backward, in terms of time and space dimensions, thereby performing attention calculation on more regions that are related to the current region, extracting information that is more related to the current region, and thus improving the accuracy of subsequent downstream tasks.

[0128] II. One-way attention window

[0129] In the aforementioned bidirectional attention window, attention calculations are performed on both the current region and both the forward and backward directions. In the unidirectional attention window, attention calculations are performed on a region selected only in one of the forward or backward directions of the current region.

[0130] Specifically, taking the calculation process of the attention matrix in one of the blocks as an example, each spatiotemporal dynamic window calculation unit receives the model structure parameters at the current position and outputs the dynamic window size in the time and space dimensions, representing the bidirectional attention matrix window size.

[0131] Specifically, the model structure parameters of the current region can be determined based on the Block number and Head number of the dynamic window computation unit in the model, and the attention window size can be output based on the input model structure parameters, which can include the time dimension window size and the spatial dimension window size.

[0132] Based on the temporal and spatial window sizes corresponding to each Head, the bidirectional window positions for which the attention matrix needs to be calculated are determined, and the attention score for the corresponding position is obtained through the formula for calculating the attention matrix.

[0133] Similar to Table 1 above, the attention window size in the temporal or spatial dimensions can also be set for each block in the head. The difference is that when determining the attention window from the input sequence, only the forward or backward region is considered, so the region is taken from the forward or backward direction as the attention window.

[0134] In the process of calculating the attention matrix, the temporal and spatial window sizes corresponding to each Head determine the unidirectional window positions that the attention matrix needs to calculate. The values ​​at the corresponding positions are obtained through the formula for calculating the attention matrix. Here, "unidirectional" refers to the output of the dynamic window calculation unit representing the forward window size. This is reflected in the fact that the attention matrix calculation only calculates all positions before the current position within the window size, as shown in Figure 7 (temporal window size of 3) and Figure 8 (spatial window size of 3). For example, in the attention window shown in Figure 7, in the temporal dimension, frame F1 performs attention calculations with frames F1, F2, and F3 respectively; frame F2 performs attention calculations with frames F2, F3, and F4 respectively, and so on. In the spatial dimension, as shown in Figure 8, the attention window only takes a region in one direction for attention calculation. For example, attention calculations are performed on the adjacent backward directions of region 6 using windows of size 3. Compared to Figure 6, with the same window size, the unidirectional attention window only takes a region in one direction.

[0135] In this embodiment, the region of interest can be selected as a unidirectional attention window in either the temporal or spatial dimensions. Therefore, an adaptive dynamic attention window can be set for different model structures, so that only regions with higher correlation need to be focused when calculating the attention matrix, resulting in less computation of the attention matrix and achieving high-quality modeling results while significantly reducing computational overhead.

[0136] Furthermore, the output sequence can also be applied to downstream tasks. For example, downstream task modules can be added, such as video decoders or text decoders, so as to execute downstream tasks based on a more accurate output sequence.

[0137] The foregoing has described the method flow provided in the embodiments of this application. The following describes the structure of the apparatus for executing the foregoing method flow.

[0138] Referring to Figure 9, a schematic diagram of the structure of a data processing device provided in an embodiment of this application includes:

[0139] Module 901 is used to acquire the input sequence;

[0140] The processing module 902 is used to input the input sequence into the large model to obtain the output sequence. The large model includes multiple attention modules. Each attention module is used to determine the size of the attention window from multiple sizes according to the model structure parameters. The attention window includes the input window size when the current region in the input sequence is calculated using the attention mechanism.

[0141] In one possible implementation, the aforementioned attention window includes the range of values ​​in the temporal and / or spatial domains of the input and the number of regions when the attention mechanism is used to calculate the attention for the current region in the input sequence.

[0142] In one possible implementation, the aforementioned plurality of attention modules include a first attention module, which is specifically configured to: receive a first input subsequence, the first input subsequence being the sequence output by the previous module; determine the size of the current attention window from a variety of sizes based on model structure parameters; calculate the attention matrix based on the size of the current attention window; and output the first output subsequence after post-processing the attention matrix.

[0143] In one possible implementation, the aforementioned attention window is a bidirectional attention window or a unidirectional attention window. A bidirectional attention window is a window that performs attention calculations by taking values ​​in the forward and backward directions in the time and / or spatial dimensions, respectively. A unidirectional attention window is a window that performs attention calculations by taking values ​​in the forward or backward directions in the time and / or spatial dimensions.

[0144] In one possible implementation, the aforementioned processing module 902 is further configured to obtain a query table, which includes multiple sizes, wherein each model structure parameter corresponds to a window size;

[0145] Each attention module is specifically used to query the size of the attention window from the query table based on the model structure parameters.

[0146] In one possible implementation, the aforementioned processing module 902 is specifically used for: acquiring training data; inputting the training data into a large model, dividing the sequence input into each attention module into multiple regions, and determining the attention magnitude between each region in each attention module and other regions; filtering out the range and / or number of regions associated with each region in each attention module based on the attention magnitude between each region in each attention module and other regions; determining the attention window corresponding to each attention module based on the range and / or number of regions associated with each region in each attention module; and constructing a lookup table based on the model structure parameters of each attention module and the attention window corresponding to each attention module.

[0147] In one possible implementation, the aforementioned input sequence includes a sequence obtained based on at least one of text, images, or videos.

[0148] Figure 10 shows a schematic diagram of the hardware structure of a data processing device 100 provided in an embodiment of this application. This data processing device 100 can be used to implement the steps of the methods shown in Figures 2 to 8.

[0149] The data processing device 100 shown in Figure 10 may include a processor 1001, a memory 1002, a communication interface 1003, and a bus 1004. The processor 1001, the memory 1002, and the communication interface 1003 can be connected via the bus 1004.

[0150] The processor 1001 is the control center of the data processing device 100. It can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adapted to the actual application scenario.

[0151] As an example, processor 1001 may include one or more CPUs, and may also include other processors, such as the CPU, NPU or GPU shown in Figure 10.

[0152] The memory 1002 in this application can be non-transitory memory, which can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. It can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0153] In one possible implementation, the memory 1002 may exist independently of the processor 1001. The memory 1002 can be connected to the processor 1001 via a bus 1004 and is used to store data, instructions, or program code. When the processor 1001 calls and executes the instructions or program code stored in the memory 1002, it can implement the methods provided in the embodiments of this application, such as the methods shown in Figures 2 to 8.

[0154] In another possible implementation, the memory 1002 can also be integrated with the processor 1001.

[0155] The communication interface 1003 is used for the data processing device 100 to connect with other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 1003 may include a receiving unit for receiving data and a transmitting unit for sending data.

[0156] Bus 1004 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 10, but this does not indicate that there is only one bus or one type of bus.

[0157] It should be noted that the structure shown in FIG10 does not constitute a limitation on the data processing device 100. In addition to the components shown in FIG10, the data processing device 100 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a data quantization device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0159] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0160] This application also provides a computer-readable storage medium storing a program for training a model or performing inference tasks. When run on a computer, the program causes the computer to execute all or part of the steps in the methods described in the embodiments shown in Figures 2 to 8. It is worth noting that the computer-readable storage medium mentioned in this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0161] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the aforementioned processor or processor functions, and one or more interfaces. When the digital processing chip integrates a memory, it can perform the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the method steps of any one or more of the foregoing embodiments based on the program code stored in the external memory.

[0162] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0163] [Corrected according to Rule 91, 23.10.2025] The target sensing device or target sensing device provided in the embodiments of this application can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the server to execute the method described in the embodiments shown in Figures 2-8 above. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, such as random access memory (RAM).

[0164] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0165] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0167] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0168] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0169] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps in this application does not imply that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved. The division of modules in this application is a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed may be through some ports, and the indirect coupling or communication connection between modules may be electrical or other similar forms, which are not limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules can be selected to achieve the purpose of the solution in this application according to actual needs.

Claims

1. A data processing method, characterized in that, include: Obtain the input sequence; The input sequence is input into a large model to obtain an output sequence. The large model includes multiple attention modules. Each attention module is used to determine the size of the attention window from multiple sizes according to the model structure parameters. The attention window includes the input window size when the current region in the input sequence is calculated using the attention mechanism.

2. The method according to claim 1, characterized in that, The attention window includes the range of values ​​in the temporal and / or spatial domains and the number of regions when calculating attention for the current region in the input sequence using an attention mechanism.

3. The method according to claim 1 or 2, characterized in that, The plurality of attention modules includes a first attention module, which is specifically used for: Receive a first input subsequence, which is the sequence output by the previous module; The size of the current attention window is determined from among the various sizes based on the model structure parameters; Calculate the attention matrix based on the size of the current attention window; The attention matrix is ​​post-processed to output the first output subsequence.

4. The method according to claim 3, characterized in that, The attention window can be a bidirectional attention window or a unidirectional attention window. A bidirectional attention window is a window that performs attention calculations by taking values ​​in the forward and backward directions in the time and / or spatial dimensions, respectively. A unidirectional attention window is a window that performs attention calculations by taking values ​​in the forward or backward directions in the time and / or spatial dimensions.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Obtain a query table, which includes the various sizes, wherein each model structure parameter corresponds to a window size; Each attention module is specifically used to query the size of the attention window from the query table based on the model structure parameters.

6. The method according to claim 5, characterized in that, The process of obtaining the query table includes: Obtain training data; The training data is input into the large model, and the sequence input into each attention module is divided into multiple regions. The attention magnitude between each region in each attention module and other regions is determined. Based on the attention magnitude between each region in each attention module and other regions, the range and / or number of regions associated with each region in each attention module are filtered out; The attention window corresponding to each attention module is determined based on the range and / or number of regions associated with each region in each attention module; The query table is constructed based on the model structure parameters of each attention module and the attention window corresponding to each attention module.

7. The method according to any one of claims 1-6, characterized in that, The input sequence includes sequences obtained based on at least one of text, images, or videos.

8. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire the input sequence; The processing module is used to input the input sequence into a large model to obtain an output sequence. The large model includes multiple attention modules. Each attention module is used to determine the size of the attention window from multiple sizes according to the model structure parameters. The attention window includes the input window size when the current region in the input sequence is calculated using the attention mechanism.

9. The apparatus according to claim 8, characterized in that, The attention window includes the range of values ​​in the temporal and / or spatial domains and the number of regions when calculating attention for the current region in the input sequence using an attention mechanism.

10. The apparatus according to claim 8 or 9, characterized in that, The plurality of attention modules includes a first attention module, which is specifically used for: Receive a first input subsequence, which is the sequence output by the previous module; The size of the current attention window is determined from among the various sizes based on the model structure parameters; Calculate the attention matrix based on the size of the current attention window; The attention matrix is ​​post-processed to output the first output subsequence.

11. The apparatus according to claim 10, characterized in that, The attention window can be a bidirectional attention window or a unidirectional attention window. A bidirectional attention window is a window that performs attention calculations by taking values ​​in the forward and backward directions in the time and / or spatial dimensions, respectively. A unidirectional attention window is a window that performs attention calculations by taking values ​​in the forward or backward directions in the time and / or spatial dimensions.

12. The apparatus according to any one of claims 8-11, characterized in that, The processing module is also used to obtain a query table, which includes the various sizes, wherein each model structure parameter corresponds to a window size; Each attention module is specifically used to query the size of the attention window from the query table based on the model structure parameters.

13. The apparatus according to claim 12, characterized in that, The processing module is specifically used for: Obtain training data; The training data is input into the large model, and the sequence input into each attention module is divided into multiple regions. The attention magnitude between each region in each attention module and other regions is determined. Based on the attention magnitude between each region in each attention module and other regions, the range and / or number of regions associated with each region in each attention module are filtered out; The attention window corresponding to each attention module is determined based on the range and / or number of regions associated with each region in each attention module; The query table is constructed based on the model structure parameters of each attention module and the attention window corresponding to each attention module.

14. The apparatus according to any one of claims 8-13, characterized in that, The input sequence includes sequences obtained based on at least one of text, images, or videos.

15. A data processing apparatus, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the device performs the method as described in any one of claims 1-7.

16. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 7.

17. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 7.

Images