Data processing method, deep learning model, and electronic device
Patent Information
- Application Number
- PCT/CN2025/109925
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-02-21
- Filing Date
- 2025-07-22
- Publication Date
- 2026-08-27
Smart Images
Figure CN2025109925_27082026_PF_FP_ABST
Abstract
Description
A data processing method, a deep learning model, and an electronic device
[0001] This application claims priority to Chinese patent application filed on February 21, 2025, with application number 202510199537.3 and entitled "A Data Processing Method, Deep Learning Model and Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, a deep learning model, and an electronic device. Background Technology
[0003] The Transformer model in artificial intelligence is a deep learning model based on an attention mechanism. The Transformer model uses attention to capture the dependencies between data points in the input sequence, thereby processing different data within the sequence. As user demands increase and the length of the input sequence gradually increases, the computational complexity of the attention mechanism also gradually increases; for example, the computational complexity of the attention mechanism can reach N*k. 2 The rapid increase in computational complexity, with N being the length of the input sequence, means that the attention mechanism requires increasingly more storage and computational resources, making it difficult for Transformer models to run efficiently on terminal devices with limited storage and computing power. Therefore, reducing the computational complexity of the attention mechanism has become an urgent problem to be solved.
[0004] Currently, attention mechanism calculations are performed using cloud servers, and the calculation results are synchronized to terminal devices, enabling Transformer models to run efficiently on terminal devices with limited storage and computing power.
[0005] However, this method relies on the computing power and bandwidth of cloud servers, resulting in significant latency; on the other hand, it increases the risk and cost of data transmission. Summary of the Invention
[0006] This application provides a data processing method, a deep learning model, and an electronic device for improving the computational efficiency of the Transformer model.
[0007] In a first aspect, a data processing method is provided, comprising: segmenting the key-value matrix corresponding to the data to be processed to obtain a first key-value matrix and a second key-value matrix; the key-value matrix includes a key matrix and a value matrix, the first key-value matrix includes a first key matrix and a first value matrix, and the second key-value matrix includes a second key matrix and a second value matrix; determining a first matrix based on the first key-value matrix, a query matrix, a pre-stored matrix, and a first weight matrix; the first weight matrix is the weight corresponding to the first key-value matrix; the pre-stored matrix is determined based on the prompt data; determining a second matrix based on the second key-value matrix, the query matrix, and the second weight matrix; the second weight matrix is the weight corresponding to the second key-value matrix; and calculating the sum of the first matrix and the second matrix to obtain a target matrix.
[0008] In the above technical solution, when calculating the attention mechanism, the key-value matrix corresponding to the data to be processed is split to obtain a first key-value matrix and a second key-value matrix. The first key-value matrix and the second key-value matrix are calculated separately according to different calculation processes, and the calculation results are summed to obtain the target matrix. Compared with the traditional calculation (i.e., the query matrix, key matrix and value matrix corresponding to the data to be processed are multiplied sequentially), the amount of computation and the amount of cached data are reduced. On the other hand, the deep learning model can directly use the pre-stored matrix corresponding to the prompt data for calculation, without having to recalculate the pre-stored matrix corresponding to the prompt data during the attention mechanism process, further reducing the amount of computation. This reduces the computational complexity and storage requirements of the attention mechanism, improves the computational efficiency of the Transformer model, and improves the performance of the Transformer model.
[0009] In any possible implementation of the first aspect, dividing the key-value matrix corresponding to the data to be processed according to a sliding window to obtain a first key-value matrix and a second key-value matrix includes: dividing the key matrix corresponding to the data to be processed according to a sliding window to obtain a first key matrix and a second key matrix, wherein the second key matrix is located within the sliding window; and dividing the value matrix corresponding to the data to be processed according to a sliding window to obtain a first value matrix and a second value matrix, wherein the second value matrix is located within the sliding window. In the above possible implementations, the key matrix and value matrix are divided according to the size of the sliding window, such that the second key matrix and the second value matrix are located within the sliding window. When the deep learning model performs inference on the second key matrix and the second value matrix, the computational complexity of the second key matrix and the second value matrix is fixed to the size of the sliding window and does not increase with the length of the data to be processed, thus reducing the computational complexity of the second key matrix and the second value matrix.
[0010] In any possible implementation of the first aspect, determining the first matrix based on the first key-value matrix, the query matrix, the pre-stored matrix, and the first weight matrix includes: calculating the product of the first key matrix and the first value matrix to obtain a first product matrix; calculating the sum of the first product matrix and the pre-stored matrix to obtain a third matrix; calculating the product of the third matrix and the query matrix to obtain a second product matrix; and calculating the product of the second product matrix and the first weight matrix to obtain the first matrix. The above possible implementation first calculates the product of the first key matrix and the first value matrix, sums the first product matrix obtained by multiplication with the pre-stored matrix, calculates the product of the summed third matrix with the query matrix, and calculates the product of the second product matrix obtained by multiplication with the first weight matrix. By changing the calculation process, the computational and data storage requirements are reduced.
[0011] In any possible implementation of the first aspect, the method further includes: caching the first product matrix in on-chip memory. In the above possible implementations, caching the first product matrix in on-chip memory reduces the bandwidth pressure on the processor's access to off-chip memory, as the calculation result does not leave the processor.
[0012] In any possible implementation of the first aspect, determining the second matrix based on the second key-value matrix, the query matrix, and the second weight matrix includes: calculating the product of the query matrix and the second key matrix to obtain a third product matrix; calculating the product of the third product matrix and the second key matrix to obtain a fourth product matrix; and calculating the product of the fourth product matrix and the second weight matrix to obtain the second matrix. These possible implementations reduce computational and data storage requirements by altering the calculation process.
[0013] In any possible implementation of the first aspect, the data to be processed includes one or more of the following: image data, text data, video data, audio data, or signals. Through these possible implementations, the deep learning model can process different types of data, thus improving its performance.
[0014] In any possible implementation of the first aspect, the method further includes: reading the key-value matrix, the pre-stored matrix, the query matrix, the first weight matrix, and the second weight matrix from off-chip memory. The above possible implementations directly obtain the pre-stored matrix from off-chip memory, eliminating the need to recalculate the pre-stored matrix corresponding to the prompt data during the attention mechanism calculation, thus reducing the computational load and complexity.
[0015] Secondly, a deep learning model is provided, comprising: a segmentation unit for segmenting the key-value matrix corresponding to the data to be processed to obtain a first key-value matrix and a second key-value matrix; the key-value matrix includes a key matrix and a value matrix, the first key-value matrix includes a first key matrix and a first value matrix, and the second key-value matrix includes a second key matrix and a second value matrix; and a calculation unit for: determining a first matrix based on the first key-value matrix, a query matrix, a pre-stored matrix, and a first weight matrix; the first weight matrix is the weight corresponding to the first key-value matrix; the pre-stored matrix is determined based on the prompt data; determining a second matrix based on the second key-value matrix, the query matrix, and the second weight matrix; the second weight matrix is the weight corresponding to the second key-value matrix; and calculating the sum of the first matrix and the second matrix to obtain a target matrix.
[0016] In any possible implementation of the second aspect, the segmentation unit is further configured to: segment the key matrix corresponding to the data to be processed according to the sliding window to obtain a first key matrix and a second key matrix, wherein the second key matrix is located within the sliding window; and segment the value matrix corresponding to the data to be processed according to the sliding window to obtain a first value matrix and a second value matrix, wherein the second value matrix is located within the sliding window.
[0017] In any possible implementation of the second aspect, the computing unit is further configured to: calculate the product of the first key matrix and the first value matrix to obtain a first product matrix; calculate the sum of the first product matrix and the pre-stored matrix to obtain a third matrix; calculate the product of the third matrix and the query matrix to obtain a second product matrix; and calculate the product of the second product matrix and the first weight matrix to obtain a first matrix.
[0018] In any possible implementation of the second aspect, the computing unit is further configured to cache the first product matrix in on-chip memory.
[0019] In any possible implementation of the second aspect, the computing unit is further configured to: calculate the product of the query matrix and the second key matrix to obtain a third product matrix; calculate the product of the third product matrix and the second value matrix to obtain a fourth product matrix; and calculate the product of the fourth product matrix and the second weight matrix to obtain a second matrix.
[0020] In any possible implementation of the second aspect, the data to be processed includes one or more of image data, text data, video data, audio data, and signals.
[0021] In any possible implementation of the second aspect, the model further includes a reading unit; the reading unit is used to read the key-value matrix, the pre-stored matrix, the query matrix, the first weight matrix, and the second weight matrix from off-chip memory.
[0022] Thirdly, an electronic device is provided, comprising off-chip memory and a processor. The off-chip memory is used to store a key-value matrix, a pre-stored matrix, a query matrix, a first weight matrix, and a second weight matrix corresponding to the data to be processed. The processor is used to execute the data processing method provided by the first aspect or any possible implementation thereof.
[0023] Fourthly, a computer-readable storage medium is provided, wherein a computer program or instructions are stored therein, which, when executed, implement the data processing method provided by the first aspect or any possible implementation thereof.
[0024] Fifthly, a computer program product is provided, comprising: a computer program, also known as code or instructions, which, when run, causes a computer to perform a data processing method as provided in the first aspect or any possible implementation thereof.
[0025] Understandably, the beneficial effects that can be achieved by the second to fifth aspects mentioned above can be referred to in the context of the first aspect or any possible implementation of the first aspect, and will not be repeated here. Attached Figure Description
[0026] Figure 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;
[0027] Figure 2 is a schematic diagram of a data processing flow provided in an embodiment of this application;
[0028] Figure 3 is a schematic diagram of determining the key matrix Ki0 and the value matrix Vi0 according to an embodiment of this application;
[0029] Figure 4 is a schematic diagram of determining the query matrix Qi, the key matrix Kia, and the value matrix Via according to an embodiment of this application;
[0030] Figure 5 is a schematic diagram of determining the first weight matrix A1i and the second weight matrix A2i according to an embodiment of this application;
[0031] Figure 6 is a schematic diagram of a key value matrix segmentation method provided in an embodiment of this application;
[0032] Figure 7 is a schematic diagram of a data processing method provided in an embodiment of this application;
[0033] Figure 8 is a schematic diagram of the structure of a deep learning model provided in an embodiment of this application. Detailed Implementation
[0034] The following sections will discuss the fabrication and use of various embodiments in detail. However, it should be understood that many applicable inventive concepts provided in this application can be implemented in a variety of specific environments. The specific embodiments discussed are merely illustrative of specific ways of implementing and using this application and technology, and do not limit the scope of this application.
[0035] Unless otherwise defined, all technical terms used herein have the same meaning as commonly known to one of ordinary skill in the art.
[0036] Each circuit or other component may be described or referred to as "for" performing one or more tasks. In this context, "for" is used to imply a structure by indicating that the circuit / component includes a structure (e.g., a circuit system) that performs one or more tasks during operation. Therefore, even when the specified circuit / component is currently inoperable (e.g., not turned on), it can still be referred to as "for performing that task." Circuits / components used with the term "for" include hardware, such as circuits that perform operations.
[0037] The technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings. In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, a, b, and c; where a, b, and c can be single or multiple.
[0038] The embodiments of this application use terms such as "first" and "second" to distinguish objects with similar names, functions, or roles. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order.
[0039] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.
[0040] Before introducing the embodiments of this application, we will first introduce and explain the relevant knowledge of deep learning models.
[0041] The Transformer model is a deep learning model based on an attention mechanism. It uses attention to capture the dependencies between data points in the input sequence, thus processing different data within the sequence. As user demands increase and the length of the input sequence gradually increases, the computational complexity of the attention mechanism also gradually increases; for example, the computational complexity of the attention mechanism can reach N*k. 2 The rapid increase in computational complexity, with N being the length of the input sequence, means that the attention mechanism requires increasingly more storage and computational resources, making it difficult for Transformer models to run efficiently on terminal devices with limited storage and computing power. Therefore, reducing the computational complexity of the attention mechanism has become an urgent problem to be solved.
[0042] In one possible embodiment, the attention mechanism is computed using a cloud server, and the computation results are synchronized to the terminal device, thereby enabling the Transformer model to run efficiently on terminal devices with limited storage and computing power. The terminal device can also be referred to as an edge device.
[0043] However, this method relies on the computing power and bandwidth of cloud servers, resulting in significant latency; on the other hand, it increases the risk and cost of data transmission.
[0044] In one possible implementation, some of the functionality of the Transformer model is implemented on the terminal device. For example, the size and computational cost of the Transformer model can be reduced through techniques such as model pruning and model quantization, enabling the Transformer model to run on terminal devices with limited storage and computing power.
[0045] However, terminal devices can only implement some of the functions of the Transformer model, not all of them, which leads to a decrease in the performance of the Transformer model running on the terminal device and makes it unable to handle complex tasks.
[0046] In one possible implementation, the Transformer model can be run on the terminal device by dynamically adjusting the computing and storage resources in the terminal device. However, the design of a dynamic adjustment strategy is complex, and the computational overhead during the adjustment process is significant.
[0047] Based on this, embodiments of this application relate to a data processing method to solve the problem that Transformer models are difficult to run efficiently on terminal devices with limited storage and computing power, thereby improving the computational efficiency of Transformer models and enabling Transformer models to run efficiently on terminal devices with limited storage and computing power.
[0048] The method provided in this application embodiment can be applied to various electronic devices, which include a deep learning model that is combined with existing deep learning frameworks (such as TensorFlow Lite and PyTorch Mobile) to form a lightweight deep learning model. Optionally, the electronic device includes terminal devices, which may include, but are not limited to: mobile phones, tablets, laptops, PDAs, mobile internet devices (MIDs), cameras, wearable devices (such as smartwatches, smart bracelets, pedometers, etc.), audio equipment, audio and video players, set-top boxes, game consoles, printers, mice, keyboards, in-vehicle equipment (e.g., equipment on vehicles such as cars, airplanes, ships, trains, and high-speed trains), virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, smart home devices (e.g., refrigerators, televisions, air conditioners, electricity meters, etc.), smart robots, workshop equipment, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or smart homes. Wireless terminals (e.g., intelligent robots, hot air balloons, drones, airplanes) in the home. Optionally, this deep learning model and method can also be applied to non-terminal devices, such as computing devices like servers; this embodiment is not limited to this.
[0049] The following example uses a mobile phone as an example to illustrate the structure of the electronic device. As shown in Figure 1, the electronic device may include components such as a radio frequency (RF) circuit 110, a memory 120, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a processor 170, and a power supply 180.
[0050] RF circuit 110 can be used to send and receive information, or to receive or send signals during a call. Specifically, it receives downlink information from the base station and processes it in processor 170; additionally, it sends uplink data to the base station. Typically, RF circuit 110 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 110 can also communicate wirelessly with networks and other devices.
[0051] The memory 120 can be used to store data, software programs, and modules; it mainly includes a program storage area and a data storage area. The program storage area can store the operating system and at least one application program required for a function, such as sound playback or image playback. The data storage area can store data created based on the use of the electronic device, such as audio data, image data, and a phone book. Furthermore, the electronic device may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. In this embodiment, the memory 120 includes off-chip memory and on-chip memory. Off-chip memory refers to memory that is not located on the same chip or on-chip system as the processor 170, i.e., it is located outside the on-chip system. On-chip memory refers to memory that is located on the same chip or on-chip system as the processor 170, i.e., it is located on the on-chip system. For example, on-chip memory may include caches, buffers, and registers. Off-chip memory can be dynamic random access memory (DRAM).
[0052] The input unit 130 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the electronic device. Specifically, the input unit 130 may include a touch screen 131 and other input devices 132. The touch screen 131 can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch screen), and drive the corresponding connected devices according to a pre-set program. Optionally, other input devices 132 may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power switch buttons, etc.), a trackball, a mouse, a joystick, etc.
[0053] Display unit 140 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device. In one example, display unit 140 may include display screen 141, which may be configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar device. Furthermore, touchscreen 131 may cover display screen 141. When touchscreen 131 detects a touch operation on or near it, it transmits the information to processor 170 to determine the type of touch event. Subsequently, processor 170 provides corresponding visual output on display screen 141 based on the type of touch event. Although in the figures, touchscreen 131 and display screen 141 are shown as two separate components to implement the input and output functions of the electronic device, in some embodiments, touchscreen 131 and display screen 141 can be integrated to achieve the input and output functions of the electronic device.
[0054] Sensor 150 may include one or more sensors for providing status assessments of various aspects of the electronic device. Sensor 150 may include a light sensor, which can be used in imaging applications, i.e., as a component of a camera or video camera. Furthermore, sensor 150 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor. Sensor 150 can detect acceleration / deceleration, orientation, on / off state, relative positioning of components, or temperature changes of the electronic device.
[0055] Audio circuitry 160, a speaker, and a microphone provide an audio interface between the user and the electronic device. Audio circuitry 160 converts received audio data into electrical signals and transmits them to the speaker, where the speaker converts them into sound signals for output. On the other hand, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 160, converted into audio data, and output to RF circuitry 110 for transmission to, for example, another mobile phone, or to memory 120 for further processing.
[0056] Processor 170 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in memory 120, and by calling data stored in memory 120, thereby providing overall monitoring of the electronic device. Processor 170 may reside on one or more chips. Optionally, processor 170 may include one or more processing units, which may include, but are not limited to: central processing unit (CPU), network processing unit (NPU), graphics processing unit (GPU), image signal processor (ISP), tensor processing unit (TPU), data processing unit (DPU), digital signal processor (DSP), microcontroller, or microprocessor. Furthermore, processor 170 may also include other hardware circuits or accelerators, such as application-specific integrated circuits (ASICs), complex programmable logic devices (CPLDs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. Optionally, the processor 170 may also be a combination of components that implement computing functions, such as a combination of one or more microprocessors, a digital signal processor, and a microprocessor. In this embodiment, the processor 170 includes a deep learning model. This deep learning model may run on an NPU, GPU, or TPU, and this embodiment does not impose any limitations on this.
[0057] The electronic device may also include a power supply 180 (e.g., a battery) to power various components. The power supply 180 can be logically connected to the processor 170 via a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Optionally, the power management system can simultaneously support fast charging and non-fast charging technologies. In practical applications, the power management system can charge the battery in the power supply 180 using either fast charging or non-fast charging technologies.
[0058] Although not shown, the electronic device may also include a wireless fidelity (WiFi) module, a Bluetooth module, etc., which will not be described in detail in the embodiments of this application. Those skilled in the art will understand that the structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0059] The data processing method provided in this application can be applied to scenarios that require real-time processing of large amounts of data, such as autonomous driving and intelligent monitoring (e.g., drone monitoring).
[0060] Figure 2 is a schematic flowchart of a data processing method provided in an embodiment of this application. This method can be applied to a processor, which can be the processor 170 shown in Figure 1 above. The processor has a deep learning model configured on it, or the processor can load the deep learning model from off-chip memory. The deep learning model can run in the NPU, GPU, or TPU of the processor 170. The deep learning model can include a Transformer model; that is, the method provided in this embodiment of the application is applied to a deep learning model in the processor. The method includes the following steps.
[0061] S201: The deep learning model divides the key-value matrix corresponding to the data to be processed according to the sliding window to obtain the first key-value matrix and the second key-value matrix. The second key-value matrix is located within the sliding window. The key-value matrix includes a key matrix and a value matrix. The first key-value matrix includes a first key matrix and a first value matrix. The second key-value matrix includes a second key matrix and a second value matrix.
[0062] The data to be processed may include any one or more of image data, text data, video data, audio data, or signals. Audio data may include natural language, and signals may include millimeter-wave signals and radar signals, etc.
[0063] In one possible embodiment, if the data to be processed is image data, the method provided in this application embodiment can be used to optimize feature aggregation of convolutional neural networks; if the data to be processed is natural language, the method provided in this application embodiment can be used to optimize the attention mechanism of the Transformer model. Optionally, the method provided in this application embodiment is also used to reduce the computational complexity of acoustic models in speech recognition.
[0064] In addition, the sliding window refers to the sliding window that the deep learning model uses during inference. For example, the size of the sliding window is fixed, and the specific data can be set according to actual needs or the experience of relevant personnel. For example, the size of the sliding window can be two tokens. Tokens are the basic units for data processing by the deep learning module. This application does not specifically limit this aspect.
[0065] In practical applications, deep learning models receive input information from users, which includes cue data (e.g., cue words) and data to be processed. Cue data guides the deep learning model's thinking (e.g., guiding it to think and reason more deeply when generating target data), limits the model's output range, and clarifies the processing task. The deep learning model determines a first input sequence corresponding to the cue data and a second input sequence corresponding to the data to be processed based on the input information. The first input sequence is an M×d matrix, and the second input sequence is an N×d matrix; that is, the first input matrix contains M×d tokens, and the second input matrix contains N×d tokens. M represents the length of the first input sequence, N represents the length of the second input sequence, and d represents the matrix dimension (e.g., d = 512). M, N, and d are all positive integers greater than 1. Optionally, the size of M can be fixed, meaning the length of the first input sequence is fixed, and consequently, the number of data items included in the cue data is fixed. The length of the second input sequence is positively correlated with the number of data points in the data to be processed. For example, the more data points the data to be processed contains, the longer the second input sequence will be; conversely, the fewer data points the data to be processed contains, the shorter the second input sequence will be. N can also represent the number of data points in the data to be processed, i.e., the number of data points included in the second input sequence.
[0066] Furthermore, the deep learning model determines the pre-stored matrix Wi corresponding to the first input sequence based on the linear transformation matrix (including the key matrix Wk and the value matrix Wv). Specifically, as shown in Figure 3, assuming that both the key matrix Wk and the value matrix Wv in the linear transformation matrix are m×d matrices, where m is a positive integer greater than 1, the deep learning model calculates the product of the first input sequence and the key matrix Wk in the linear transformation matrix to obtain the key matrix Ki0 corresponding to the first input sequence, and calculates the product of the first input sequence and the value matrix Wv in the linear transformation matrix to obtain the value matrix Vi0 corresponding to the first input sequence. Here, both the key matrix Ki0 and the value matrix Vi0 corresponding to the first input sequence are M×d matrices. The deep learning model transposes the key matrix Ki0 to obtain the d×M transpose matrix Ki0. T And calculate the transpose matrix Ki0. TThe product of the sum matrix Vi0 is used to obtain the pre-stored matrix Wi, which is a d×d matrix. The deep learning model stores this pre-stored matrix Wi in off-chip memory.
[0067] Optionally, the linear transformation matrix also includes a query matrix Wq. The deep learning model determines the key-value matrix KVi and query matrix Qi corresponding to the second input sequence based on the linear transformation matrix. The key-value matrix KVi includes a key matrix Kia and a value matrix Via. Specifically, assuming the query matrix Wq, key matrix Wk, and value matrix Wv are all m×d matrices, as shown in Figure 4, the deep learning model calculates the product of the second input sequence and the query matrix Wq in the linear transformation matrix to obtain the query matrix Qi; it calculates the product of the second input sequence and the key matrix Wk in the linear transformation matrix, and transposes the result to obtain the key matrix Kia; it calculates the product of the second input sequence and the value matrix Wv in the linear transformation matrix to obtain the value matrix Via. Here, the query matrix Qi and the value matrix Via are both N×d matrices, and the key matrix Kia is a d×N matrix. The deep learning model stores the key-value matrix KVi and the query matrix Qi in off-chip memory. The key matrix Kia can also be represented as Kia. T .
[0068] In one possible embodiment, the deep learning model determines a first weight matrix A1i and a second weight matrix A2i based on a second input sequence, a linear transformation matrix, and an activation function. Specifically, the sum of the query matrix Wq, the key matrix Wk, and the value matrix Wv in the linear transformation matrix is calculated to obtain an m×d linear matrix. For example, as shown in Figure 5, the product of the second input sequence and the linear matrix is calculated to obtain an N×d intermediate matrix; the product of the N×d intermediate matrix and the N×2 activation function matrix corresponding to the activation function is calculated to obtain an N×2 weight matrix. The first column of the weight matrix is used as the first weight matrix A1i, and the second column is used as the second weight matrix A2i. Both the first weight matrix A1i and the second weight matrix A2i are N×1 column matrices.
[0069] In this model, the first weight matrix A1i represents the weights corresponding to the first key-value matrix KVi1, and the second weight matrix A2i represents the weights corresponding to the second key-value matrix KVi2. The deep learning model stores the first weight matrix A1i and the second weight matrix A2i in off-chip memory.
[0070] In one possible embodiment, when the deep learning model needs to perform attention mechanism calculations, the deep learning model reads the key-value matrix KVi (including the key matrix Kia and the value matrix Via), the pre-stored matrix Wi, the query matrix Qi, the first weight matrix A1i, and the second weight matrix A2i from off-chip memory. The deep learning module partitions the key-value matrix KVi according to a sliding window (i.e., according to the size of the sliding window) to obtain the first key-value matrix KVi1 and the second key-value matrix KVi2. The first key-value matrix KVi1 includes the first key matrix Kia1 and the first value matrix Via1; the second key-value matrix KVi2 includes the second key matrix Kia2 and the second value matrix Via2.
[0071] The specific process of splitting the key-value matrix KVi will be explained below.
[0072] In one possible embodiment, the deep learning model divides the key matrix Kia in the key value matrix KVi along the column direction according to the size of the sliding window to obtain a first key matrix Kia1 and a second key matrix Kia2. The first key matrix Kia1 is the matrix in the key matrix Kia other than the second key matrix Kia2, that is, the matrix in the key matrix Kia located outside the sliding window. The second key matrix Kia2 is the matrix in the key matrix Kia located within the sliding window. The column direction is the direction in which the columns of the key matrix Kia are located.
[0073] In one possible embodiment, the deep learning model partitions the value matrix Via in the key matrix KVi along the row direction according to the size of the sliding window to obtain a first value matrix Via1 and a second value matrix Via2. The first value matrix Via1 is the matrix in the value matrix KVi excluding the second value matrix Via2, that is, the first value matrix Via1 is the matrix in the value matrix KVi located outside the sliding window, and the second key matrix Kia2 is the matrix in the value matrix KVi located inside the sliding window; the row direction is the direction in which the rows of the key matrix Kia are located.
[0074] For example, as shown in Figure 6, the key matrix Kia is a d×N matrix, the value matrix Via is an N×d matrix, and L represents the size of the sliding window. The d×N key matrix Kia is partitioned along the column direction according to the size of the sliding window to obtain a d×(NL) first key matrix Kia1 and a d×L second key matrix Kia2. Similarly, the N×d value matrix Via is partitioned along the row direction according to the size of the sliding window to obtain a (NL)×d first value matrix Via1 and an L×d second value matrix Via2. L is less than N, and in practical applications, L is much smaller than N.
[0075] Among them, the first bond matrix Kia1 can be called the long-range bond matrix, and the second bond matrix Kia2 can also be called the short-range bond matrix; the first value matrix Via1 can also be called the long-range value matrix, and the second value matrix can also be called the short-range value matrix.
[0076] S202: The deep learning model determines the first matrix B1 based on the first key-value matrix KVi1, the query matrix Qi, the pre-stored matrix Wi, and the first weight matrix A1i.
[0077] In one possible embodiment, the deep learning model computes the product of the first bond matrix Kia1 of d×(NL) and the first value matrix Via1 of (NL)×d to obtain the first product matrix KVi1 of d×d.
[0078] In one possible embodiment, the deep learning model caches the first product matrix KVi1 in on-chip memory, which may include caches, buffers, and registers. In this embodiment, the deep learning model stores the calculated first product matrix KVia1 in on-chip memory, eliminating the need to transfer the calculation result to off-chip memory and reducing the processor's access bandwidth pressure.
[0079] Furthermore, the deep learning model calculates the sum of the first product matrix KVi1 and the pre-stored matrix Wi in d×d to obtain the third matrix B2 in d×d; calculates the product of the third matrix B2 and the query matrix Qi in N×d to obtain the second product matrix B3 in d×d; and calculates the product of the second product matrix B3 and the first weight matrix A1i in N×1 to obtain the first matrix B1 in d×1.
[0080] S203: The deep learning model determines the second matrix B4 based on the second key-value matrix KVi2, the query matrix Qi, and the second weight matrix A2i.
[0081] In one possible embodiment, the deep learning model calculates the product of an N×d query matrix Qi and a d×L second key matrix Kia2 to obtain an N×L third product matrix QKi; calculates the product of the third product matrix QKi and the L×d second value matrix Via2 to obtain an N×d fourth product matrix QKVi; and calculates the product of the fourth product matrix QKVi and the N×1 second weight matrix A2i to obtain an N×1 second matrix B4.
[0082] S204: The deep learning model calculates the sum of the first matrix B1 and the second matrix B4 to obtain the target matrix.
[0083] The data in the target matrix is the target data, which is the target data output by the deep learning model based on the data to be processed.
[0084] For ease of understanding, the specific process of the data processing method provided in the embodiments of this application will be described below with reference to Figure 4.
[0085] For example, as shown in Figure 7, assuming the sliding window size is L, the method provided in this embodiment includes the following steps S701-S710. S701: Divide the key matrix Kia along the column direction of the key matrix Kia according to the sliding window size L to obtain a first key matrix Kia1 and a second key matrix Kia2 (i.e., divide the key matrix Kia). Wherein, the key matrix Kia is a d×N matrix, the first key matrix Kia1 is a d×(NL) matrix, and the second key matrix is a d×L matrix. S702: Divide the value matrix Via along the row direction of the value matrix Via according to the sliding window size L to obtain a first value matrix Via1 and a second value matrix Via2 (i.e., divide the value matrix Via). Wherein, the value matrix Via is an N×d matrix, the first value matrix Via1 is a (NL)×d matrix, and the second value matrix Via2 is an L×d matrix. S703: Calculate the product of the first key matrix Kia1 and the first value matrix Via1 to obtain a d×d first product matrix KVi1. S704: Calculate the sum of the first product matrix KVi1 and the pre-stored matrix Wi in d×d form to obtain the third matrix B2 in d×d form. S705: Calculate the product of the third matrix B2 and the query matrix Qi in N×d form to obtain the second product matrix B3 in d×d form. S706: Calculate the product of the second product matrix B3 and the first weight matrix A1i in N×1 form to obtain the first matrix B1 in d×1 form. S707: Calculate the product of the query matrix Qi in N×d form and the second key matrix Kia2 in d×L form to obtain the third product matrix QKi in N×L form. S708: Calculate the product of the third product matrix QKi and the second value matrix Via2 in L×d form to obtain the fourth product matrix QKVi in N×d form. S709: Calculate the product of the fourth product matrix QKVi and the second weight matrix A2i in d×d form to obtain the second matrix B4 in N×1 form. S710: Calculate the sum of the first matrix B1 and the second matrix B4 to obtain the target matrix.
[0086] The data processing method provided in this application embodiment divides the key-value matrix corresponding to the data to be processed according to the size of the sliding window to obtain a first key-value matrix and a second key-value matrix. Different calculation processes are performed on different key-value matrices, and the first product matrix KVi1 obtained by multiplying the first key matrix and the first value matrix during the calculation process is cached in on-chip memory. Compared with traditional calculation (i.e., performing the product calculation of the query matrix, key matrix and value matrix sequentially on the input sequence and caching the calculation results of each step in off-chip memory), the amount of computation and the amount of cached data are reduced. On the other hand, in this application embodiment, the deep learning model in the processor directly obtains the pre-stored matrix corresponding to the prompt data from off-chip memory, without having to recalculate the pre-stored matrix corresponding to the prompt data when calculating the attention mechanism, further reducing the amount of computation, thereby reducing the computational complexity and storage requirements, improving the real-time performance and response speed of the attention module, and improving the performance of the Transformer model.
[0087] The foregoing primarily describes the solutions provided in this application from the perspective of deep learning models. It is understood that, to achieve the aforementioned functions, deep learning models include corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0088] This application embodiment can divide the deep learning model into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. The following description uses the division of each functional module according to its own function as an example.
[0089] When using integrated units, deep learning models can be implemented in software. For example, a deep learning model can include multiple virtual modules that can be used to implement different functions within the deep learning model. Each virtual module can include software code.
[0090] Figure 8 shows a schematic diagram of the structure of a deep learning model involved in the above embodiments. The deep learning model may include a segmentation unit 801 and a computation unit 802. The segmentation unit 801 is used to support the deep learning model in executing S201 in the above method embodiments and / or other steps in the embodiments of this application; the computation unit 802 is used to support the deep learning model in executing S202 and S203 in the above method embodiments and / or other steps in the embodiments of this application.
[0091] Optionally, the deep learning model may also include a reading unit 803, which is used to read the key-value matrix, pre-stored matrix, query matrix, first weight matrix and second weight matrix corresponding to the data to be processed from off-chip memory.
[0092] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here in the embodiments of this application.
[0093] In another embodiment of this application, an electronic device is also provided, which includes a processor and off-chip memory; wherein the off-chip memory is used to store a key-value matrix, a pre-stored matrix, a query matrix, a first weight matrix and a second weight matrix corresponding to the data to be processed, and the processor is used to execute the steps in the method embodiments provided above.
[0094] It is understood that all relevant content of each step involved in the above method embodiments can be referenced in the embodiments of the electronic device, and the embodiments of this application will not be repeated here.
[0095] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. This readable storage medium may include various media capable of storing program code, such as a USB flash drive, external hard drive, read-only memory, random access memory, magnetic disk, or optical disk. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product.
[0096] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed.
[0097] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0098] In another embodiment of this application, a readable storage medium is also provided, which stores computer-executable instructions. When a device (which may be a microcontroller, chip, etc.) or a processor executes the computer-executable instructions, the device performs the steps in the above method embodiments.
[0099] In another embodiment of this application, a computer program product is also provided, the computer program product including computer instructions stored in a readable storage medium; at least one processor of the device can read the computer instructions from the readable storage medium, and the at least one processor executes the computer instructions to cause the device to perform the steps in the above method embodiments.
[0100] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A data processing method, characterized in that, The method includes: The key-value matrix corresponding to the data to be processed is split to obtain a first key-value matrix and a second key-value matrix; the key-value matrix includes a key matrix and a value matrix, the first key-value matrix includes a first key matrix and a first value matrix, and the second key-value matrix includes a second key matrix and a second value matrix; A first matrix is determined based on the first key-value matrix, the query matrix, the pre-stored matrix, and the first weight matrix; the first weight matrix is the weight corresponding to the first key-value matrix; the pre-stored matrix is determined based on the prompt data. The second matrix is determined based on the second key-value matrix, the query matrix, and the second weight matrix; the second weight matrix is the weight corresponding to the second key-value matrix. The sum of the first matrix and the second matrix is calculated to obtain the target matrix.
2. The method according to claim 1, characterized in that, The process of splitting the key-value matrix corresponding to the data to be processed to obtain a first key-value matrix and a second key-value matrix includes: The key matrix corresponding to the data to be processed is divided according to the sliding window to obtain the first key matrix and the second key matrix, wherein the second key matrix is located within the sliding window; The value matrix corresponding to the data to be processed is divided according to the sliding window to obtain the first value matrix and the second value matrix, wherein the second value matrix is located within the sliding window.
3. The method according to claim 1 or 2, characterized in that, The step of determining the first matrix based on the first key-value matrix, the query matrix, the pre-stored matrix, and the first weight matrix includes: Calculate the product of the first key matrix and the first value matrix to obtain the first product matrix; Calculate the sum of the first product matrix and the pre-stored matrix to obtain the third matrix; Calculate the product of the third matrix and the query matrix to obtain the second product matrix; The product of the second product matrix and the first weight matrix is calculated to obtain the first matrix.
4. The method according to claim 3, characterized in that, The method further includes: The first product matrix is cached in on-chip memory.
5. The method according to any one of claims 1-4, characterized in that, The step of determining the second matrix based on the second key-value matrix, the query matrix, and the second weight matrix includes: Calculate the product of the query matrix and the second key matrix to obtain the third product matrix; Calculate the product of the third product matrix and the second value matrix to obtain the fourth product matrix; The product of the fourth product matrix and the second weight matrix is calculated to obtain the second matrix.
6. The method according to any one of claims 1-5, characterized in that, The data to be processed includes any one or more of the following: image data, text data, video data, audio data, or signals.
7. The method according to any one of claims 1-6, characterized in that, The method further includes: Read the key-value matrix, the pre-stored matrix, the query matrix, the first weight matrix, and the second weight matrix from off-chip memory.
8. A deep learning model, characterized in that, The model includes: A segmentation unit is used to segment the key-value matrix corresponding to the data to be processed to obtain a first key-value matrix and a second key-value matrix; the key-value matrix includes a key matrix and a value matrix, the first key-value matrix includes a first key matrix and a first value matrix, and the second key-value matrix includes a second key matrix and a second value matrix; and The calculation unit is configured to: determine a first matrix based on the first key-value matrix, the query matrix, the pre-stored matrix, and the first weight matrix; the first weight matrix is the weight corresponding to the first key-value matrix; the pre-stored matrix is determined based on the prompt data; The second matrix is determined based on the second key-value matrix, the query matrix, and the second weight matrix; the second weight matrix is the weight corresponding to the second key-value matrix. The sum of the first matrix and the second matrix is calculated to obtain the target matrix.
9. The model according to claim 8, characterized in that, The segmentation unit is also used for: The key matrix corresponding to the data to be processed is divided according to the sliding window to obtain the first key matrix and the second key matrix, wherein the second key matrix is located within the sliding window; The value matrix corresponding to the data to be processed is divided according to the sliding window to obtain the first value matrix and the second value matrix, wherein the second value matrix is located within the sliding window.
10. The model according to claim 8 or 9, characterized in that, The computing unit is also used for: Calculate the product of the first key matrix and the first value matrix to obtain the first product matrix; Calculate the sum of the first product matrix and the pre-stored matrix to obtain the third matrix; Calculate the product of the third matrix and the query matrix to obtain the second product matrix; The product of the second product matrix and the first weight matrix is calculated to obtain the first matrix.
11. The model according to claim 10, characterized in that, The computing unit is also configured to cache the first product matrix in on-chip memory.
12. The model according to any one of claims 8-11, characterized in that, The computing unit is also used for: Calculate the product of the query matrix and the second key matrix to obtain the third product matrix; Calculate the product of the third product matrix and the second value matrix to obtain the fourth product matrix; The product of the fourth product matrix and the second weight matrix is calculated to obtain the second matrix.
13. The model according to any one of claims 8-12, characterized in that, The data to be processed includes any one or more of image data, text data, video data, audio data, and signals.
14. The model according to any one of claims 8-12, characterized in that, The model also includes a reading unit; The reading unit is used to read the key-value matrix, the pre-stored matrix, the query matrix, the first weight matrix, and the second weight matrix from off-chip memory.
15. An electronic device, characterized in that, The electronic device includes off-chip memory and a processor. The off-chip memory is used to store the key-value matrix, pre-stored matrix, query matrix, first weight matrix and second weight matrix corresponding to the data to be processed. The processor is used to execute the data processing method as described in any one of claims 1-7.
16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on the device, cause the device to perform the data processing method as described in any one of claims 1-7.
17. A computer program product, characterized in that, The computer program product includes a computer program that, when run on a device, causes the device to perform the data processing method as described in any one of claims 1-7.