Matrix computation device

Abstract

A matrix computation device disposed on a chip comprises a configuration packet splitting unit, a combining unit, a memory capable of receiving and storing matrix operands and transmitting same to a matrix multiplication unit capable of multiplying matrix operands on the basis of a received configuration packet, said matrix multiplication unit comprising a systolic array, a matrix operand element delay unit, a buffer memory and a local control unit capable of controlling the order of transmission of matrix operand elements to the systolic array and further capable of controlling the transmission of data from the systolic array to an external device. The technical result consists in reducing the energy consumption of the matrix computation device.

Description

MATRIX CALCULATION DEVICE AREA OF TECHNOLOGY

[0001] The claimed technical solution generally relates to the field of microprocessors, and in particular to a device for matrix calculations. LEVEL OF TECHNOLOGY

[0002] Today, the modern world performs a large number of data operations. With the development and implementation of machine learning algorithms in all areas of life, the need has arisen for devices capable of effectively handling such algorithms, and neural networks in particular. Some of the key operations performed during inference (the process of obtaining neural network output) and training neural networks are matrix manipulation operations, such as matrix multiplication. General-purpose processors, due to their architecture, are not suited for performing machine learning processes such as inference and neural network training at high speed, thus being ineffective for such operations. Therefore, in recent years, the development of specialized devices, such as matrix computing devices, vector processors, and other devices specifically designed for parallel computing, has become particularly important.

[0003] Matrix computing devices are designed to perform complex matrix operations, such as as part of an inference accelerator and neural network training unit. Matrix operations are used to implement neural networks, which underlie machine learning algorithms, such as computer vision algorithms, speech and audio recognition algorithms, natural language processing, and so on. Matrix operations (matrix multiplication, convolution operations) are used to implement fundamental neural network operations, such as updating neural network weights. Therefore, the speed of machine learning processes directly depends on the speed of matrix calculations performed in such devices. Therefore, the development of matrix computing hardware is currently a significant challenge.

[0004] Thus, a solution disclosed in US Patent No. US 10489479 Bl (HABANA LABS LTD [IL]), published on November 26, 2019, is known from the prior art. This solution discloses a device for matrix calculations based on a systolic array, containing, among other things, a systolic array configured to calculate the sum of products of elements of operand matrices, memory, as well as elements intended for loading and converting input data into elements of the array being processed.

[0005] A drawback of this solution is the presence of data preprocessing units in the architecture that perform matrix transposition before loading into the systolic array, which consequently increases the device's footprint and increases the overall power consumption. Furthermore, the architecture of this solution does not support or support multi-input summation of multiplication results.

[0006] Common disadvantages of the known state-of-the-art solutions include the lack of a hardware device for matrix calculations that provides high performance when performing computational operations on matrix elements and has a small area and low power consumption.

[0007] Accordingly, the goal of this technical solution is to create a matrix computing device with a small footprint and capable of multi-input summation of systolic array elements. Furthermore, this solution should reduce the device's block access frequency and ensure computations are performed without downtime when changing matrices. Furthermore, the claimed solution should include circuits for multiplying large matrices, whose sizes exceed the matrix computing device's cache. DISCLOSURE OF THE INVENTION

[0008] The stated technical solution proposes a new approach to the architecture of a matrix computing device, which ensures a reduction in the device area and an increase in the device performance.

[0009] The technical result achieved by solving this problem is a reduction in the area of ​​the device (due to the use of multi-input adders)

[0010] An additional technical result that emerges from solving the above problem is an increase in the performance of the matrix computing device due to a reduction in computational delays. [OOP] The specified technical results are achieved thanks to a matrix computing device located on a crystal, containing: • a configuration package separation unit connected to an external control device, configured to receive, separate and control the execution of configuration packages; • a memory configured with the possibility of receiving from at least external memory and / or the memory of a vector processor, storing the operand matrices, and transmitting them to the matrix multiplication unit based on the received configuration packets; • a combining unit configured to receive a plurality of scalar vector elements from the memory of a vector processor, combine said elements into a vector, and transfer said vector to the memory; • a matrix multiplication unit, configured to multiply operand matrices based on the received configuration package, including: o a systolic array designed to calculate the sum of the products of the elements of the operand matrices; o a block of delays for elements of operand matrices, designed with the ability to synchronize calculations within the systolic array; o a buffer memory connected to an external device, configured to accumulate the results of calculations from the systolic array and transmit the accumulated results to an external device; o a local control unit configured to control the order of transmission of matrix operand elements to the systolic array and control the transmission of data from the systolic array to an external device. s

[0012] In one particular implementation option, the systolic array consists of multiple computing elements.

[0013] In another particular embodiment, a plurality of computing elements are combined into blocks.

[0014] In another particular embodiment, each computing element is at least a multiplier-adder.

[0015] In another particular embodiment, the size of the systolic array is 128x128 computational elements.

[0016] In another particular embodiment, the computing elements are combined into blocks of 8 computing elements each.

[0017] In another particular implementation variant, the operand matrices are the weight matrices and feature matrices of the neural network.

[0018] In another particular embodiment, the configuration packets are instructions for at least: retrieving data from external memory, retrieving data from vector processor memory, and performing a matrix multiplication operation.

[0019] In another particular embodiment, the configuration packet separation unit receives configuration packets via a common multiplexed bus with identification fields that correspond to the packet type.

[0020] In another particular embodiment, the matrix multiplication unit further comprises an input data controller configured to pause and resume sending data to the systolic array depending on the state of the external receiving device.

[0021] In another particular embodiment, the memory is a multi-port multi-bank memory.

[0022] In another particular embodiment, the computational elements of the systolic array contain active and shadow registers.

[0023] In another particular embodiment, the active and shadow registers are configured to store elements of operand matrices.

[0024] In another particular embodiment, loading of matrix operand elements into shadow registers is performed in the process of calculating the sum of the products of matrix operand elements that are in active registers. BRIEF DESCRIPTION OF DRAWINGS

[0025] The features of the claimed technical solution and a detailed description are given below in the form of attached drawings.

[0026] Fig. 1 illustrates the block diagram of the matrix computing device.

[0027] Fig. 2 illustrates an example of a systolic array consisting of computing elements.

[0028] Fig. 3 illustrates an example of a systolic array consisting of computing elements combined into blocks. IMPLEMENTATION OF THE INVENTION

[0029] Below we will describe the concepts and terms necessary for understanding this technical solution.

[0030] The proposed technical solution offers a new approach to creating a matrix computing device designed to perform matrix multiplication operations and associated data preparation and buffering. The proposed solution provides a matrix computing device with high performance (high computation speed) and a small footprint. Furthermore, the proposed device's architecture reduces the block access frequency within the device and enables computations to be performed without downtime when changing matrices. Furthermore, another advantage achieved with the proposed solution is the ability to multiply large matrices, whose sizes exceed the matrix computing device's cache.

[0031] The term "instructions" as used in this application may generally refer to software instructions or software commands that are written in a given programming language to perform a specific function, such as, for example, executing configuration packages in a matrix computing device, etc. The instructions may be implemented in a variety of ways, including, for example, streaming configuration of machine instructions, object-oriented methods, etc. The instructions that perform the processes described in this solution may be transmitted over wired channels from control devices, for example, a control device for an inference and training accelerator of neural networks.

[0032] Fig. 1 shows a block diagram of a matrix computing device 100. Said device 100 includes a configuration packet separator unit 110, a memory 120, a combining unit 130, a matrix multiplication unit 140, which includes a systolic array 150, a delay unit 160, a buffer memory 170, and a local control unit 180.

[0033] The claimed technical solution is aimed at creating the architecture of a new hardware device for matrix computing 100 that ensures high efficiency (performance) when performing matrix multiplication operations and has a small footprint. The claimed architecture is based on an array of processing elements that perform operations on elements of operand matrices, such as weight matrices and feature matrices of a neural network. The use of the claimed device 100 enables the execution of matrix multiplication operations with high speed (efficiency, performance) by reducing latency during operation execution and eliminating latency when changing matrices in the device 100. Device 100 is primarily used, for example, to accelerate deep learning computations, perform convolution operations, update neural network weights, etc., including as part of neural network training and inference accelerators.

[0034] In this solution, the term neural network inference should be understood as a continuous process of a trained neural network operating to produce a logical conclusion.

[0035] As indicated above, device 100 is primarily part of an accelerator and can be controlled by configuration streams received from a control device (external to device 100) of such an accelerator. However, it should be noted that the application aspects of this technical solution are not limited to this example, and device 100 can be used in conjunction with other elements that provide the ability to interact with data at the tensor (matrix) level. Thus, in one particular embodiment, matrix computing device 100 is connected to external memory for retrieving data from it. Device 100 is also connected to the memory of a vector processor, which is also external to said device 100. The result of calculations received from device 100 can also be transmitted to an external device, for example, to a vector processor for completing neural network calculations, etc., without limitation.

[0036] The elements of the matrix computing device 100 are located on an integrated circuit (IC) chip. In one particular embodiment, the IC chip may be part of another IC chip, such as a neural network accelerator, which also contains other components associated with the device 100 (a control unit, a vector processor, etc.).

[0037] Configuration packet separation unit 110 is connected to an external control device and is configured to receive, separate, and control the execution of configuration packets. Said unit 110 is designed to receive a configuration task for the entire device 100 and divide said task into subtasks for the corresponding elements of the device 100. It is worth noting that this separation allows for the asynchronous execution of operations, i.e. each configuration packet will be processed independently by the element of device 100 corresponding to this packet.

[0038] To control device 100, an external control device, such as an external control device for the inference and neural network training accelerator, sends configuration packets to block 110 to perform the required data operations. In this solution, configuration packets are understood to mean a set of instructions for a specific element of device 100. Such instruction sets may include at least: instructions for retrieving data from external memory, instructions for retrieving data from vector processor memory, and instructions for performing a matrix multiplication operation.

[0039] Thus, in one particular embodiment, the configuration packets may contain the following types of packets: packets configuring the loading of data from external memory and from the vector processor memory, and packets configuring the matrix multiplication process. Thus, in another particular embodiment, block 110 receives a configuration task from an external control device, which contains, for example, two configuration packets that are aimed at loading data from external memory and the vector processor memory, and a third configuration packet for executing the matrix multiplication operation. These packets are distributed to the corresponding elements of device 100, such as memory 120 for the two configuration packets that are aimed at loading data from external memory and the vector processor memory, and block 140 for the configuration packet for executing the matrix multiplication operation.It will be apparent to those skilled in the art that the number of configuration packets will depend on the data sources (external memory, vector processor memory, etc.). All configuration packets are fed to device 100 via a common multiplexed bus with an identification field that allows for distinguishing the types of packets being transmitted.

[0040] Configuration packages are necessary for streamlining data loading processes and, among other things, enable interaction with data exceeding the memory 120 of device 100. For example, data loaded into memory 120 from external memory or vector processor memory in a single command may not correspond to a single matrix multiplication task. For example, the weights for the entire neural network can be loaded in a single command, and then the weight data can be read gradually, as needed for the matrix multiplication operation. Conversely, multiplication operations for very large matrices, which may not fit in memory 120, can be broken down into multiplication operations for smaller matrices. Accordingly, configuration packages are necessary for specifying the order of data reading and the order of computational operations performed on the data.Thus, in another specific implementation example, the operation of reading both weights and features from memory 120 will be performed based on a single configuration packet, which contains the necessary set of commands to control the entire matrix multiplication process. This packet may contain data such as the frequency of weight changes in systolic array 150 (the time interval after which new weights should be loaded into the computational elements of systolic array 150), the order of reading matrix operand elements from memory 120, etc. In other words, configuration packets contain data that enables the ability to configure parameters during calculations and ensures optimal use of memory 120 in various scenarios.

[0041] Memory 120 is a multi-port multi-bank memory.

[0042] Thus, memory 120 is configured to store operand matrices received from external memory and / or vector processor memory and output the operand matrix elements in a specific order. NVM memory can be used as external memory. Multiport shared memory can be used as vector processor memory. The operand matrices are weight matrices and feature matrices of the neural network. Accordingly, the elements of the operand matrices are values ​​located at a specific location in the matrix.

[0043] After loading the operand matrices (the neural network's weight and feature matrices), the data is stored in memory 120 until a command is received to perform a matrix multiplication operation. Upon receiving such a command, the weights and features are read from memory 120 in a specific order and fed to systolic array 150 for direct computation.

[0044] The feature matrix size can be N x 128 (N is any natural number, but it must fit within memory 120). The weight matrix is ​​always 128 x 128. One of the features of the proposed solution is the elimination of the need for matrix transposition before matrix multiplication, due to the fact that the weight matrix is ​​prepared in transposed form. This means that the multiplier 140 receives matrix rows, rather than rows and columns, which ensures implicit inversion of the second matrix. This feature also further reduces the footprint of device 100, since the matrices are loaded using a single set of connecting elements (wires), and eliminates the need for a transposer device within device 100.

[0045] A combining unit 130 configured to receive a plurality of scalar vector elements from the memory of a vector processor, combine said elements into a vector, and transfer said vector to the memory.

[0046] Thus, the specified block 130 is designed to synchronize scalar vector elements received from the vector processor memory. The specified block may be FIFO-type memory. Thus, the vector processor consists of a plurality of physically located scalar elements designed to perform computational operations on vector elements (for example, a single operation on 128 vector elements in parallel, performed by 128 scalar elements). Accordingly, in this case, the vector elements arriving at device 100 from the scalar elements will have different latencies. The specified block 130 synchronizes the incoming vector elements from the scalar elements of the vector processor, which, in turn, allows the entire vector to be loaded into memory 120.

[0047] The matrix multiplication block 140 consists of at least the following elements: a systolic array 150, a delay block 160, a buffer memory 170, a local control block 180.

[0048] Let us consider the specified elements in more detail.

[0049] Systolic array 150 is designed to directly calculate the sum of products of matrix operand elements. A more detailed example of a systolic array is shown in Fig. 2. Array 150 consists of a plurality of processing elements 210 (PE). In turn, each processing element 210 within systolic array 150 may represent, for example, a multiplier-adder and additional auxiliary logic. The size of the systolic array is 128x128 unit (scalar) multipliers-adders. Thus, device 100 is configured to alternately load weights and features of the neural network, i.e., before the matrix multiplication operation, weights (which do not change for a long time) can be loaded into the systolic array, and after that, the elements of the neural network features will be fed, for example, in the form of a vector.It is also worth noting that, due to the fact that in the systolic array, when calculating the sum of products, switching occurs from row to row of computing elements 210, then when one vector of the feature matrix is ​​supplied, the specified vector is multiplied by the entire matrix, which makes the process of matrix multiplication more efficient, compared to other specialized devices, such as a vector processor.

[0050] Thus, in one particular embodiment, computing elements 210 contain active and shadow registers. These registers are used to store elements of the operand matrices. Let's examine this feature in more detail. The presence of shadow registers allows for loading new weights into these shadow registers directly during the process of multiplying the supplied feature matrix elements by the previously loaded weight matrix elements located in the active registers. Accordingly, after loading the new weights and after the flow of feature matrix elements has ended, seamless switching to the new weights with minimal downtime is ensured.

[0051] In another particular embodiment, the computing elements 210 of the systolic array 150 may be combined into blocks. This example is shown in more detail in Fig. 3. Thus, the computing elements 210 may be combined into block computing elements 310, for example, with dimensions of 4 (in height) by 2 (in width). Accordingly, in such an embodiment, the size of the systolic array is 16x128 block elements of size 4x2. Moreover, in yet another particular embodiment, the block computing elements may be combined, for example, with dimensions of 8x1 elements, etc.

[0052] Combining the 210 elementary computational elements of the systolic array 150 into blocks 310 further optimizes power consumption and reduces computational latency. Vertical aggregation enables the use of multi-input adders, which conserve chip area, reduce power consumption, and reduce computational latency. Horizontal aggregation further reduces computational latency.

[0053] Also, to reduce the power consumption of elementary mathematical addition and multiplication blocks, some implementations allow the use of non-standard floating-point number representation formats, such as formats other than IEEE 754. Some implementations can save space and power by simplifying the pre- and post-normalization circuits when representing numbers in a non-normalized format (Exp8 + Int9), where the hidden bit of a floating-point number is pre-calculated, after which the mantissa, sign, and this hidden bit are converted to a signed integer.

[0054] Let's look at the process of loading and changing weights in the systolic array 150.

[0055] The number of cycles required to load weights (weight matrix elements) is equal to the systolic array size: 150–128 cycles. As noted earlier, loading weights for a subsequent matrix multiplication operation can occur during the execution of the preceding multiplication operation. If the execution of the preceding matrix multiplication task takes fewer cycles than required for preloading the weights, an additional pause occurs until all weights are loaded into the shadow registers. The process of reading weights and features from the neural network is controlled by the configuration package that manages the entire matrix multiplication.

[0056] Let's consider the process of pausing calculations in systolic array 150.

[0057] Since the systolic array 150 is one large mathematical block with a fixed delay (latency), without a global enabling signal (clock enable), then after stopping the data supply to the input of array 150, data will continue to exit array 150 for some time. Accordingly, for the accumulation of results, buffer memory 170 is provided in block 140.

[0058] Thus, said memory 170 is a FIFO-type circular buffer with scalar outputs. Said memory 170 is necessary for accumulating data received from the output of array 150. For example, if a data feed to array 150 is interrupted due to the unavailability of the receiver of array 150's calculation results, such as the unavailability of an external device, then the data received from array 150's output will be stored in said memory 170 until this flow stops. Furthermore, said memory 170 is also necessary to mitigate delays caused by the direct calculation of the sum of the products of matrix operand elements by systolic array 150.

[0059] In another specific embodiment, this suspension can be accomplished using a credit counter. Receipt of valid data by systolic array 150 is accompanied by a decrement in the credit counter. If the counter decrements to zero, the circuit emits a Stop signal, halting the sending of data to the systolic array. When the receiving device is again ready to receive output from the systolic array, each valid vector received is accompanied by an increment in the credit counter. If the increment and decrement signals arrive at the counter simultaneously, its value remains unchanged.

[0060] The delay block 160 is a hardware block designed to ensure the correct operation of the systolic array by delaying the elements arriving at the systolic array 150 to form data in the format required by the systolic array 150. That is, the said block 160 performs the ordering of the input elements arriving at the systolic array in such a way that subsequent elements arrive at the array with a certain delay necessary for the implementation of correct calculations within the systolic array 150.

[0061] Thus, let us consider the process of calculations within the systolic array 150. Since the elements of the operand matrices will arrive at the computational elements 210 (Fig. 2) sequentially, the calculations will not occur simultaneously. Accordingly, the output of the first left computational element 210 (Fig. 2, first row, first column) will occur, for example, in two clock cycles. That is, in the specified element 210, the operation A*B will occur in two clock cycles. Further, the result of this operation will go to the element 210 located in the second row, first column in Fig. 2. However, the specified result will arrive only after two clock cycles. To ensure the correct operation of the systolic array 150, and, consequently, the specified calculations, block 160 is located in block 140, which delays the incoming matrix elements to the subsequent computational elements 210. The specified block can act as a delay ladder with a certain step, for example, two clock cycles. That is,For each subsequent element 210 in the column, the delay will be greater by a certain fixed parameter, 1 cycle, 2 cycles, 3 cycles, etc.

[0062] The local control unit 180 is a hardware and software unit designed to receive data streams and control streams from the control device 110 and to convert said streams into sets of low-level commands for controlling the systolic array 150.

[0063] Since the systolic array 150 itself is a set of the simplest computing elements 210, which are not physically intended for receiving and processing commands, then for implementing control of the said elements 210, the device 100 contains a block 180. Thus, the said block 180 is configured to receive commands from other elements of the device 100, for example, commands about the start of loading matrix elements, about the end of loading, etc., and converting the said commands into the simplest signals, such as stopping calculations, switching shadow registers, etc. Accordingly, based on the received data flows and control flows, the block 180 controls the order of transferring the matrix elements-operands to the systolic array, i.e., it controls the block 160. In addition, in another particular embodiment, the local block 180 controls the transfer of data from the systolic array to an external device, i.e.the data received by memory 170 is sent to an external device via block 180.

[0064] Thus, in the submitted application materials, various embodiments of the matrix computing device were described.

[0065] The specific selection of elements of the device 100 for implementing various software, hardware and / or architectural solutions may vary while maintaining the required functionality provided.PC17RU2025 / 000009

[0066] The submitted application materials disclose preferred embodiments of the technical solution and should not be construed as limiting other, specific embodiments within the scope of the requested legal protection that are obvious to specialists in the relevant technical field. Therefore, the scope of this technical solution is limited only by the scope of the attached claims.

Claims

FORMULA 1. An on-chip matrix computing device comprising: • a configuration package separation unit connected to an external control device, configured to receive, separate and control the execution of configuration packages; • a memory configured with the possibility of receiving from at least external memory and / or the memory of a vector processor, storing the operand matrices, and transmitting them to the matrix multiplication unit based on the received configuration packets; • a combining unit configured to receive a plurality of scalar vector elements from the memory of a vector processor, combine said elements into a vector, and transfer said vector to the memory; • a matrix multiplication unit, configured to multiply operand matrices based on the received configuration package, including: o a systolic array designed to calculate the sum of the products of the elements of the operand matrices; o a block of delays for elements of operand matrices, designed with the ability to synchronize calculations within the systolic array; o a buffer memory connected to an external device, configured to accumulate the results of calculations from the systolic array and transmit the accumulated results to an external device; o a local control unit configured to control the order of transmission of matrix operand elements to the systolic array and control the transmission of data from the systolic array to an external device.

2. The device according to paragraph 1, characterized in that the systolic array consists of a plurality of computing elements.

3. The device according to paragraph 2, characterized in that a plurality of computing elements are combined into blocks.

4. The device according to any of paragraphs 2-3, characterized in that each computing element is at least a multiplier-adder.

5. The device according to paragraph 2, characterized in that the size of the systolic array is equal to 128x128 computing elements.

6. The device according to paragraph 3, characterized in that the computing elements are combined into blocks consisting of computing elements.

7. The device according to paragraph 1, characterized in that the operand matrices are matrices of weights and matrices of features of a neural network.

8. The device according to claim 1, characterized in that the configuration packets represent instructions for at least: obtaining data from external memory, obtaining data from the memory of a vector processor, and the operation of calculating matrix multiplication.

9. The device according to paragraph 1, characterized in that the configuration packet separation unit receives configuration packets via a common multiplexed bus with identification fields that correspond to the packet type.

10. The device according to claim 1, characterized in that the matrix multiplication unit additionally contains an input data controller configured to pause and resume sending data to the systolic array depending on the state of the external receiving device.

11. The device according to claim 1, characterized in that the memory is a multi-port multi-bank memory.

12. The device according to paragraph 2, characterized in that the computing elements of the systolic array contain active and shadow registers.

13. The device according to item 12, characterized in that the active and shadow registers are designed with the possibility of storing elements of operand matrices.

14. The device according to paragraphs 12-13, characterized in that the loading of matrix operand elements into shadow registers is performed in the process of calculating the sum of the products of matrix operand elements that are in active registers.

Images