334 results about "Systolic array" patented technology

The invention discloses an accelerator for accelerating the reasoning process of an unstructured sparse large language model and an operation method of the accelerator. The accelerator comprises a processor module, a direct memory access module, a memory controller module, a DDR memory module, an interconnection bus module, a calculation core module and a router module. The interconnection bus module is connected with the direct memory access module, the memory controller module and the computing core module; the processor module is connected with the direct memory access module; the DDR memory module is connected with the memory controller module; the router module is connected with the computing core module and the adjacent router module; and the calculation core module obtains to-be-calculated data from the off-chip DDR memory module through the interconnection bus module. According to the method, the limit that data reuse is carried out only along a single row or a single column on a traditional systolic array is broken through, cross-row and cross-column data reuse is achieved, the pressure of hardware equipment is relieved, the positions of non-zero elements are coded through bit masks, and the calculation efficiency is improved.

Systems and methods are provided to enable parallelized multiply-accumulate operations in a systolic array. Each column of the systolic array can include multiple busses enabling independent transmission of input partial sums along the respective bus. Each processing element of a given columnar bus can receive an input partial sum from a prior element of the given columnar bus, and perform arithmetic operations on the input partial sum. Each processing element can generate an output partial sum based on the arithmetic operations, provide the output partial sum to a next processing element of the given columnar bus, without the output partial sum being processed by a processing element of the column located between the two processing elements that uses a different columnar bus. Use of columnar busses can enable parallelization to increase speed or enable increased latency at individual processing elements.

Techniques for performing large tensor tiling (LTT) in hardware are enabled. LTT divides a large tensor (e.g., of unsupported size) into overlapping tiles (e.g., having supported tensor size(s)). A tensor may be processed processing the tiles. The output of each processed tile is stored, for example, in a systolic array considering the tile's placement in the large tensor. The output of all processed tiles is identical to the output of processing the large tensor. Tiles may be processed by reusing data overlapping boundaries shared with other tiles. In some examples, overlapping data may be reused (e.g., written once) or partly reused (e.g., written twice). Tiling large tensors with boundary duplication supports dynamic adaptation to a wide variety of tensor sizes, avoids re-reading duplicated data, and avoids reorganizing hardware for large tiles, which reduces power consumption and area, reduces complexity, and increases processing efficiency.

The invention belongs to the field of machine learning, and discloses an FPGA (Field Programmable Gate Array) superposition processor acceleration system and method based on state space duality, which comprises a sparse predefined data acquirer, a reconfigurable systolic array, a partial sum cache, an element-by-element operation cache, a function calculation module, an on-chip memory management module and the like. Zero elements are eliminated through the sparse predefined data acquirer, redundant data are reduced, and redundant calculation is remarkably reduced; the reconfigurable systolic array flexibly supports multiple calculation modes, and the utilization rate of hardware resources is increased; the part and the cache realize cross-cycle accumulation and element-by-element operation cache integration results, the function calculation module completes nonlinear operation, and the on-chip memory management module optimizes result cache, so that on-chip operation of SSD calculation is ensured, off-chip memory access is remarkably reduced, and reasoning efficiency and energy efficiency are improved; by adopting the system, the memory occupation is effectively reduced, the element-by-element calculation efficiency is improved, and the sparse calculation redundancy is reduced, so that the reasoning process of the Mamba2 model is remarkably accelerated.

An apparatus for accelerating neural networks, includes: a memory for storing graph input data including vertices and edges; an aggregation engine that processes the accumulation of features and generates feature vectors by taking the graph input data and performing an aggregation operation on the graph input data; an on-chip cache for caching the feature vectors; and a combination engine that generates a systolic array for matrix multiplications based on the feature vectors taken from the on-chip cache and weights taken from the memory.

The invention discloses a reconfigurable systolic array tensor IP core based on an FPGA. The architecture comprises a core controller, a shift distributor, a data processing array and a four-stage data cache. The tensor core can adapt to general tensor multiplication and addition operations in Prefill and Decode stages of the generative neural network at the same time through reconfigurable design. The core controller receives the tensor multiplication instruction and the configuration information and decouples the tensor multiplication instruction and the configuration information according to a data stream to generate a plurality of block tensor multiplication instructions. The shift distributor decodes a block tensor multiplication instruction, decoded mode information controls data communication between a first-level cache and a second-level cache (Distrided RAM and Bram), a control signal of a data processing array is generated, and the control signal enters the array together with data to flow. And the data processing array is connected with different secondary BRAM caches according to the received configuration information and is used for processing data in a bubble-free pulsating flow manner. The four-stage data cache realizes the multi-stage pinpong operation of the loading of the operation data of the tensor operation core, and realizes the multi-stage pipeline design of the data from the off-chip HBM to the on-chip URAM, from the URAM to the BRAM and from the BRAM to the operation unit. Aiming at a single-user edge end generation type network reasoning task, a core controller generates a data processing unit mode configuration signal in a fine-grained (tensor element level) manner, and the data processing unit mode configuration signal is transmitted by a shift distributor and flows in parallel with operation data, so that the mode configuration efficiency and the hardware resource utilization rate are greatly improved, the problem of high fan-out caused by broadcast control is reduced, and the user experience is improved. A control flow is designed by means of the structural characteristics of the systolic array, and a complex control algorithm is realized with low resource occupation.

The invention provides a flexible convolution operation accelerator based on a systolic array. The flexible convolution operation accelerator is used for solving the technical problem that the hardware scale of an existing convolution operation accelerator is difficult to flexibly adjust according to different hardware platforms. The system comprises an input cache module, a pulsation matrix, an accumulation logic unit and an output cache module which are connected in sequence, the input cache module comprises a weight cache module and an image cache module, and the weight cache module and the image cache module are both connected with the pulsation matrix. The weight cache module, the image cache module, the pulsation matrix, the accumulation logic unit and the output cache module are all connected with the controller, and the controller and the output cache module are both connected with an external memory through a BUS. The systolic array and the flexible cache module are utilized, the functions of data storage and data format conversion can be achieved at the same time, hardware logic overhead is reduced, high memory access efficiency is achieved, and configuration of the accelerator scale and various convolution parameters is supported.

Systems and methods are provided to perform multiply-accumulate operations of reduced precision numbers in a systolic array. Each row of the systolic array can receive reduced inputs from a respective reducer. The reducer can receive a particular input and generate multiple reduced inputs from the input. The reduced inputs can include reduced input data elements and / or a reduced weights. The systolic array may lack support for inputs with a first bit-length and the reducers may reduce the bit-length of a given input from the first bit-length to a second shorter bit-length and provide multiple reduced inputs with second shorter bit-length to the array. The systolic array may perform multiply-accumulate operations on each unique combination of the multiple reduced input data elements and the reduced weights to generate multiple partial outputs. The systolic array may sum the partial outputs to generate the output.

Disclosed is a method and apparatus with deep learning operations. A deep learning apparatus includes a processor, configured to support a plurality of different operation modes, including a systolic array having a plurality of multiplier accumulator (MAC) units, and a control circuit configured to respectively control, for each the plurality of different operation modes, select operations of the plurality of MAC units and data movements among the plurality of MAC units.

A disclosed approximate tensor processing unit (APTPU) includes two main components: (1) approximate processing elements (APEs) consisting of a low-precision multiplier and an approximate adder, and (2) pre-approximate units (PAUs) which are shared among the APEs in the APTPU's systolic array, functioning as the steering logic to pre-process the operands and feed them to the APEs. Performance of the disclosed APTPU across various configurations and various workloads shows that the disclosed APTPU's systolic array achieves delay, area, and power reductions, while realizing comparable accuracy to previous designs.

A systolic array can implement an architecture tailored to perform matrix multiplications on sparse matrices. Each processing element in the systolic array may include a register configured to store a value, and a multiplexor configured to select an input element from multiple input data buses based on metadata associated with the value. Each processing element may also include a multiplier configured to multiply the selected input element with the value to generate a multiplication result, and an adder configured to add the multiplication result to a partial sum input to generate a partial sum output.

The invention discloses a data processing method and device for sparse matrix multiplication, equipment and a medium, and the method comprises the steps: carrying out the preprocessing of a sparse matrix, and obtaining a sparse matrix block and a sparse index corresponding to the sparse matrix block; according to the number of non-zero elements in the sparse index, corresponding index coordinates are distributed to a calculation unit PE in the one-dimensional systolic array; performing multiply-accumulate calculation according to a first calculation path in the PE, and outputting a calculation result; performing multiplication calculation according to a second calculation path in the PE, and outputting a multiplication result corresponding to the PE; performing packet summation through a packet summation network to obtain at least one packet summation result; and reordering the at least one group summation result to obtain an output matrix. Therefore, two types of sparse matrix multiplication of SDDMM and SpMM are efficiently processed by fully utilizing the characteristics of the matrix inner product and the matrix row product, and the intermediate calculation result is temporarily stored and reused, so that the data movement and storage overhead is reduced, and the data processing efficiency is improved.

Aspects of the disclosure are directed to hardware acceleration of structured sparse workloads with block quantization. A hardware accelerator can receive compressed input matrices, for example as part of a workload for training or processing a machine learning model. The hardware accelerator can multiply the compressed input matrix with a gains matrix loaded in one or more matrix multiply units (MXUs) of the hardware accelerator. The input matrices can be further provided in a block data type format, in which blocks of mantissas are represented with a single shared scaling factor. An MXU can multiply the block data, shift or cast the block data according to a shared scaling factor to generate an output product. To that end, block data type matrices exhibiting structured sparsity patterns can be accelerated without affecting the overall accuracy or quality of the output to the workload being processed.

A hardware accelerator including tiles arranged in a systolic array. At each of the tiles, the systolic array receives a first input block that includes first input matrix elements of a first input matrix. In each of a plurality of multiplication iterations, at each of the tiles, the systolic array receives a respective second input block. The systolic array computes tile products of the first input matrix elements and second input matrix elements included in the second input blocks. The systolic array adds the tile products to column-wise partial sums and transmits the column-wise partial sums to subsequent tiles along accumulator rings included in array columns of the systolic array. In a subset of the multiplication iterations, the systolic array outputs product block rows of a product matrix. The product block rows each include product matrix blocks computed as rows of the column-wise partial sums.

A processing apparatus includes a processing resource including a general-purpose parallel processing engine and a matrix accelerator. The matrix accelerator includes first circuitry to receive a command to perform operations associated with an instruction, second circuitry to configure the matrix accelerator according to a physical depth of a systolic array within the matrix accelerator and a logical depth associated with the instruction, third circuitry to read operands for the instruction from a register file associated with the systolic array, fourth circuitry to perform operations for the instruction via one or more passes through one or more physical pipeline stages of the systolic array based on a configuration performed by the second circuitry, and fifth circuitry to write output of the operations to the register file associated with the systolic array.

The mixed-base FFT hardware accelerator comprises an input logic unit, an operation logic unit, a data conversion unit and an output logic unit which are arranged on an FPGA and connected in sequence, an operation unit is arranged in the operation logic unit, the operation unit is connected with the data conversion unit, the systolic array is arranged in the operation unit, and the systolic array is connected with the data conversion unit. The pulse sequence adopts mixed base to accelerate calculation of FFT (Fast Fourier Transform); the data conversion unit comprises a data truncation module and a data splicing unit which are connected in sequence, the operation unit is connected with the data truncation module, and the data splicing unit is connected with the output logic unit; the input logic unit, the operation unit, the data truncation module, the data splicing unit and the output logic unit are all connected with the controller, and the input logic unit, the data splicing unit, the data splicing unit and the output logic unit are all connected with the DDR memory through a data bus. The method can adapt to requirements of different performances, and fast calculates FFT of different points.

To improve utilization of a systolic array, each row of the array is provided with two or more general purpose row input buses. Each of the general purpose row input data buses can be operable to transfer either feature map (FMAP) input elements or weight values into the processing elements of the corresponding row of the array. The flexibility of the general purpose row input buses allows weights to be loaded in parallel into the array to speed up weight loading times to reduce the overall latency.

The invention belongs to the field of machine learning, discloses an expert parallelism processing method and system for a large language model based on MoE, and realizes efficient parallel processing of the MoE model by dynamically distributing expert quantization bit width and sparse mode, predicting and prefetching to-be-activated expert parameters, grouping tokens to generate task queues and dynamically configuring hardware accelerators. Firstly, an importance score is calculated based on expert historical activation frequency, weight distribution and a model structure, so that quantization precision and a sparse proportion are adaptively allocated, and resource waste and precision loss of a unified strategy are avoided; secondly, predicting an expert to be activated by using a current layer hidden state and a historical activation sequence, loading parameters to a special cache in advance, reducing high-bandwidth memory access, and relieving bandwidth peak scrambling; moreover, tokens are grouped through a token-expert mapping table, a task queue is created, and dynamic configuration of a systolic array is combined, so that the problem of expert heterogeneity after compression is solved, and efficient and parallel hybrid precision matrix operation is ensured.

Systems or methods of the present disclosure may provide an integrated circuit system that includes a programmable logic device that includes a clock, one or more local controllers, programmable logic units implementing a systolic array to compute a matrix multiplication, and embedded memory blocks. The embedded memory blocks include a single port random access memory (SPRAM). The one or more local controllers are configured to, on a first set of alternating clock cycles of the clock, load matrix sub-elements from two rows of a matrix into corresponding matrix element of the SPRAM. The one or more local controllers are configured to, on a second set of alternating clock cycles of the clock, read out the matrix elements from the SPRAM to the systolic array to compute the matrix multiplication.

The embodiment of the invention provides a processor architecture supporting in-memory matrix operation based on a static memory and a processor. Wherein the processor is provided with the provided processor architecture. A tensor core module in the processor architecture comprises a plurality of matrix operation units, a plurality of in-memory computing macro units are arranged in the matrix operation units, the plurality of in-memory computing macro units are connected by adopting a pulsation data path to form a two-dimensional pulsation array, and each in-memory computing macro unit comprises a plurality of parallel memory banks. And the memory bank comprises a static memory array supporting in-memory matrix operation. Therefore, according to the processor architecture provided by the scheme, the in-memory calculation macro unit based on the static memory is introduced to serve as the matrix operation unit to support in-memory matrix operation, and by adopting the matrix operation unit, the overall performance of the processor can be improved, and the high energy efficiency requirements of generative model reasoning and training can be better met.

An OpenCL-defined scalable runtime-flexible programmable accelerator architecture for accelerating convolutional neural network (CNN) inference in cloud / edge computing is provided, referred to herein as Systolic-CNN. Existing OpenCL-defined programmable accelerators (e.g., field-programmable gate array (FPGA)-based accelerators) for CNN inference are insufficient due to limited flexibility for supporting multiple CNN models at runtime and poor scalability resulting in underutilized accelerator resources and limited computational parallelism. Systolic-CNN adopts a highly pipelined and paralleled one-dimensional (1-D) systolic array architecture, which efficiently explores both spatial and temporal parallelism for accelerating CNN inference on programmable accelerators (e.g., FPGAs). Systolic-CNN is highly scalable and parameterized, and can be easily adapted by users to efficiently utilize the coarse-grained computation resources for a given programmable accelerator. In addition, Systolic-CNN is runtime-flexible and can be time-shared to accelerate a variety of CNN models at runtime without the need to recompile the programmable accelerator kernel hardware or reprogram the programmable accelerator.

The invention discloses a data processing method and system for a multi-head potential attention model based on TTD compression and a terminal, and the method comprises the steps: constructing a large language model, and processing a plurality of linear layers in the large language model in a TTD compression and decomposition mode, thereby reducing the weight number in the model, and improving the data processing efficiency. And on a hardware level, targeted optimization is carried out on a data stream subjected to linear layer reasoning after TTD compression, so that a final model reasoning result is output. According to the method, a multi-head potential attention structure in a large language model is converted, so that the requirement for KV cache during model reasoning is reduced, the weight number is reduced, the long text output capacity of the model on edge equipment is improved, matrix calculation in the reasoning process is carried out subsequently by adopting a calculation structure of a group vector systolic array, and the reasoning efficiency is improved. And limited hardware resources are efficiently utilized.

A compiler tool and a method of operating a compiler tool for selecting between executing a matrix multiplication operation in a weight stationary mode or in an output stationary mode on a systolic array with reconfigurable processing elements are presented. The compiler tool receives configuration parameters, energy parameters, and performance parameters as well as dimensions of the matrices to be multiplied and estimates energy consumptions and performance numbers for executing the matrix multiplication operation in the weight stationary and the output stationary modes. The compiler tool selects between operating the matrix multiplication operation in the weight stationary and the output stationary mode based on the estimated energy consumption and the estimated performance numbers.

The invention discloses an accelerator supporting an attention mechanism, which is used for realizing matrix multiplication operation, scaling operation or nonlinear activation processing related to the attention mechanism, and is characterized in that a weight cache state machine module is used for controlling weight data to flow from a system bus to a weight cache unit; the thread cache state machine module is used for controlling thread data to flow from the system bus to the thread cache unit; the attention tensor processing unit state machine module is used for controlling loading and flowing of thread data and weight data from the two-dimensional systolic array in different directions, the thread data is controlled to be transmitted to the next processing unit in each period, and the weight data is spread to the processing units in the corresponding directions in a broadcast mode; the calculation result of the two-dimensional systolic array is transmitted to different branches according to whether nonlinear activation processing needs to be executed or not. According to the method, data filling is carried out on the matrix so that the matrix can be matched with the systolic array data stream, and then the attention mechanism with the variable sequence length is accelerated through hardware.

The invention provides a high-efficiency matrix engine (RVME) architecture based on RISC-V matrix extension and a calculation method, and the architecture comprises an instruction buffering and decoding module, a matrix loading / storage module, a matrix register file, a parallel outer product array and an element-by-element operation module; the matrix register file comprises a Tile register and an Acculator register; the storage modules are respectively used for storing an input matrix and an accumulation result and supporting efficient data access and parallel computing; the matrix loading / storage module significantly improves the data loading efficiency through cache line alignment and matrix transposition optimization; the instruction buffering and decoding module cooperates with a main processor through a reordering buffer area and an instruction buffer area to ensure efficient scheduling and execution of instructions. The parallel outer product array is adopted to replace a traditional systolic array, the idle period in the calculation process is eliminated through multicast data flow scheduling and a ping-pong buffer read-write mechanism, and matrix multiplication and addition operation with high calculation utilization rate and low delay is achieved.

A system and method for matrix multiplication using a systolic array configurable between a plurality of operating modes. A pulsation processor may receive a data type indicator for matrix multiplication. For a first data type, the systolic processor may load right-hand side data from a right-hand matrix register into data processing units of the systolic array between rows 0 and M-1, and pass respective rows of left-hand side data through corresponding rows of the systolic array between rows 0 and M-1. For the second data type, the pulsation processor may segment each element of the left-hand-side data and the right-hand-side data into a respective first element half and a respective second element half, and move each element half through a corresponding row of the pulsation array between rows 0 and 2M-1.

The invention relates to a data processing method for matrix calculation, a pulsation matrix, a device, equipment and a medium. The data processing method comprises the steps that data in a first input matrix and data in a second input matrix are read, the first input matrix is provided with M rows and K columns, and the second input matrix is provided with K rows and N columns; inputting the read data of the first input matrix into the systolic matrix, and enabling the data of the first input matrix to be transversely and circularly propagated in the systolic array; inputting the read data of the second input matrix into the systolic array, and enabling the data of the second input matrix to be circularly propagated in the systolic array along the diagonal direction; the processing unit is used for executing multiply-accumulate operation on data flowing through the first input matrix and data flowing through the second input matrix, and a calculated intermediate result is resided in the processing unit; and in response to determining that the multiply-accumulate operation is completed, unidirectionally propagating the final calculation result residing in each processing unit along the diagonal direction so as to move the final calculation result out of the systolic array.

The invention discloses a self-attention mechanism calculation method, a calculation array, a calculation device, a calculation system and a computer readable storage medium. The self-attention mechanism calculation method is applied to a self-attention mechanism calculation array, the self-attention mechanism calculation array comprises a plurality of product accumulators, the plurality of product accumulators form a systolic array, and the self-attention mechanism calculation method comprises the following steps: performing calculation through the plurality of product accumulators according to a feature tensor and a weight tensor to obtain an intermediate tensor; and a result tensor is calculated according to the intermediate tensor through a plurality of product accumulators. Therefore, frequent memory access of the intermediate tensor between the self-attention mechanism calculation array and the external memory is avoided, the calculation efficiency is improved, and the problems of intermediate tensor multiplexing and storage optimization are solved.

A processing apparatus is described herein that includes a general-purpose parallel processing engine comprising a matrix accelerator including one or more systolic arrays, at least one of the one or more systolic arrays comprising multiple pipeline stages, each pipeline stage of the multiple pipeline stages including multiple processing elements, the multiple processing elements configured to perform processing operations on input matrix elements based on output sparsity metadata. The output sparsity metadata indicates to the multiple processing elements to bypass multiplication for a first row of elements of a second matrix and multiply a second row of elements of the second matrix with a column of matrix elements of a first matrix.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a systolic array scheduling processing method, device, equipment and medium, and the method comprises the steps: obtaining a data processing model, and compiling the data processing model into an initial processing task based on hardware architecture features of a systolic array, acquiring to-be-processed data and analyzing data characteristics of the to-be-processed data, monitoring a real-time operation state of the systolic array processing device, inputting the real-time operation state, the data characteristics and an initial processing task into a scheduling decision module to generate a scheduling strategy, executing the scheduling strategy to complete data processing, and collecting performance data; and updating optimization parameters of the compiling module and strategy generation parameters of the scheduling decision module based on the performance data. According to the method, the initial processing task is generated through compiling, the scheduling strategy is dynamically generated in combination with the data features and the running state, parameter updating is achieved through performance data feedback, self-adaptive closed-loop optimization of compiling and scheduling is formed, and the calculation efficiency and the energy efficiency ratio are improved.