272 results about "Systolic array" patented technology

The invention discloses an accelerator for accelerating the reasoning process of an unstructured sparse large language model and an operation method of the accelerator. The accelerator comprises a processor module, a direct memory access module, a memory controller module, a DDR memory module, an interconnection bus module, a calculation core module and a router module. The interconnection bus module is connected with the direct memory access module, the memory controller module and the computing core module; the processor module is connected with the direct memory access module; the DDR memory module is connected with the memory controller module; the router module is connected with the computing core module and the adjacent router module; and the calculation core module obtains to-be-calculated data from the off-chip DDR memory module through the interconnection bus module. According to the method, the limit that data reuse is carried out only along a single row or a single column on a traditional systolic array is broken through, cross-row and cross-column data reuse is achieved, the pressure of hardware equipment is relieved, the positions of non-zero elements are coded through bit masks, and the calculation efficiency is improved.

The invention belongs to the field of machine learning, and discloses an FPGA (Field Programmable Gate Array) superposition processor acceleration system and method based on state space duality, which comprises a sparse predefined data acquirer, a reconfigurable systolic array, a partial sum cache, an element-by-element operation cache, a function calculation module, an on-chip memory management module and the like. Zero elements are eliminated through the sparse predefined data acquirer, redundant data are reduced, and redundant calculation is remarkably reduced; the reconfigurable systolic array flexibly supports multiple calculation modes, and the utilization rate of hardware resources is increased; the part and the cache realize cross-cycle accumulation and element-by-element operation cache integration results, the function calculation module completes nonlinear operation, and the on-chip memory management module optimizes result cache, so that on-chip operation of SSD calculation is ensured, off-chip memory access is remarkably reduced, and reasoning efficiency and energy efficiency are improved; by adopting the system, the memory occupation is effectively reduced, the element-by-element calculation efficiency is improved, and the sparse calculation redundancy is reduced, so that the reasoning process of the Mamba2 model is remarkably accelerated.

The invention discloses a reconfigurable systolic array tensor IP core based on an FPGA. The architecture comprises a core controller, a shift distributor, a data processing array and a four-stage data cache. The tensor core can adapt to general tensor multiplication and addition operations in Prefill and Decode stages of the generative neural network at the same time through reconfigurable design. The core controller receives the tensor multiplication instruction and the configuration information and decouples the tensor multiplication instruction and the configuration information according to a data stream to generate a plurality of block tensor multiplication instructions. The shift distributor decodes a block tensor multiplication instruction, decoded mode information controls data communication between a first-level cache and a second-level cache (Distrided RAM and Bram), a control signal of a data processing array is generated, and the control signal enters the array together with data to flow. And the data processing array is connected with different secondary BRAM caches according to the received configuration information and is used for processing data in a bubble-free pulsating flow manner. The four-stage data cache realizes the multi-stage pinpong operation of the loading of the operation data of the tensor operation core, and realizes the multi-stage pipeline design of the data from the off-chip HBM to the on-chip URAM, from the URAM to the BRAM and from the BRAM to the operation unit. Aiming at a single-user edge end generation type network reasoning task, a core controller generates a data processing unit mode configuration signal in a fine-grained (tensor element level) manner, and the data processing unit mode configuration signal is transmitted by a shift distributor and flows in parallel with operation data, so that the mode configuration efficiency and the hardware resource utilization rate are greatly improved, the problem of high fan-out caused by broadcast control is reduced, and the user experience is improved. A control flow is designed by means of the structural characteristics of the systolic array, and a complex control algorithm is realized with low resource occupation.

The invention provides a flexible convolution operation accelerator based on a systolic array. The flexible convolution operation accelerator is used for solving the technical problem that the hardware scale of an existing convolution operation accelerator is difficult to flexibly adjust according to different hardware platforms. The system comprises an input cache module, a pulsation matrix, an accumulation logic unit and an output cache module which are connected in sequence, the input cache module comprises a weight cache module and an image cache module, and the weight cache module and the image cache module are both connected with the pulsation matrix. The weight cache module, the image cache module, the pulsation matrix, the accumulation logic unit and the output cache module are all connected with the controller, and the controller and the output cache module are both connected with an external memory through a BUS. The systolic array and the flexible cache module are utilized, the functions of data storage and data format conversion can be achieved at the same time, hardware logic overhead is reduced, high memory access efficiency is achieved, and configuration of the accelerator scale and various convolution parameters is supported.

Systems and methods are provided to perform multiply-accumulate operations of reduced precision numbers in a systolic array. Each row of the systolic array can receive reduced inputs from a respective reducer. The reducer can receive a particular input and generate multiple reduced inputs from the input. The reduced inputs can include reduced input data elements and / or a reduced weights. The systolic array may lack support for inputs with a first bit-length and the reducers may reduce the bit-length of a given input from the first bit-length to a second shorter bit-length and provide multiple reduced inputs with second shorter bit-length to the array. The systolic array may perform multiply-accumulate operations on each unique combination of the multiple reduced input data elements and the reduced weights to generate multiple partial outputs. The systolic array may sum the partial outputs to generate the output.

Disclosed is a method and apparatus with deep learning operations. A deep learning apparatus includes a processor, configured to support a plurality of different operation modes, including a systolic array having a plurality of multiplier accumulator (MAC) units, and a control circuit configured to respectively control, for each the plurality of different operation modes, select operations of the plurality of MAC units and data movements among the plurality of MAC units.

The invention discloses a data processing method and device for sparse matrix multiplication, equipment and a medium, and the method comprises the steps: carrying out the preprocessing of a sparse matrix, and obtaining a sparse matrix block and a sparse index corresponding to the sparse matrix block; according to the number of non-zero elements in the sparse index, corresponding index coordinates are distributed to a calculation unit PE in the one-dimensional systolic array; performing multiply-accumulate calculation according to a first calculation path in the PE, and outputting a calculation result; performing multiplication calculation according to a second calculation path in the PE, and outputting a multiplication result corresponding to the PE; performing packet summation through a packet summation network to obtain at least one packet summation result; and reordering the at least one group summation result to obtain an output matrix. Therefore, two types of sparse matrix multiplication of SDDMM and SpMM are efficiently processed by fully utilizing the characteristics of the matrix inner product and the matrix row product, and the intermediate calculation result is temporarily stored and reused, so that the data movement and storage overhead is reduced, and the data processing efficiency is improved.

Aspects of the disclosure are directed to hardware acceleration of structured sparse workloads with block quantization. A hardware accelerator can receive compressed input matrices, for example as part of a workload for training or processing a machine learning model. The hardware accelerator can multiply the compressed input matrix with a gains matrix loaded in one or more matrix multiply units (MXUs) of the hardware accelerator. The input matrices can be further provided in a block data type format, in which blocks of mantissas are represented with a single shared scaling factor. An MXU can multiply the block data, shift or cast the block data according to a shared scaling factor to generate an output product. To that end, block data type matrices exhibiting structured sparsity patterns can be accelerated without affecting the overall accuracy or quality of the output to the workload being processed.

A hardware accelerator including tiles arranged in a systolic array. At each of the tiles, the systolic array receives a first input block that includes first input matrix elements of a first input matrix. In each of a plurality of multiplication iterations, at each of the tiles, the systolic array receives a respective second input block. The systolic array computes tile products of the first input matrix elements and second input matrix elements included in the second input blocks. The systolic array adds the tile products to column-wise partial sums and transmits the column-wise partial sums to subsequent tiles along accumulator rings included in array columns of the systolic array. In a subset of the multiplication iterations, the systolic array outputs product block rows of a product matrix. The product block rows each include product matrix blocks computed as rows of the column-wise partial sums.

The mixed-base FFT hardware accelerator comprises an input logic unit, an operation logic unit, a data conversion unit and an output logic unit which are arranged on an FPGA and connected in sequence, an operation unit is arranged in the operation logic unit, the operation unit is connected with the data conversion unit, the systolic array is arranged in the operation unit, and the systolic array is connected with the data conversion unit. The pulse sequence adopts mixed base to accelerate calculation of FFT (Fast Fourier Transform); the data conversion unit comprises a data truncation module and a data splicing unit which are connected in sequence, the operation unit is connected with the data truncation module, and the data splicing unit is connected with the output logic unit; the input logic unit, the operation unit, the data truncation module, the data splicing unit and the output logic unit are all connected with the controller, and the input logic unit, the data splicing unit, the data splicing unit and the output logic unit are all connected with the DDR memory through a data bus. The method can adapt to requirements of different performances, and fast calculates FFT of different points.

To improve utilization of a systolic array, each row of the array is provided with two or more general purpose row input buses. Each of the general purpose row input data buses can be operable to transfer either feature map (FMAP) input elements or weight values into the processing elements of the corresponding row of the array. The flexibility of the general purpose row input buses allows weights to be loaded in parallel into the array to speed up weight loading times to reduce the overall latency.

The invention belongs to the field of machine learning, discloses an expert parallelism processing method and system for a large language model based on MoE, and realizes efficient parallel processing of the MoE model by dynamically distributing expert quantization bit width and sparse mode, predicting and prefetching to-be-activated expert parameters, grouping tokens to generate task queues and dynamically configuring hardware accelerators. Firstly, an importance score is calculated based on expert historical activation frequency, weight distribution and a model structure, so that quantization precision and a sparse proportion are adaptively allocated, and resource waste and precision loss of a unified strategy are avoided; secondly, predicting an expert to be activated by using a current layer hidden state and a historical activation sequence, loading parameters to a special cache in advance, reducing high-bandwidth memory access, and relieving bandwidth peak scrambling; moreover, tokens are grouped through a token-expert mapping table, a task queue is created, and dynamic configuration of a systolic array is combined, so that the problem of expert heterogeneity after compression is solved, and efficient and parallel hybrid precision matrix operation is ensured.

Systems or methods of the present disclosure may provide an integrated circuit system that includes a programmable logic device that includes a clock, one or more local controllers, programmable logic units implementing a systolic array to compute a matrix multiplication, and embedded memory blocks. The embedded memory blocks include a single port random access memory (SPRAM). The one or more local controllers are configured to, on a first set of alternating clock cycles of the clock, load matrix sub-elements from two rows of a matrix into corresponding matrix element of the SPRAM. The one or more local controllers are configured to, on a second set of alternating clock cycles of the clock, read out the matrix elements from the SPRAM to the systolic array to compute the matrix multiplication.

The embodiment of the invention provides a processor architecture supporting in-memory matrix operation based on a static memory and a processor. Wherein the processor is provided with the provided processor architecture. A tensor core module in the processor architecture comprises a plurality of matrix operation units, a plurality of in-memory computing macro units are arranged in the matrix operation units, the plurality of in-memory computing macro units are connected by adopting a pulsation data path to form a two-dimensional pulsation array, and each in-memory computing macro unit comprises a plurality of parallel memory banks. And the memory bank comprises a static memory array supporting in-memory matrix operation. Therefore, according to the processor architecture provided by the scheme, the in-memory calculation macro unit based on the static memory is introduced to serve as the matrix operation unit to support in-memory matrix operation, and by adopting the matrix operation unit, the overall performance of the processor can be improved, and the high energy efficiency requirements of generative model reasoning and training can be better met.

An OpenCL-defined scalable runtime-flexible programmable accelerator architecture for accelerating convolutional neural network (CNN) inference in cloud / edge computing is provided, referred to herein as Systolic-CNN. Existing OpenCL-defined programmable accelerators (e.g., field-programmable gate array (FPGA)-based accelerators) for CNN inference are insufficient due to limited flexibility for supporting multiple CNN models at runtime and poor scalability resulting in underutilized accelerator resources and limited computational parallelism. Systolic-CNN adopts a highly pipelined and paralleled one-dimensional (1-D) systolic array architecture, which efficiently explores both spatial and temporal parallelism for accelerating CNN inference on programmable accelerators (e.g., FPGAs). Systolic-CNN is highly scalable and parameterized, and can be easily adapted by users to efficiently utilize the coarse-grained computation resources for a given programmable accelerator. In addition, Systolic-CNN is runtime-flexible and can be time-shared to accelerate a variety of CNN models at runtime without the need to recompile the programmable accelerator kernel hardware or reprogram the programmable accelerator.

The invention discloses a data processing method and system for a multi-head potential attention model based on TTD compression and a terminal, and the method comprises the steps: constructing a large language model, and processing a plurality of linear layers in the large language model in a TTD compression and decomposition mode, thereby reducing the weight number in the model, and improving the data processing efficiency. And on a hardware level, targeted optimization is carried out on a data stream subjected to linear layer reasoning after TTD compression, so that a final model reasoning result is output. According to the method, a multi-head potential attention structure in a large language model is converted, so that the requirement for KV cache during model reasoning is reduced, the weight number is reduced, the long text output capacity of the model on edge equipment is improved, matrix calculation in the reasoning process is carried out subsequently by adopting a calculation structure of a group vector systolic array, and the reasoning efficiency is improved. And limited hardware resources are efficiently utilized.

The invention discloses an accelerator supporting an attention mechanism, which is used for realizing matrix multiplication operation, scaling operation or nonlinear activation processing related to the attention mechanism, and is characterized in that a weight cache state machine module is used for controlling weight data to flow from a system bus to a weight cache unit; the thread cache state machine module is used for controlling thread data to flow from the system bus to the thread cache unit; the attention tensor processing unit state machine module is used for controlling loading and flowing of thread data and weight data from the two-dimensional systolic array in different directions, the thread data is controlled to be transmitted to the next processing unit in each period, and the weight data is spread to the processing units in the corresponding directions in a broadcast mode; the calculation result of the two-dimensional systolic array is transmitted to different branches according to whether nonlinear activation processing needs to be executed or not. According to the method, data filling is carried out on the matrix so that the matrix can be matched with the systolic array data stream, and then the attention mechanism with the variable sequence length is accelerated through hardware.

The invention provides a high-efficiency matrix engine (RVME) architecture based on RISC-V matrix extension and a calculation method, and the architecture comprises an instruction buffering and decoding module, a matrix loading / storage module, a matrix register file, a parallel outer product array and an element-by-element operation module; the matrix register file comprises a Tile register and an Acculator register; the storage modules are respectively used for storing an input matrix and an accumulation result and supporting efficient data access and parallel computing; the matrix loading / storage module significantly improves the data loading efficiency through cache line alignment and matrix transposition optimization; the instruction buffering and decoding module cooperates with a main processor through a reordering buffer area and an instruction buffer area to ensure efficient scheduling and execution of instructions. The parallel outer product array is adopted to replace a traditional systolic array, the idle period in the calculation process is eliminated through multicast data flow scheduling and a ping-pong buffer read-write mechanism, and matrix multiplication and addition operation with high calculation utilization rate and low delay is achieved.

A system and method for matrix multiplication using a systolic array configurable between a plurality of operating modes. A pulsation processor may receive a data type indicator for matrix multiplication. For a first data type, the systolic processor may load right-hand side data from a right-hand matrix register into data processing units of the systolic array between rows 0 and M-1, and pass respective rows of left-hand side data through corresponding rows of the systolic array between rows 0 and M-1. For the second data type, the pulsation processor may segment each element of the left-hand-side data and the right-hand-side data into a respective first element half and a respective second element half, and move each element half through a corresponding row of the pulsation array between rows 0 and 2M-1.

The invention relates to a data processing method for matrix calculation, a pulsation matrix, a device, equipment and a medium. The data processing method comprises the steps that data in a first input matrix and data in a second input matrix are read, the first input matrix is provided with M rows and K columns, and the second input matrix is provided with K rows and N columns; inputting the read data of the first input matrix into the systolic matrix, and enabling the data of the first input matrix to be transversely and circularly propagated in the systolic array; inputting the read data of the second input matrix into the systolic array, and enabling the data of the second input matrix to be circularly propagated in the systolic array along the diagonal direction; the processing unit is used for executing multiply-accumulate operation on data flowing through the first input matrix and data flowing through the second input matrix, and a calculated intermediate result is resided in the processing unit; and in response to determining that the multiply-accumulate operation is completed, unidirectionally propagating the final calculation result residing in each processing unit along the diagonal direction so as to move the final calculation result out of the systolic array.

The invention discloses a self-attention mechanism calculation method, a calculation array, a calculation device, a calculation system and a computer readable storage medium. The self-attention mechanism calculation method is applied to a self-attention mechanism calculation array, the self-attention mechanism calculation array comprises a plurality of product accumulators, the plurality of product accumulators form a systolic array, and the self-attention mechanism calculation method comprises the following steps: performing calculation through the plurality of product accumulators according to a feature tensor and a weight tensor to obtain an intermediate tensor; and a result tensor is calculated according to the intermediate tensor through a plurality of product accumulators. Therefore, frequent memory access of the intermediate tensor between the self-attention mechanism calculation array and the external memory is avoided, the calculation efficiency is improved, and the problems of intermediate tensor multiplexing and storage optimization are solved.

A processing apparatus is described herein that includes a general-purpose parallel processing engine comprising a matrix accelerator including one or more systolic arrays, at least one of the one or more systolic arrays comprising multiple pipeline stages, each pipeline stage of the multiple pipeline stages including multiple processing elements, the multiple processing elements configured to perform processing operations on input matrix elements based on output sparsity metadata. The output sparsity metadata indicates to the multiple processing elements to bypass multiplication for a first row of elements of a second matrix and multiply a second row of elements of the second matrix with a column of matrix elements of a first matrix.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a systolic array scheduling processing method, device, equipment and medium, and the method comprises the steps: obtaining a data processing model, and compiling the data processing model into an initial processing task based on hardware architecture features of a systolic array, acquiring to-be-processed data and analyzing data characteristics of the to-be-processed data, monitoring a real-time operation state of the systolic array processing device, inputting the real-time operation state, the data characteristics and an initial processing task into a scheduling decision module to generate a scheduling strategy, executing the scheduling strategy to complete data processing, and collecting performance data; and updating optimization parameters of the compiling module and strategy generation parameters of the scheduling decision module based on the performance data. According to the method, the initial processing task is generated through compiling, the scheduling strategy is dynamically generated in combination with the data features and the running state, parameter updating is achieved through performance data feedback, self-adaptive closed-loop optimization of compiling and scheduling is formed, and the calculation efficiency and the energy efficiency ratio are improved.

A programmable device may be configured to support machine learning training operations using matrix multiplication circuitry. In some embodiments, the multiplication is implemented on a systolic array. The systolic array includes an array of processing elements, each of which includes hybrid floating-point dot-product circuitry.

The application discloses an edge computing platform and system based on FPGA real-time target recognition detection, and relates to the field of microelectronic chips.The edge computing platform comprises an interconnected FPGA and DDR, and an ISP module, a pre-processing module, a VDMA module, an inference accelerator, a CPU and a character superposition module are arranged on the FPGA.The inference accelerator is used for deploying a preset convolutional neural network model, reading any frame of second image and weight parameter data about convolution kernels in the convolutional neural network model from the DDR, accelerating the execution of the algorithm of the convolutional neural network model by using a sphygmic array cluster module, and generating an inference result vector output to the DDR;the sphygmic array cluster module is integrated with a Winograd fast convolution algorithm and a multi-channel sphygmic array.Compared with the prior art, the application realizes the accelerated operation of the convolutional neural network model, and thus improves the inference efficiency.

The invention relates to a network-on-chip-based reconfigurable systolic array method, device and equipment and a medium, and the method comprises the steps: obtaining a relative position relation between input data and output data; if the relative position relationship is a parallel relationship, determining an input data row from the first to Nth row nodes, and determining a first output data row from the first to Nth row nodes, determining a destination node of each input data row node in the input data rows in the first output data row according to the row number of the first output data rows and the row number of each input data row, and performing multiplication and / or additive operation by using the reconfigurable systolic array circuit based on input data of each input data row node, and outputting the data at the destination node of the first output data row. Therefore, the problems of low energy efficiency and area efficiency of matrix operation are solved, and the method has relatively high energy efficiency and area efficiency.

One embodiment provides a graphics processor comprising a base die including a plurality of chiplet sockets and a plurality of chiplets coupled with the plurality of chiplet sockets, at least one of the plurality of chiplets including a plurality of processing elements and a matrix accelerator coupled with the plurality of processing elements, the matrix accelerator having circuitry to perform a matrix multiply accumulate operation on matrix data having a tiled memory layout.

A processing apparatus includes a general-purpose parallel processing engine including a set of multiple processing elements including a single precision floating-point unit, a double precision floating point unit, and an integer unit; a matrix accelerator including one or more systolic arrays; a first register file coupled with a first read control circuit, wherein the first read control circuit couples with the set of multiple processing elements and the matrix accelerator to arbitrate read requests to the first register file from the set of multiple processing elements and the matrix accelerator; and a second register file coupled with a second read control circuit, wherein the second read control circuit couples with the matrix accelerator to arbitrate read requests to the second register file from the matrix accelerator and limit access to the second register file by the set of multiple processing elements.

The invention provides a CNN acceleration method, system and device based on a Winograd algorithm and a medium, and the method comprises the steps: training a CNN network, obtaining a trained model parameter, carrying out the fixed-point processing of the trained model parameter, and storing the model parameter in a DDR3; reading a convolution kernel from the DDR3, and converting the convolution kernel into a convolution kernel matrix in a Winograd domain by using the transformation matrix; reading input feature mapping from the DDR3, and converting the input feature mapping into an input feature mapping matrix of a Winograd domain by using the input transformation matrix; distributing the convolution kernel matrix and the input feature mapping matrix to each processing unit of a systolic array for point product operation to generate an intermediate output matrix; and collecting an intermediate output matrix, and performing inverse transformation on the intermediate output matrix by using a predefined output transformation matrix to generate an output matrix, so as to solve the problems of low calculation efficiency, large operand and low network performance and accuracy when the CNN is realized.

A neural network accelerator can perform energy-efficient multiply-and-accumulate operations of a neural network by Booth encoding a stationary operand, such as weights, before a compute phase. The Booth-encoding circuitry generates and stores Booth encoded multipliers in a Booth encoded multiplier storage and a precomputed compensation value representing a sum of the compensation bits of the Booth encoded multipliers in a Booth compensation storage. Per-cycle Booth encoding and compute of the sum of the compensation bits are avoided during multiply-accumulate operations because Booth encoding is applied to stationary operands. The Booth encoder can be located at the periphery where the multiplicands are loaded onto the accelerator shared across multiple compute columns and / or tiles to amortize the Booth encoder area overhead. The Booth encoder supports reconfigurable operand bit widths (e.g., 16-, 8-, 4-, and 2-bit). The approach is applicable to single-instruction-multiple data (SIMD) arrays, systolic arrays, and analog / digital compute-in-memory arrays.