166 results about "Systolic array" patented technology

Aspects of the disclosure are directed to hardware acceleration of structured sparse workloads with block quantization. A hardware accelerator can receive compressed input matrices, for example as part of a workload for training or processing a machine learning model. The hardware accelerator can multiply the compressed input matrix with a gains matrix loaded in one or more matrix multiply units (MXUs) of the hardware accelerator. The input matrices can be further provided in a block data type format, in which blocks of mantissas are represented with a single shared scaling factor. An MXU can multiply the block data, shift or cast the block data according to a shared scaling factor to generate an output product. To that end, block data type matrices exhibiting structured sparsity patterns can be accelerated without affecting the overall accuracy or quality of the output to the workload being processed.

A hardware accelerator including tiles arranged in a systolic array. At each of the tiles, the systolic array receives a first input block that includes first input matrix elements of a first input matrix. In each of a plurality of multiplication iterations, at each of the tiles, the systolic array receives a respective second input block. The systolic array computes tile products of the first input matrix elements and second input matrix elements included in the second input blocks. The systolic array adds the tile products to column-wise partial sums and transmits the column-wise partial sums to subsequent tiles along accumulator rings included in array columns of the systolic array. In a subset of the multiplication iterations, the systolic array outputs product block rows of a product matrix. The product block rows each include product matrix blocks computed as rows of the column-wise partial sums.

The invention belongs to the field of machine learning, discloses an expert parallelism processing method and system for a large language model based on MoE, and realizes efficient parallel processing of the MoE model by dynamically distributing expert quantization bit width and sparse mode, predicting and prefetching to-be-activated expert parameters, grouping tokens to generate task queues and dynamically configuring hardware accelerators. Firstly, an importance score is calculated based on expert historical activation frequency, weight distribution and a model structure, so that quantization precision and a sparse proportion are adaptively allocated, and resource waste and precision loss of a unified strategy are avoided; secondly, predicting an expert to be activated by using a current layer hidden state and a historical activation sequence, loading parameters to a special cache in advance, reducing high-bandwidth memory access, and relieving bandwidth peak scrambling; moreover, tokens are grouped through a token-expert mapping table, a task queue is created, and dynamic configuration of a systolic array is combined, so that the problem of expert heterogeneity after compression is solved, and efficient and parallel hybrid precision matrix operation is ensured.

Systems or methods of the present disclosure may provide an integrated circuit system that includes a programmable logic device that includes a clock, one or more local controllers, programmable logic units implementing a systolic array to compute a matrix multiplication, and embedded memory blocks. The embedded memory blocks include a single port random access memory (SPRAM). The one or more local controllers are configured to, on a first set of alternating clock cycles of the clock, load matrix sub-elements from two rows of a matrix into corresponding matrix element of the SPRAM. The one or more local controllers are configured to, on a second set of alternating clock cycles of the clock, read out the matrix elements from the SPRAM to the systolic array to compute the matrix multiplication.

An OpenCL-defined scalable runtime-flexible programmable accelerator architecture for accelerating convolutional neural network (CNN) inference in cloud / edge computing is provided, referred to herein as Systolic-CNN. Existing OpenCL-defined programmable accelerators (e.g., field-programmable gate array (FPGA)-based accelerators) for CNN inference are insufficient due to limited flexibility for supporting multiple CNN models at runtime and poor scalability resulting in underutilized accelerator resources and limited computational parallelism. Systolic-CNN adopts a highly pipelined and paralleled one-dimensional (1-D) systolic array architecture, which efficiently explores both spatial and temporal parallelism for accelerating CNN inference on programmable accelerators (e.g., FPGAs). Systolic-CNN is highly scalable and parameterized, and can be easily adapted by users to efficiently utilize the coarse-grained computation resources for a given programmable accelerator. In addition, Systolic-CNN is runtime-flexible and can be time-shared to accelerate a variety of CNN models at runtime without the need to recompile the programmable accelerator kernel hardware or reprogram the programmable accelerator.

The invention provides a high-efficiency matrix engine (RVME) architecture based on RISC-V matrix extension and a calculation method, and the architecture comprises an instruction buffering and decoding module, a matrix loading / storage module, a matrix register file, a parallel outer product array and an element-by-element operation module; the matrix register file comprises a Tile register and an Acculator register; the storage modules are respectively used for storing an input matrix and an accumulation result and supporting efficient data access and parallel computing; the matrix loading / storage module significantly improves the data loading efficiency through cache line alignment and matrix transposition optimization; the instruction buffering and decoding module cooperates with a main processor through a reordering buffer area and an instruction buffer area to ensure efficient scheduling and execution of instructions. The parallel outer product array is adopted to replace a traditional systolic array, the idle period in the calculation process is eliminated through multicast data flow scheduling and a ping-pong buffer read-write mechanism, and matrix multiplication and addition operation with high calculation utilization rate and low delay is achieved.

The invention relates to a data processing method for matrix calculation, a pulsation matrix, a device, equipment and a medium. The data processing method comprises the steps that data in a first input matrix and data in a second input matrix are read, the first input matrix is provided with M rows and K columns, and the second input matrix is provided with K rows and N columns; inputting the read data of the first input matrix into the systolic matrix, and enabling the data of the first input matrix to be transversely and circularly propagated in the systolic array; inputting the read data of the second input matrix into the systolic array, and enabling the data of the second input matrix to be circularly propagated in the systolic array along the diagonal direction; the processing unit is used for executing multiply-accumulate operation on data flowing through the first input matrix and data flowing through the second input matrix, and a calculated intermediate result is resided in the processing unit; and in response to determining that the multiply-accumulate operation is completed, unidirectionally propagating the final calculation result residing in each processing unit along the diagonal direction so as to move the final calculation result out of the systolic array.

The invention discloses a self-attention mechanism calculation method, a calculation array, a calculation device, a calculation system and a computer readable storage medium. The self-attention mechanism calculation method is applied to a self-attention mechanism calculation array, the self-attention mechanism calculation array comprises a plurality of product accumulators, the plurality of product accumulators form a systolic array, and the self-attention mechanism calculation method comprises the following steps: performing calculation through the plurality of product accumulators according to a feature tensor and a weight tensor to obtain an intermediate tensor; and a result tensor is calculated according to the intermediate tensor through a plurality of product accumulators. Therefore, frequent memory access of the intermediate tensor between the self-attention mechanism calculation array and the external memory is avoided, the calculation efficiency is improved, and the problems of intermediate tensor multiplexing and storage optimization are solved.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a systolic array scheduling processing method, device, equipment and medium, and the method comprises the steps: obtaining a data processing model, and compiling the data processing model into an initial processing task based on hardware architecture features of a systolic array, acquiring to-be-processed data and analyzing data characteristics of the to-be-processed data, monitoring a real-time operation state of the systolic array processing device, inputting the real-time operation state, the data characteristics and an initial processing task into a scheduling decision module to generate a scheduling strategy, executing the scheduling strategy to complete data processing, and collecting performance data; and updating optimization parameters of the compiling module and strategy generation parameters of the scheduling decision module based on the performance data. According to the method, the initial processing task is generated through compiling, the scheduling strategy is dynamically generated in combination with the data features and the running state, parameter updating is achieved through performance data feedback, self-adaptive closed-loop optimization of compiling and scheduling is formed, and the calculation efficiency and the energy efficiency ratio are improved.

The application discloses an edge computing platform and system based on FPGA real-time target recognition detection, and relates to the field of microelectronic chips.The edge computing platform comprises an interconnected FPGA and DDR, and an ISP module, a pre-processing module, a VDMA module, an inference accelerator, a CPU and a character superposition module are arranged on the FPGA.The inference accelerator is used for deploying a preset convolutional neural network model, reading any frame of second image and weight parameter data about convolution kernels in the convolutional neural network model from the DDR, accelerating the execution of the algorithm of the convolutional neural network model by using a sphygmic array cluster module, and generating an inference result vector output to the DDR;the sphygmic array cluster module is integrated with a Winograd fast convolution algorithm and a multi-channel sphygmic array.Compared with the prior art, the application realizes the accelerated operation of the convolutional neural network model, and thus improves the inference efficiency.

A neural network accelerator can perform energy-efficient multiply-and-accumulate operations of a neural network by Booth encoding a stationary operand, such as weights, before a compute phase. The Booth-encoding circuitry generates and stores Booth encoded multipliers in a Booth encoded multiplier storage and a precomputed compensation value representing a sum of the compensation bits of the Booth encoded multipliers in a Booth compensation storage. Per-cycle Booth encoding and compute of the sum of the compensation bits are avoided during multiply-accumulate operations because Booth encoding is applied to stationary operands. The Booth encoder can be located at the periphery where the multiplicands are loaded onto the accelerator shared across multiple compute columns and / or tiles to amortize the Booth encoder area overhead. The Booth encoder supports reconfigurable operand bit widths (e.g., 16-, 8-, 4-, and 2-bit). The approach is applicable to single-instruction-multiple data (SIMD) arrays, systolic arrays, and analog / digital compute-in-memory arrays.

The invention relates to the technical field of hardware acceleration, and provides a convolutional neural network hardware acceleration method and system, and the method comprises the steps: obtaining a convolutional neural network instruction which comprises an img2col instruction, a systolic array multiplication instruction and a col2img instruction; analyzing an operation instruction code and a function code field of each instruction, determining a target register, a source register and calculation parameters, determining an instruction function and generating a control signal; data to be operated are taken out from the source register and transmitted to the corresponding operation module together with the control signal and the calculation parameter to be calculated, and then the data are written back to the target register; through hardware optimization strategies such as multi-level address superposition, systolic array data multiplexing, double-buffer window construction and a parallel comparison tree, extra power consumption caused by multiple times of data loading and storage and intermediate result calculation is reduced, and data handling overhead, calculation delay and control complexity are greatly reduced.

The invention discloses a method and system for improving the calculation efficiency of an end-side convolution acceleration engine and a storage medium, and belongs to the technical field of convolution acceleration engines, and the method comprises the steps: determining the weight of the end-side convolution acceleration engine, the size of an input feature map and the size of an output feature map under different conditions; correspondingly dividing an on-chip memory, which is divided into a plurality of memory blocks with preset sizes in advance, of the end-side convolution acceleration engine into three on-chip memory parts according to the sizes of the weight, the input feature graph and the output feature graph and the priorities of the weight, the input feature graph and the output feature graph from high to low, the weights, the input feature maps and the output feature maps are independently stored; and carrying out convolution calculation in a systolic array mode by using the independently stored weights, the input feature map and the output feature map. According to the specification of the current convolution calculation, the features and the weights are dynamically split, and the on-chip memory is dynamically allocated, so that the number of repeated data carrying times is reduced, and the calculation efficiency of the end-side convolution acceleration engine is improved.

The vector processor and the data processing method provided by the application add a systolic array acceleration unit in a processor channel to realize the calculation between vectors. The storage unit on the original architecture is fully utilized, the data throughput is increased, the calculation between more vector data is realized, the acceleration effect of the systolic array accelerator is fully utilized, and the utilization rate of the calculation is greatly improved. The systolic array accelerator can support multi-precision and ultra-low bit quantization calculation, improve the efficiency of vector calculation, and the parallelism and scalability of the vector processor can greatly improve the data calculation density, thereby effectively improving the computing power.

The present disclosure is directed to an apparatus that includes a systolic array having an initial systolic stage that is clocked at a first edge of a clock signal. The apparatus further includes control logic configured to clock a first subset of a plurality of additional systolic stages of the systolic array at the first edge of the clock signal. The control logic is further configured to clock a second subset of the plurality of systolic stages at a second edge of the clock signal.

The invention discloses a neural network processor, which is applied to the field of artificial intelligence and comprises a systolic array, a multiplication accumulator array, a processing scheduler, a data dispatcher and a memory. The invention provides a hybrid SA / MAC array collaborative architecture, which solves the problem of high vacancy rate of a single systolic array or a multiplication accumulator array by dynamically allocating tasks through a processing scheduler, can adapt to various network layer structures of large, medium and small sizes, and improves the processing efficiency; compared with a pure multiplication accumulator array, the hardware area and the implementation complexity are reduced, and the hardware cost is reduced; compared with a pure systolic array, the data loading and carrying-out time is shortened, a small amount of tasks which cannot be efficiently completed by the systolic array are processed through the multiplication accumulator array, the systolic array is prevented from being repeatedly started, and the total calculation time consumption is reduced; the data dispatcher manages the data flow direction in a unified mode, it is ensured that data interaction between the systolic array and the multiplication accumulator array is accurate and efficient, and respective operation is not affected.

An AI-accelerating processor system may include memory that stores a value at a first precision level. The system may include a systolic array configured to perform computation. The systolic array may include rounding circuits. Each rounding circuit may round the value at the first precision level to a second precision level that is lower than the first precision level. At least a first rounding circuit and a second rounding circuit are configured to round the same value differently to respectively generate at least a first rounded value and a second rounded value. The systolic array may also include processing elements that are configured to receive a version of the value in one or more collective operations. At least a first processing element and a second processing element are configured to perform computations involving the value by respectively using the first rounded value and the second rounded value.

A systolic array includes a plurality of processing elements, the systolic array including a first systolic array and a second systolic array that are obtained by partitioning the systolic array, and a controller. The second systolic array is positioned downstream of the first systolic array. Each of the plurality of processing elements includes a selector that selectively outputs an output signal, and the controller switches the output signals of the plurality of processing elements from the selectors between first processing elements positioned on a last stage of the first systolic array and second processing elements, the second processing elements being processing elements of the second systolic array and processing elements of the first systolic array except for the first processing elements.

The present application relates to neural network inference hardware technical field, particularly to a kind of lightweight neural network processor storage architecture collaborative optimization method, comprising: step 1: the difference of on-chip data on bandwidth demand and access mode is analyzed, and the differential single-port storage setting is carried out to the buffer storage of each data in neural network processor;Step 2: the GEMM module of neural network processor is based on semi-pulsating array setting, and ALU operation fusion processing module is used, to reduce the storage bandwidth and bit width of data;Step 3: the storage structure is optimized by introducing the dynamic adjustment strategy based on timing analysis, to output the optimized neural network processor.The present application guarantees the computing performance, significantly improves the area efficiency and access efficiency of storage system, realizes the collaborative optimization of bandwidth, area and power consumption.

To prevent deterioration in operation performance in a matrix product operation of a small matrix.SOLUTION: A systolic array 10 having a plurality of processing elements 1 includes a first systolic array 10 divided from the systolic array 10, a second systolic array 10 divided from the systolic array 10 and in a subsequent stage of the first systolic array 10, and a control unit 100, and the plurality of processing elements 1 include a selector 15,16 that selects and outputs an output signal. The control unit 100 switches the output signal from the selector 15,16 between the processing element 1 in the final stage of the first systolic array 10 and the processing elements 1 in the stages other than the final stage of the first systolic array 10 and the processing elements 1 of the second systolic array 10 among the plurality of processing elements 1.SELECTED DRAWING: Figure 3

The invention belongs to the technical field of semiconductor integrated circuit technology and deep learning hardware acceleration, and relates to a computing structure configurable convolutional neural network acceleration system based on a Systolic array, which comprises a control processor and a coprocessor, the control processor is connected with the coprocessor through a high-speed expansion interface and a low-speed signal interface; the control processor is used for sending self-defined computing structure configuration data and image data to the coprocessor through the high-speed expansion interface; the control processor is further configured to send a layer indication signal, a convolution mode signal, and a computational mode signal to the coprocessor over the low speed signal interface. According to the method, parallel calculation of a plurality of basic calculation paths can be completed at the same time, calculation tasks under different input channel dimensions and different output channel dimensions in the convolution calculation process can be completed after calculation structure configuration, and the method has the advantages of being high in calculation throughput and low in energy consumption ratio and is very suitable for the high calculation power requirement of the convolutional neural network.

In one example, a neural network accelerator can execute a set of instructions to: load a first weight data element from a memory into a systolic array, the first weight data element having a first coordinate; extract, from the instructions, information indicating a first subset of input data elements to be obtained from the memory, the first subset based on a stride of a transpose convolution operation and a second coordinate of the first weight data element in a rotated array of weight data elements; obtain the first subset of input data elements from the memory based on the information; load the first subset of input data elements into the systolic array; and control the systolic array to perform a first computation based on the first weight data element and the first subset of input data elements to produce an output data element of an array of output data elements.

The application relates to a system and method for architecture-level fault injection based on systolic array accelerator reliability analysis, relates to the technical field of CNN accelerator analysis, and solves the problems that, in the existing CNN reliability analysis method, the reliability analysis of the software level and the architecture level produces contradictory conclusions, and the analysis from the software level alone cannot meet the actual reliability characteristics of the underlying hardware application, and the like, the system comprises an execution simulator, a fault injection module and an analysis module; the execution simulator is responsible for calculation simulation, the fault injection module customizes and simulates fault generation, and the analysis module performs data analysis and result recording; by adopting the method, period-accurate fault injection can be performed on three types of registers under different systolic array sizes, a plurality of different fault types are supported, and the analysis framework can generate detailed intermediate process data information and result record information; the application is simple to implement, cross-platform, has good expansibility, and meets the application requirements.

The application belongs to the field of information technology and provides a column reconfigurable systolic array for a Transformer model. The main idea is to realize that each column computing unit of the array can work together for a single operator or can be split to work individually for multiple operators. The main scheme includes a reconfigurable computing unit of a two-dimensional network, which is composed of row and column distributed computing units, data is transferred from the previous row to the next row and from the previous column to the next column; a register unit of a two-dimensional network, which is composed of row and column distributed register units, data is transferred from the previous row to the next row and from the previous column to the next column, and the data transfer direction is opposite to that of the reconfigurable computing unit of the two-dimensional network; and the register and the computing unit of the same row and column are connected through a data path. A mixed parallel line is supported to improve the hardware efficiency of the Transformer-based model.

A specialized integrated circuit for artificial neural networks is integrated with high bandwidth memory. The neural network includes a systolic array of interconnected processing elements, including upstream processing elements and downstream processing elements. Each processing element includes a pair of input / output ports for concurrent forward and backward propagation. The processing elements can be used for convolution, in which case the pair of input / output ports can support fast and efficient scanning of a kernel over activations.

The invention relates to an improved systolic array computing device capable of improving pipeline efficiency, the device comprises an input characteristic value cache array, a multiply-accumulate array and an output characteristic value cache array, and the multiply-accumulate array is composed of N * N multiply-accumulate units. In order to improve the utilization rate of the array, data paths connected end to end are added in the horizontal direction and the vertical direction respectively, so that the array forms a Torus annular interconnection structure; input characteristic values are injected through diagonal positions, partial sum results are annularly accumulated in the vertical direction, so that continuous flow of input and partial sum is achieved, the structure remarkably reduces assembly line filling and emptying delay, hardware redundancy caused by input and output FIFO arrays in a traditional systolic array is avoided, calculation of a 512 * 512 matrix through a 256 * 256 array is taken as an example, and the structure has the advantages of being simple in structure, convenient to operate and low in cost. The calculation delay is reduced from 2560 clock cycles to 2176 clock cycles, and the array utilization rate is improved to 94%. The method has the advantages of being high in efficiency, low in cost, good in expansibility and suitable for neural network and matrix calculation acceleration.

A neural network accelerator based on a multi-channel systolic array includes a novel systolic array composed of a configurable number of basic computational units. This systolic array includes a configurable number and direction of weight data stream channels and feature data stream channels. The multi-channel systolic array can arbitrate the allocation of data stream channels and basic computational units according to the scale of the neural network, achieving parallel four-thread acceleration for up to four types of neural networks. The multi-channel systolic array feeds a row of feature data in each computation cycle, using a data stream composed of entire rows of data to improve upon the discrete data stream, significantly reducing acceleration processing time. This invention achieves accelerated neural network computation based on a multi-channel systolic array, which significantly optimizes acceleration processing time and hardware resource utilization compared to traditional systolic arrays; it can achieve more efficient accelerated computation while saving hardware overhead.

The invention discloses a QR decomposition matrix inversion optimization method and system. The method comprises the following steps: receiving signal data, and carrying out FFT (Fast Fourier Transform) preprocessing and data downsampling; double-buffer optimization parallel processing is adopted, and FFT signal data are stored based on a double-buffer FIFO data storage structure; calculating a covariance matrix of the FFT signal data; adopting Givens rotation to gradually eliminate off-diagonal elements of the covariance matrix, carrying out Givens rotation calculation, and adopting an optimized CORDIC calculation method to calculate a Givens rotation factor; a calculation process of Givens rotation is accelerated by adopting a Systolic array architecture; and outputting a final QR decomposition result according to a Givens rotation calculation result. According to the method, the data flow is optimized, the calculation throughput rate is improved, hardware resource occupation is reduced, a low-power-consumption and high-performance hardware implementation scheme is provided for efficient matrix calculation, and the method has wide engineering application value.

The invention discloses a hardware-friendly Transform column balance pruning model compression and efficient deployment method. A model compression algorithm, a lightweight parameter storage format, an operation data buffer, a systolic array operation block, a vector operation unit, a nonlinear operator unit, a data flow controller and a DMA unit are included. A model compression algorithm and an efficient deployment architecture are explored according to Transform network software and hardware collaborative reasoning requirements: in a software level, the scale calculation complexity of model parameter quantities is reduced through a fine-grained column balance structured pruning strategy, and parameters are stored in a single-instruction multi-data-stream format and the parameter storage efficiency is optimized through mask code storage; according to the hardware level, an edge computing-oriented Transform special accelerator architecture is designed, so that the architecture can support column balance structured pruning characteristics and a lightweight parameter storage scheme in an original manner. According to the Transform model compression and efficient deployment method, the parameter sparsity after structured pruning is fully utilized, so that the parameter storage pressure of a hardware architecture is reduced, the complex balance of an arithmetic unit is ensured, the operation efficiency of an accelerator is improved, and load balance and efficient reasoning during software and hardware collaborative optimization are realized; the method is widely applicable to efficient deployment scenes of Transform models for edge calculation.

The invention provides a dynamic folding and unfolding method based on a Systolic array, belongs to the technical field of integrated circuit design and parallel computing, and aims at solving the problems of resource idleness, bandwidth bottleneck and other pain points of a traditional Systolic array matrix multiplication, and provides a 4 * 4 basic array dynamic folding / unfolding framework from four aspects of an array structure, an operation mode and the like. A storage / pulse / half-flow mode is intelligently switched, a three-dimensional resource collaborative scheduling mechanism is established, a QC write-back channel is reserved, resources are utilized to the maximum extent, and the core problem of a traditional scheme is solved.