80 results about "Sparse matrix multiplication" patented technology

The invention discloses a tensor core-based method for accelerating arbitrary precision sparse matrix multiplication and addition operation, which comprises the following steps of: aiming at an architecture of a tensor core on A100 and a programming model thereof, combining program characteristics of deep learning parallel processing sparse matrix, combining an arbitrary precision sparse matrix multiplication and addition operation operator with parallel computing acceleration equipment, and performing parallel computing acceleration on the sparse matrix. The method comprises the following steps: decomposing a general matrix into a sparse matrix in a set format, partitioning the sparse matrix and assigning a corresponding parallel computing hardware unit for the sparse matrix, reducing memory conflicts and access delay by using a shared memory, calling a high-level language API function library to execute loading, computing and storing processes of a computing data volume, and finally performing specification on output data of a function. Therefore, the final result of the random-precision sparse matrix multiplication and addition operation on the parallel computing acceleration equipment is obtained. According to the method, parallel computing resources such as the tensor core on the A100 can be fully utilized, the program calculation amount and the communication overhead are effectively reduced, and the program operation performance is greatly improved.

The invention discloses an image super-resolution method and system based on semantic perception tokens, and relates to the technical field of computer vision, and the method comprises the steps: generating semantic confidence and grouping information through the aggregation of content perception tokens, and decoupling a basic residual error into a texture enhancement and degradation inhibition guidance graph; in combination with a static semantic constraint mask and a sparse matrix multiplication mechanism, progressive focusing of attention is realized; a diffusion time step embedding and cooperative modulator is introduced, semantic guidance information is dynamically injected into a multi-step denoising process, adaptive attention features and diffusion reconstruction features are fused, and finally a high-fidelity and high-resolution image is output. According to the method, content-adaptive high-resolution image reconstruction is realized through collaborative modulation of a sparse attention mechanism guided by semantic grouping and diffusion denoising guided by semantic decoupling.

The invention relates to an efficient sparse attention algorithm optimization method and device, equipment and a medium, and the method comprises the steps: obtaining to-be-processed data, and carrying out the layernorm calculation of the to-be-processed data based on importance perception, and obtaining the normalized data; performing row product sparse matrix multiplication and packet hash multi-head attention processing on the normalized data to obtain an attention score matrix; performing softmax calculation based on shift operation according to the attention score matrix to obtain a softmax calculation result; and performing row product sparse matrix multiplication on the softmax calculation result and the attention matrix median vector to obtain an attention calculation result. In the whole process, the sparse attention algorithm is optimized through improved layernorm, softmax and matrix multiplication, the data volume in the attention algorithm processing process is reduced, and the efficiency of the attention algorithm can be remarkably improved.

The invention discloses a multiplication acceleration method and device for double sparse matrixes, equipment, a medium and a product. The method comprises the following steps: positioning a compression matrix set matched with a first sparse matrix and a second sparse matrix in a global memory of a computing chip according to a sparse matrix multiplication demand; carrying the compression matrix set and the second sparse matrix step by step from the global memory to a hardware register of the computing chip in a data block form; and gradually calculating a multiplication result of the first sparse matrix and the second sparse matrix through a sparse calculation unit of the calculation chip according to the data loaded in the hardware register by several times. According to the technical scheme of the embodiment of the invention, the hardware acceleration performance of the sparse computing unit in the computing chip can be brought into full play, so that the overhead of computing, bandwidth and storage resources in the double sparse matrix multiplication process is greatly optimized, and the method is particularly suitable for a model computing scene of a hybrid expert model.

One embodiment provides a graphics processor comprising a memory interface and a processing cluster array including a plurality of processing resources interconnected via a switched interconnect network, at least one of the plurality of processing resources including a matrix accelerator configured to execute an instruction to perform a multi-dimensional sparse matrix multiply and accumulate operation having input in a sparse microscaling format including merged sparsity and scaling metadata.

The invention discloses a hardware accelerator and equipment oriented to triple sparse matrix multiplication and an application method thereof.The hardware accelerator comprises a high-bandwidth memory HBM, a crossbar switch network and an on-chip processing unit which are connected in sequence, and the on-chip processing unit comprises a hierarchical cache module, a global controller and a plurality of computing chips; each calculation piece comprises an RA calculation array, a TP calculation array and a local controller, wherein the RA calculation array and the TP calculation array are respectively used for executing front-end operation T = R * A and rear-end operation C = T * P in triple sparse matrix multiplication. The method aims at solving the problem that when a traditional universal processor processes triple sparse matrix multiplication, due to irregular memory access, uneven calculation load and sharp increase of middle parts and results, huge off-chip data carrying is confronted with serious performance and energy efficiency bottlenecks, and the calculation performance and energy efficiency of triple sparse matrix multiplication are improved.

The invention discloses a data processing method and device for sparse matrix multiplication, equipment and a medium, and the method comprises the steps: carrying out the preprocessing of a sparse matrix, and obtaining a sparse matrix block and a sparse index corresponding to the sparse matrix block; according to the number of non-zero elements in the sparse index, corresponding index coordinates are distributed to a calculation unit PE in the one-dimensional systolic array; performing multiply-accumulate calculation according to a first calculation path in the PE, and outputting a calculation result; performing multiplication calculation according to a second calculation path in the PE, and outputting a multiplication result corresponding to the PE; performing packet summation through a packet summation network to obtain at least one packet summation result; and reordering the at least one group summation result to obtain an output matrix. Therefore, two types of sparse matrix multiplication of SDDMM and SpMM are efficiently processed by fully utilizing the characteristics of the matrix inner product and the matrix row product, and the intermediate calculation result is temporarily stored and reused, so that the data movement and storage overhead is reduced, and the data processing efficiency is improved.

The invention provides a data processing acceleration method and device for a sparse matrix multiplication operator, and the method comprises the steps: obtaining the sparse matrix multiplication operator needing to be processed in a data processing process, compressing a sparse matrix, and enabling a storage form of the compressed sparse matrix to comprise a non-zero element array, a row pointer array, a global column index array and a local column index array. The global column indexes record the column indexes of non-zero elements in the sparse matrix in a group of continuous preset number of sparse rows divided by the sparse matrix, and the global column indexes in each column index array are not repeated. The local column index records a position index of each non-zero element in the global column index array. And accessing corresponding row data in the dense matrix by using the global column index and reading the row data to the temporary cache region. And accessing corresponding row data in the dense matrix in the temporary cache region by using the local column index. And calculating and accumulating products of the non-zero elements and corresponding row data in the dense matrix, and outputting a result matrix. The data processing speed can be increased, and the memory access overhead can be reduced.

The embodiment of the invention discloses a hybrid expert model optimization method, device, equipment, medium and program, and the method comprises the steps: carrying out the sparse conversion of each sparse data structure of a target hybrid expert model, and obtaining a structured sparse data structure; performing data arrangement optimization on the structured sparse data structure; optimizing an operator of the target hybrid expert model in combination with a sparse calculation unit of a calculation chip to obtain a sparse operator; and updating an original operator of the target hybrid expert model according to the sparse operator to obtain a structured sparse hybrid expert model. According to the technical scheme provided by the embodiment of the invention, the structured sparse optimization of the hybrid expert model can be realized, so that the acceleration performance of sparse calculation hardware of the hybrid expert model during the calculation of sparse matrix multiplication can be fully exerted, and the overhead of calculation, bandwidth and storage resources in the sparse matrix multiplication operation process can be greatly optimized.

The invention provides a GPU acceleration system and method based on medium sparseness large language model sparse matrix multiplication, and belongs to the technical field of computers. The system cuts a large model weight matrix through an unstructured cutting module to obtain a sparse weight matrix; a sparse matrix is converted into an MLSF format through a sparse tensor kernel layer, a slot filling layer and a residual element layer in the sparse matrix preprocessing module so as to minimize storage and decoding overhead; a sparse matrix multiplication GPU operation module is used, an MLSF format and a dense matrix are used as input, a fragmentation strategy and asynchronous loading are combined, a pipeline mechanism of data loading, decoding and calculation is constructed, and high overlapping of calculation and memory access operation is achieved. According to the method, dense matrix calculation with sparse matrix multiplication performance exceeding highly optimized performance is realized under medium sparseness, performance improvement of 5.29 times at most is obtained, and technical support is provided for efficient deployment of a large language model.

The disclosure relates to a method and a system for predicting the operation time of sparse matrix vector multiplication. The method comprises constructing a convolutional neural network comprising an input layer, a feature processing layer, a data splicing layer and an output layer for outputting prediction results. The method further comprises acquiring a plurality of groups of sparse matrices with known sparse matrix vector multiplication operation time as sample data, inputting the sample data into the convolutional neural network to train the convolutional neural network, and inputting the sparse matrix to be classified into the trained convolutional neural network to realize the prediction of the operation time of sparse matrix vector multiplication.

Aspects of the disclosure provide for methods, systems, and apparatuses, including computer-readable storage media, for sparse matrix multiplication. A system for matrix multiplication includes an array of sparse shards. Each sparse shard can be configured to receive an input sub-matrix and an input sub-vector, where the input sub-matrix has a number of non-zero values equal to or less than a predetermined maximum non-zero threshold. The sparse shard can, by a plurality of multiplier circuits, compute one or more products of vector values multiplied with respective non-zero values of the input sub-matrix. The sparse shard can generate, as output to the sparse shard and using the one or more products, a shard output vector that is the product of applying the shard input vector to the shard input matrix.

An operation method for a sparse matrix, a processor, and a computing device. The operation method executed by a processor comprises: a sparse matrix processing unit acquiring a first numerical matrix and a first index matrix corresponding to a first sparse matrix, and a second numerical matrix and a second index matrix corresponding to a second sparse matrix (401); inputting non-zero elements of each column in the first numerical matrix and a row number indicated by each column in the first index matrix, non-zero elements of each row in the second numerical matrix and a column number indicated by each row in the second index matrix into a sparse matrix operation unit (402); and, on the basis of the row number indicated by each column in the first index matrix and the column number indicated by each row in the second index matrix, the sparse matrix operation unit performing outer product operation on the non-zero elements of each column in the first numerical matrix and of each row in the second numerical matrix, so as to obtain a result matrix generated by performing matrix multiplication operation on the first sparse matrix and the second sparse matrix (403). The present application can reduce the number of zero elements participating in sparse matrix multiplication, thus improving the efficiency of performing sparse matrix multiplication.

The invention discloses a matrix multiplication optimization method and system based on an RISC-V architecture, belongs to the technical field of machine learning, and aims to solve the technical problem of how to effectively improve the performance of sparse matrix multiplication. Comprising the following steps: constructing a sparse matrix adaptive to a machine learning model; the sparse matrix is converted into a structured sparse matrix, non-zero elements in the structured sparse matrix are stored in a vector register file, and column indexes corresponding to the non-zero elements are stored in a scalar register file; determining expansion factors of the internal circulation and the external circulation, and adjusting the expansion factors of the internal circulation; executing instructions in different loop iterations in a staggered manner in a staggered manner; designing and realizing a user-defined vector index multiply-add instruction; a custom vector index multiply-add instruction is executed.

The name of the invention is combining MX and sparsity representation. One embodiment provides a graphics processor comprising a memory interface and a processing cluster array comprising a plurality of processing resources interconnected via a switched interconnect network, at least one of the plurality of processing resources comprising a matrix accelerator, the matrix accelerator is configured to execute instructions to perform multi-dimensional sparse matrix multiplication and accumulation operations with inputs in a sparse microscaling format that includes merged sparsity and scaling metadata.

Disclosed embodiments relate to accelerating multiplication of sparse matrices. In one example, a processor is to fetch and decode an instruction having fields to specify locations of first, second, and third matrices, and an opcode indicating the processor is to multiply and accumulate matching non-zero (NZ) elements of the first and second matrices with corresponding elements of the third matrix, and executing the decoded instruction as per the opcode to generate NZ bitmasks for the first and second matrices, broadcast up to two NZ elements at a time from each row of the first matrix and each column of the second matrix to a processing engine (PE) grid, each PE to multiply and accumulate matching NZ elements of the first and second matrices with corresponding elements of the third matrix. Each PE further to store an NZ element for use in a subsequent multiplications.

The invention relates to the technical field of sparse matrix multiplication, and discloses a high-performance SpMM kernel implementation method based on a dense tensor core. The method comprises the following steps: storing a sparse matrix in a BlockNM sparse format, and pre-fetching metadata in the sparse matrix into a shared memory of a GPU (Graphics Processing Unit); asynchronously copying the sparse matrix and the dense matrix to a shared memory, setting a four-stage pipeline scheduling strategy, and controlling SpMM operations of different iteration processes; loading the sparse matrix and the dense matrix in the shared memory into a register, setting an offset for each thread accessing the shared memory, and enabling each thread to access different banks in the shared memory based on the offset; the matrix multiplication and addition operation is executed in the register to obtain a calculation result, the calculation result is temporarily stored in the shared memory from the register, then the calculation result in the shared memory is written back to the global memory, and the efficiency of sparse matrix multiplication is improved.

The invention provides a sparse matrix multiplication acceleration method based on a tensor processor, which relates to the field of high-performance computing, and comprises the following steps: carrying out block processing on a sparse matrix to obtain sparse matrix blocks matched with the register granularity of the tensor processor; performing block processing on the dense matrix to obtain dense matrix blocks; generating a matrix multiplication task set based on the sparse matrix block and the dense matrix block; and distributing the task set to a plurality of threads, controlling the plurality of threads to respectively call the tensor processors to execute matrix multiplication operation, and aggregating calculation results of the threads to obtain an output matrix. Compared with the prior art, the sparse matrix blocks are processed into the sparse matrix blocks matched with the register granularity of the tensor processor, the index coding and decoding overhead, the reconstruction overhead and the communication overhead of the sparse matrix are reduced, and therefore the operation performance of sparse matrix multiplication based on the tensor processor is improved. The system has the same beneficial effects.

The invention discloses an AI data processing method and system based on edge computing, and relates to the field of edge AI processing, and the method comprises the steps: defining a weight matrix of an AI model according to an edge computing node address, inputting structured data into the AI model for multi-dimensional feature analysis, and extracting high-order data representation; analyzing the processing mode through an instruction decoding method to obtain a sparseness level, and performing parameter importance evaluation through a dynamic importance scoring method based on distribution characteristics represented by high-order data and an aging weight value to generate a deformable sparse mask matrix; performing dynamic pruning fusion on a weight matrix of the AI model through sparse matrix multiplication based on the deformable sparse mask matrix to generate a sparse data processing graph; according to the method, through node resource sensing and feature mapping, the AI model weight matrix is accurately matched with the edge computing node capability, and the operation efficiency and stability of the AI model on the edge computing node are improved.

The invention discloses a parallel acceleration method suitable for a two-phase color gradient lattice Boltzmann method, and relates to the technical field of computational fluid mechanics. By taking the total number of fluid points and boundary solid points in the porous medium as the size of a data space, distributing a storage space for the fluid points and the boundary solid points, and constructing a storage information array; the storage information array comprises storage positions of each fluid point and each boundary solid point and a mapping relation between the storage positions and points in the adjacent direction; carrying out regional division on the porous medium to obtain a plurality of flow field regions; a graphics processor is distributed to each flow field area, and according to the storage information array, parallel numerical simulation of a color gradient lattice Boltzmann method is carried out on each flow field area through each graphics processor; in the numerical simulation process, sparse matrix multiplication is adopted as matrix multiplication during collision simulation. The method can improve the efficiency of multiphase flow numerical simulation through the color gradient LBM.

The invention discloses a sparse reasoning method, system, equipment and product for large language model reasoning, and relates to the technical field of artificial intelligence. The method comprises the following steps: firstly, acquiring a weight matrix of a large language model, and then carrying out pruning sparsification processing on the weight matrix to obtain a sparse matrix; the storage format of the sparse matrix is converted into a bitmap coding storage format which is suitable for tensor calculation core perception and adopts a multi-level block structure to respectively correspond to different calculation granularities in a GPU / NPU hardware architecture so as to obtain a sparse model, and then the sparse model is deployed to respond to a reasoning request to perform reasoning service; sparse matrix multiplication is completed through a sparse matrix multiplication kernel based on on-chip storage bitmap coding so as to perform reasoning, so that through an innovative sparse matrix storage format and calculation optimization, the storage efficiency and the calculation performance in the reasoning process are remarkably improved, and especially in a low-sparseness scene, the reasoning efficiency is greatly improved. And the performance blank of the existing sparse reasoning framework in the field is filled.

The present disclosure provides a neural network training method and a computing device, which relates to the technical field of artificial intelligence. The scheme includes: splitting a weight matrix into В weight blocks each having the same size and satisfying N:M sparsity constraint; and then by using В sparse mask sets defined for the В weight blocks respectively and В random number indices generated for the В weight blocks respectively, generating a mask matrix for implementing sparse matrix multiplication operation. There is no need to pre-train a dense weight matrix of huge-scale in advance nor to fine-tune a trained model again for the N:M sparsity constraint, which saves a lot of training time and improves training efficiency.

The present invention discloses a method, apparatus, device, medium and product for accelerating the multiplication of double sparse matrices. The method includes: locating a set of compressed matrices matching a first sparse matrix and a second sparse matrix in the global memory of a computing chip according to the sparse matrix multiplication requirement; sequentially transferring the set of compressed matrices and the second sparse matrix from the global memory to the hardware registers of the computing chip in the form of data blocks; and gradually calculating the multiplication result of the first sparse matrix and the second sparse matrix by a sparse computing unit of the computing chip according to the data loaded in batches in the hardware registers. The technical solution of the embodiments of the present invention can give full play to the hardware acceleration performance of the sparse computing unit in the computing chip, and thus greatly optimize the overhead of computing, bandwidth and storage resources in the process of multiplying double sparse matrices, and is particularly applicable to the model calculation scenario of a mixture-of-experts model.

The present application is applicable to the technical field of integrated circuits and neural networks. Provided are a matrix multiplication apparatus and method based on a systolic array, and an electronic device. The matrix multiplication apparatus based on a systolic array comprises: a systolic array formed by means of arranging processing elements, wherein two adjacent processing elements in each column are connected by a shift, each processing element has transmission channels in a row direction, a column direction and a diagonal direction, the row direction is used for inputting a row vector of a first matrix, and the column direction and the diagonal direction are used for inputting a column vector of a second matrix; a plurality of first control signal channels, which are used for inputting a first control signal for controlling the operating state of each shift; and a plurality of second control signal channels, which are used for inputting a second control signal for controlling each column vector of the second matrix to be input in the column direction or the diagonal direction. Therefore, a hardware structure taking into consideration a sparse matrix multiplication operation and a dense matrix multiplication operation is achieved, and the calculation complexity of matrix multiplication is reduced.

The present invention discloses a sparse matrix multiplication array with high DSP resource utilization rate for graph neural networks based on FPGA, including a preprocessing module for generating required functional configuration parameters, eigenvector matrix, and adjacency matrix according to the used GNN model and its dataset, and controlling the slice size to slice the eigenvector matrix and adjacency matrix; an eigenmatrix cache module for caching the valid values of the eigenvector matrix after slicing; an adjacency matrix cache module for caching the valid values of the adjacency matrix after slicing; a congestion mitigation array for transmitting the valid values of the adjacency matrix to the pairing array according to the congestion mitigation strategy; the pairing array obtains the valid values of the adjacency matrix transmitted by the congestion mitigation array and the valid values of the eigenvector matrix transmitted by the eigenmatrix cache module, and inputs the valid values of the adjacency matrix and the valid values of the eigenvector matrix into the multiplication array in a certain order; the multiplication array performs multiplication processing according to the received valid values of the adjacency matrix and the valid values of the eigenvector matrix, and outputs a partial aggregation result.

The invention discloses a method and a device for generating a calculation model for sparse matrix multiplication. The method comprises the following steps: acquiring a matrix data set for training; determining a first position of an all-zero data block in a first matrix and a second position of an all-zero data block in a second matrix in the matrix data set; updating network parameters of the initial deep neural network based on the first matrix, the second matrix, the first position and the second position; and repeating the above steps until a training stopping condition is reached, obtaining the target deep neural network, and determining the target deep neural network as a calculation model for sparse matrix multiplication. Therefore, through sparse information of the first position of the all-zero data block in the first matrix and the second position of the all-zero data block in the second matrix in the matrix data set, calculation saving brought by sparsity is considered in the training process of the calculation model, so that the multiplication times required by matrix multiplication are reduced, and the calculation complexity is reduced.

The invention provides a hybrid expert model reasoning optimization method and device and electronic equipment. The method comprises the following steps: receiving a plurality of expert models of a hybrid expert model distributed by a terminal, and receiving operation weights of the plurality of expert models, gating weights of the hybrid expert model and feature vectors of text data sent by the terminal; based on the feature vector and the gating weight of the text data, calculating selection weights of a plurality of expert models, determining a plurality of target expert models from the plurality of expert models, and generating masks of the plurality of target expert models; based on the feature vector of the text data, the masks of the multiple target expert models and the operation weights of the multiple target expert models, sparse matrix multiplication is carried out, and multiple calculation results are obtained; and summarizing the plurality of calculation results to obtain a final calculation result. According to the hybrid expert model reasoning optimization method provided by the invention, the target expert model is determined by adopting the equipment side, and sparse matrix multiplication is performed, so that the data transmission time delay is reduced, and the calculation efficiency is improved.

The application discloses sparse matrix multiplication acceleration hardware, a recommendation system acceleration method and an AI chip. A data loading unit in the hardware establishes a data carrying task for dense data in a sparse matrix multiplication task, carries the dense data to a register, a normalization acceleration unit group sends each left and right matrix block to a multiplication calculation unit group after performing a pre-multiplication activation operation according to the left and right matrix blocks obtained from the register, the multiplication calculation unit group is used for selecting a multiplication calculation unit to perform multiplication calculation according to the data size of the left and right matrix blocks, and providing a calculation result to an addition calculation unit group to perform addition calculation, and the normalization acceleration unit group is also used for performing normalization calculation and / or activation operation according to the addition calculation result when the post-multiplication normalization and / or post-multiplication activation operation is needed, and obtaining a final result. The technical scheme of the embodiment can realize fast sparse matrix multiplication calculation under limited calculation resources.

The application discloses a kind of semi-precision sparse matrix multiplication multicore parallel method and device for vector processor.There are three kinds of multicore parallel modes according to the dimension of matrix and the number of computing core, suitable for a variety of computing scenarios, make full use of the multicore architecture of vector processor.At the same time, it reduces the calculation redundancy under part of matrix dimension specification, improves the parallelism of sparse matrix multiplication calculation, helps to play the computing performance of vector processor.Each multicore parallel mode is to parallel multiple computing cores in the dimension of weight matrix and dense input matrix, and to realize sparse matrix multiplication in different dimensions.The theoretical calculation efficiency of sparse matrix multiplication calculation in each multicore parallel mode is obtained based on the dimension specification of two matrices.Then the multicore parallel mode with the maximum theoretical calculation efficiency is selected for sparse matrix multiplication calculation.This can automatically adapt the optimal mode to perform calculation, with high versatility and improved calculation efficiency.

The invention discloses a sparse matrix multiplication calculation method and device. The method comprises the following steps: acquiring a first matrix and a second matrix which need to be subjected to matrix multiplication; determining a target dimension of the basic block, and dividing the first matrix and the second matrix based on the target dimension to obtain a corresponding first target data block and a corresponding second target data block; performing dynamic planning solution on the first target data block and the second target data block to obtain an optimal calculation strategy set; and calculating the first target data block and the second target data block based on the optimal calculation strategy set to obtain a target calculation result. According to the method, the first target data block and the second target data block divided based on the basic blocks are dynamically planned and solved to obtain the optimal calculation strategy set, so that the optimal calculation strategy set automatically adapts to matrix multiplication of different sizes and sparseness levels, and a more efficient and more flexible sparse matrix acceleration scheme is provided for LLM.