30 results about "Sparse matrix multiplication" patented technology

The invention discloses a hardware accelerator and equipment oriented to triple sparse matrix multiplication and an application method thereof.The hardware accelerator comprises a high-bandwidth memory HBM, a crossbar switch network and an on-chip processing unit which are connected in sequence, and the on-chip processing unit comprises a hierarchical cache module, a global controller and a plurality of computing chips; each calculation piece comprises an RA calculation array, a TP calculation array and a local controller, wherein the RA calculation array and the TP calculation array are respectively used for executing front-end operation T = R * A and rear-end operation C = T * P in triple sparse matrix multiplication. The method aims at solving the problem that when a traditional universal processor processes triple sparse matrix multiplication, due to irregular memory access, uneven calculation load and sharp increase of middle parts and results, huge off-chip data carrying is confronted with serious performance and energy efficiency bottlenecks, and the calculation performance and energy efficiency of triple sparse matrix multiplication are improved.

The invention provides a sparse matrix multiplication acceleration method based on a tensor processor, which relates to the field of high-performance computing, and comprises the following steps: carrying out block processing on a sparse matrix to obtain sparse matrix blocks matched with the register granularity of the tensor processor; performing block processing on the dense matrix to obtain dense matrix blocks; generating a matrix multiplication task set based on the sparse matrix block and the dense matrix block; and distributing the task set to a plurality of threads, controlling the plurality of threads to respectively call the tensor processors to execute matrix multiplication operation, and aggregating calculation results of the threads to obtain an output matrix. Compared with the prior art, the sparse matrix blocks are processed into the sparse matrix blocks matched with the register granularity of the tensor processor, the index coding and decoding overhead, the reconstruction overhead and the communication overhead of the sparse matrix are reduced, and therefore the operation performance of sparse matrix multiplication based on the tensor processor is improved. The system has the same beneficial effects.

The invention discloses a sparse reasoning method, system, equipment and product for large language model reasoning, and relates to the technical field of artificial intelligence. The method comprises the following steps: firstly, acquiring a weight matrix of a large language model, and then carrying out pruning sparsification processing on the weight matrix to obtain a sparse matrix; the storage format of the sparse matrix is converted into a bitmap coding storage format which is suitable for tensor calculation core perception and adopts a multi-level block structure to respectively correspond to different calculation granularities in a GPU / NPU hardware architecture so as to obtain a sparse model, and then the sparse model is deployed to respond to a reasoning request to perform reasoning service; sparse matrix multiplication is completed through a sparse matrix multiplication kernel based on on-chip storage bitmap coding so as to perform reasoning, so that through an innovative sparse matrix storage format and calculation optimization, the storage efficiency and the calculation performance in the reasoning process are remarkably improved, and especially in a low-sparseness scene, the reasoning efficiency is greatly improved. And the performance blank of the existing sparse reasoning framework in the field is filled.

The application discloses a kind of semi-precision sparse matrix multiplication multicore parallel method and device for vector processor.There are three kinds of multicore parallel modes according to the dimension of matrix and the number of computing core, suitable for a variety of computing scenarios, make full use of the multicore architecture of vector processor.At the same time, it reduces the calculation redundancy under part of matrix dimension specification, improves the parallelism of sparse matrix multiplication calculation, helps to play the computing performance of vector processor.Each multicore parallel mode is to parallel multiple computing cores in the dimension of weight matrix and dense input matrix, and to realize sparse matrix multiplication in different dimensions.The theoretical calculation efficiency of sparse matrix multiplication calculation in each multicore parallel mode is obtained based on the dimension specification of two matrices.Then the multicore parallel mode with the maximum theoretical calculation efficiency is selected for sparse matrix multiplication calculation.This can automatically adapt the optimal mode to perform calculation, with high versatility and improved calculation efficiency.

Apparatuses, systems, and methods to enable matrix multiplication acceleration by modifying an input to apply sparsity through sparse activation filtering. In at least one embodiment, a neural network modifies pixels within an image through sparse activation filtering to enable use of one or more matrix multiplication acceleration units to perform a sparse patch embedding operation.

The invention relates to an N: M sparse matrix multiplication operator optimization method and device, equipment and a storage medium. The method comprises the following steps of: performing task decomposition on N: M sparse matrix multiplication by using a multi-layer partitioning algorithm based on a multi-level scheduling abstraction and storage structure of a GPU (Graphics Processing Unit); carrying out sparsity perception preprocessing on the N: M sparse matrix multiplication after task decomposition, and carrying out optimization processing on input matrixes in different sparsity scenes by using a packaging strategy and a non-packaging strategy respectively; and optimizing the N: M sparse matrix multiplication workflow by adopting prefetching, and calculating the optimized input matrix according to the optimized N: M sparse matrix multiplication workflow to obtain a final calculation result. According to the embodiment of the invention, performance optimization of N: M sparse matrix multiplication is realized, wide input scenes can be adapted, and efficient reasoning on the premise of not influencing the model effect is realized.

The invention provides a sparse matrix multiplication device and method and a control method, and the method comprises the steps: a mask generation module reads a first matrix and a second matrix from a memory, sets the position corresponding to each non-zero element of the first matrix in a first register to be 1, and sets the position corresponding to each zero element to be 0; setting a position corresponding to each non-zero element of a second matrix in a second register to be 1, and setting a position corresponding to each zero element to be 0; the data compression module compresses each non-zero element value of the first matrix and the second matrix and the position index of each non-zero element value; the mask analysis module performs logic AND operation on values of corresponding positions of the first register and the second register to obtain an activation signal of each multiplication module; and the multiplication module performs multiplication and addition operation on corresponding non-zero element values of the first matrix and the second matrix when the activation signal is 1, and enters a dormant state when the activation signal is 0. According to the invention, hardware power consumption of sparse matrix multiplication can be reduced.

The present disclosure relates to sparse matrix multiplication in hardware. Methods, systems, and apparatuses, including computer-readable storage media, are provided for sparse matrix multiplication. A system for matrix multiplication includes an array of sparse tiles. Each sparse tile can be configured to receive an input submatrix and an input subvector, where the input submatrix has a number of non-zero values that is equal to or less than a predetermined maximum non-zero threshold. The sparse tile can compute, through a plurality of multiplier circuits, one or more products of a vector value multiplied by respective non-zero values of the input submatrix. The sparse tile can generate a tile output vector that is an output of the sparse tile and uses the one or more products, the tile output vector being a product of applying the tile input vector to the tile input matrix.

This application relates to a memory-aware sparse matrix multiplication method suitable for edge embedded platforms. The method includes: first, dividing the sparse matrix into row blocks and storing them in column-major order as column segments; second, dividing the dense matrix and the result matrix into column blocks, with the layout as continuous row segments. Blocking parameters are determined based on the on-chip cache capacity to ensure that each block of the result matrix can reside in the cache. Active column segments of the sparse matrix are traversed, and matching dense matrix row segments are preloaded into registers for element reuse. Non-zero elements of the column segments are multiplied and added to elements of the dense row segments, and the results are accumulated into the corresponding blocks of the result matrix. After completing all column segment operations, the result matrix format is restored, and the final multiplication result is output. This method can reduce computational latency and memory power consumption.

This application discloses an apparatus, method, and program product for accelerating unstructured sparse matrix multiplication computation. The apparatus includes a preprocessing unit, a partial sum generation unit, and a merging unit. The preprocessing unit is configured to convert the unstructured sparse weight matrix to be multiplied into multiple column groups by compressing non-zero elements within the matrix rows while retaining their original column indices, and aggregating non-zero elements from different rows along the column direction. The partial sum generation unit is configured to, for each column group, perform a scalar-vector multiplication operation on each non-zero element within the group with the corresponding row data of the dense matrix to obtain the product result, and combine all product results of each column group as a partial sum data block. The merging unit is configured to accumulate all partial sum data blocks to obtain a merged result. This application significantly improves the execution efficiency of unstructured sparse SpMM in large language models.

The application provides a linear scale electronic structure calculation method and system based on a dyeing superposition state and a terminal, based on the dyeing superposition state, and is used for solving sparse operator or matrix functions with spatial locality or limited correlation length, such as a density matrix, an inverse square root of an overlap matrix, and an operator required by an orthogonal representation transformation, which are involved in a Kohn-Sham self-consistent process under a local basis representation. On the premise of maintaining linear scale complexity, the application breaks through the dependence of a traditional algorithm on sparse matrix-sparse matrix multiplication (SpMSpM), and reconstructs core calculation into regular sparse matrix-dense matrix multiplication (SpMM). Through transformation of a bottom layer calculation paradigm, the regularity of Kohn-Sham electronic structure self-consistent calculation and the parallel adaptation ability of a bottom layer hardware are significantly improved, and inherent bottlenecks, such as a large pre-factor and low hardware execution efficiency, caused by the limitation of a core operator (SpMSpM) in a traditional linear scale algorithm are fundamentally relieved.

The embodiment of the present disclosure provides a model parameter determination method, a model inference system, a decoding device and electronic equipment, which relates to the technical field of model inference, and comprises the following steps: determining a to-be-decoded code stream in a first weight code stream based on a weight parameter, the first weight code stream being obtained by performing prefix encoding processing on an initial weight matrix of a current network layer; and performing prefix decoding processing on the to-be-decoded code stream to obtain target weight data, the target weight data being used for sparse matrix multiplication calculation of the current network layer. The method provided by the present disclosure compresses the data transmission amount through prefix encoding processing, reduces the bandwidth requirement, and simultaneously reduces the bandwidth overhead during model network inference through the cooperative processing of prefix decoding of the to-be-decoded code stream determined by the weight parameter part, thereby improving the model processing efficiency.

The present disclosure relates to sparse matrix multiplication in hardware. Methods, systems, and apparatuses, including computer-readable storage media, are provided for sparse matrix multiplication. A system for matrix multiplication includes an array of sparse tiles. Each sparse tile can be configured to receive an input submatrix and an input subvector, where the input submatrix has a number of non-zero values that is equal to or less than a predetermined maximum non-zero threshold. The sparse tile can compute, through a plurality of multiplier circuits, one or more products of a vector value multiplied by respective non-zero values of the input submatrix. The sparse tile can generate a tile output vector that is an output of the sparse tile and uses the one or more products, the tile output vector being a product of applying the tile input vector to the tile input matrix.

The invention provides a data processing method for sparse matrix multiplication, and the method comprises the following steps: determining a first target accumulator based on the number of non-zero elements of a processing unit of a first sparse matrix, the number of non-zero elements of a corresponding processing unit in a second sparse matrix, and a first preset selection strategy, performing analog matrix multiplication based on the first target accumulator to obtain the number of non-zero elements of a corresponding output unit of the output matrix; determining a second target accumulator based on the number of non-zero elements of a processing unit of the first sparse matrix, the number of non-zero elements of a corresponding output unit of the output matrix and a second preset selection strategy; the target accumulator is a hybrid accumulator at least comprising a merging accumulator; and using a second target accumulator to accumulate and sum the products of the non-zero elements in the processing units of the first sparse matrix and the non-zero elements in the corresponding processing units in the second sparse matrix to obtain the corresponding output units of the output matrix. According to the invention, the operation performance of sparse matrix multiplication is improved.

The application relates to the field of electronic information technology, and particularly discloses a co-occurrence matrix construction method based on multi-GPU coding, equipment and a medium. The method adopts a binary perception sparse compression matrix to perform structured coding on an original sparse vector, divides binary data into fixed-length bit segments and converts the binary data into an integer form, and then performs reconstruction calculation of sparse matrix multiplication on a GPU through bitwise AND and single-bit counting. Compared with the prior art, the method converts originally scattered sparse index operations into batch and dense bit segment operations, so that the SIMD characteristics of the GPU are fully utilized, and the memory access efficiency and the calculation throughput are improved.

This invention discloses an image super-resolution method and system based on semantically aware tokens, belonging to the field of computer vision technology. The method includes: generating semantic confidence and grouping information through content-aware token aggregation; decoupling the basic residual into a texture enhancement and degradation suppression guided map; achieving progressive attention focusing by combining static semantic constraint masks and sparse matrix multiplication mechanisms; introducing diffusion time-step embedding and a collaborative modulator to dynamically inject semantically guided information into a multi-step denoising process; fusing adaptive attention features and diffusion reconstruction features; and finally outputting a high-fidelity, high-resolution image. This invention achieves content-adaptive high-resolution image reconstruction through a semantically grouped guided sparse attention mechanism and semantically decoupled guided diffusion denoising collaborative modulation.

The invention relates to the technical field of electronic information, and particularly discloses a co-occurrence matrix construction method and device based on multi-GPU (Graphics Processing Unit) coding and a medium. According to the method, a binary sensing sparse compression matrix is adopted to carry out structured coding on an original sparse vector, binary data are segmented into fixed-length bit segments and are converted into integer forms, and the integer forms of the fixed-length bit segments are obtained; and reconstruction calculation of sparse matrix multiplication is realized on the GPU through bitwise and single-bit counting. Compared with the prior art, the method provided by the invention has the advantages that the original scattered sparse index operation is converted into batch and dense bit segment operation, so that the SIMD (Single Instruction Multiple Data) characteristic of the GPU is fully utilized, and the memory access efficiency and the calculation throughput rate are improved.

The application discloses a CPU double-path acceleration method for large-scale LLM sparse matrix multiplication, and relates to the technical field of high-performance computing. The method first constructs a fast matrix multiplication algorithm lookup table, generates an optimized algorithm through random walk and performs Hensel promotion to compensate for numerical errors; then performs two-stage division and zero block identification on the weight matrix, classifies it into complete zero blocks, near-zero blocks and dense blocks; then aggregates non-zero elements for the near-zero blocks and merges dense blocks into large blocks using dynamic programming; uses multi-core CPU to concurrently perform sparse path and dense path calculation, respectively calls sparse kernels and matching fast algorithms, and finally fuses the results based on position mapping using vector instructions. The application solves the problem of low utilization of unstructured sparse matrices through double-path shunting strategy and software and hardware co-optimization, significantly improves the calculation throughput and hardware resource utilization while ensuring the model accuracy.

Methods, systems, and apparatus, including computer-readable storage media for hardware-accelerated fine-grained sparse computation. The accelerator provides for improved performance for structured fine-grained sparse AI workloads, for example by accelerating sparse matrix multiplication required to execute or train AI models. Sparse data is compressed to remove zero-valued elements before being streamed into a matrix multiplication unit (MXU) of the accelerator. The accelerator stores a gains matrix, which can be the matrix for multiplying with the received input matrix. The accelerator uses an index array mapping locations of elements in the compressed matrix with locations of elements in the matrix's precompressed form, to generate a multiplier matrix from the gains matrix. Aspects of the disclosure also provide for accelerated gains matrix loading in a hardware accelerator or other type of processor. The accelerator can load the gains matrix more efficiently in a compressed form, and then un-compress the matrix once loaded.

The application relates to the technical field of artificial intelligence, in particular to a neural network inference method, system, device, medium and computer product based on dynamic sparsification, the method comprising the following steps: acquiring an input sequence, generating a query matrix, a key matrix and a value matrix according to the input sequence; calculating the product of the query matrix and the key matrix to generate an initial attention score matrix; performing dynamic N:M structured sparse pruning on the initial attention score matrix to generate a sparse matrix; performing sparse matrix multiplication operation on the sparse matrix and the value matrix in a hardware calculation unit to generate an intermediate result; wherein the hardware calculation unit is configured to skip the multiplication and addition operation on zero value elements according to the N:M sparse mode of the sparse matrix; and performing sparse-aware normalization processing on the intermediate result to generate an attention output. By adopting the scheme, lossless conversion from algorithm sparsification to efficient hardware execution is realized, and substantial end-to-end acceleration and energy efficiency improvement of neural network inference are finally achieved.

The present disclosure discloses a matrix multiplication circuit, a method for performing sparse matrix multiplication by using the matrix multiplication circuit, and related products. The matrix multiplication circuit can be implemented in a combined processing device including a computing device. The combined processing device can also include an interface device and other processing devices. The computing device interacts with the other processing devices to jointly complete a user-specified computing operation. The combined processing device can also include a storage device connected to the computing device and the other processing devices, respectively, for storing data of the computing device and the other processing devices. The scheme of the present disclosure provides a circuit supporting sparse matrix multiplication, which can simplify processing and improve the processing efficiency of the machine.

The invention relates to the technical field of ocean observation and artificial intelligence, in particular to a Transform model accelerator for an ocean drifting buoy, which comprises a multi-head self-attention device for screening high-correlation positions through low-precision pre-calculation and generating a mask matrix, and executing high-precision sparse matrix multiplication based on the mask matrix, in combination with Softmax operation and residual connection, outputting residual feature data; a layer normalization calculation module receives the residual feature data, adopts a dynamic Tanh function to replace traditional layer normalization operation, performs normalization calculation on the residual feature data, and outputs normalized feature data; according to the high-energy-efficiency Transform edge accelerator architecture provided by the invention, the calculation energy efficiency of a Transform model in a resource-limited edge scene is effectively improved, and both the calculation energy efficiency and the calculation precision are taken into account for an ocean numerical calculation scene.

The application belongs to the technical field of CPU data processing, and particularly relates to a sparse matrix multiplication distribution system of a graph convolutional neural network based on FPGA. The application is characterized in that a sparse matrix preprocessing module is arranged in a CPU calculator, and an FPGA module is arranged in a memory; an input sparse matrix in a CSC format is sent to the memory, a CSC format file compresses the sparse matrix into three arrays, i.e., a row index record array, a column index record array and a non-zero value array, the sparse matrix preprocessing module retrieves a data stream from the memory, and performs calculation on the row index record array, the column index record array and the non-zero value array of the sparse matrix, thereby realizing sparse matrix multiplication distribution. The CPU data processing rate of the sparse matrix multiplication distribution system of the graph convolutional neural network based on FPGA is fast.

The invention relates to the technical field of graph calculation and brain-like calculation, in particular to a locally enhanced sparse matrix multiplied sparse vector calculation method and device, and the method comprises the steps: dividing a column into a long column and a short column based on the number of non-zero elements in a sparse matrix column, and splitting the long column into a plurality of continuous long column fragments; reordering according to a target fragment list constructed by the short-column fragments and the long-column fragments, processing fragments in a new fragment list by utilizing each thread block on the GPU, caching the generated update pairs into a shared hash table of the target thread block, aggregating the update pairs meeting a preset condition in the shared hash table, and storing the aggregated update pairs in the shared hash table; and writing the aggregated update pairs back to the global output vector in batches to obtain a locally enhanced sparse matrix multiplied sparse vector calculation result. Therefore, the problems of insufficient data locality mining, increased memory access delay and reduction of the overall performance and resource utilization rate of the SpMSpV on the GPU due to relatively high overhead of a method based on sorting reduction in related technologies are solved.

The application discloses a hardware accelerator for triple sparse matrix multiplication, a device and an application method thereof. The hardware accelerator comprises a high bandwidth memory (HBM), a crossbar network and an on-chip processing unit connected in sequence. The on-chip processing unit comprises a hierarchical cache module, a global controller and a plurality of computing tiles. Each computing tile comprises an RA computing array, a TP computing array and a local controller. The RA computing array and the TP computing array are respectively used for performing front-end operation T=RxA and back-end operation C=TxP in the triple sparse matrix multiplication. The application aims to solve the problem that a conventional general-purpose processor faces a serious performance and energy efficiency bottleneck due to massive off-chip data transfer caused by irregular memory access, uneven computing load and explosive intermediate part and results when processing the triple sparse matrix multiplication, and improve the computing performance and energy efficiency of the triple sparse matrix multiplication.

The application discloses an attention calculation method, system and computer program product. The attention calculation method comprises: obtaining a query matrix Q; obtaining a key matrix K and a value matrix V, wherein the key matrix K and the value matrix V comprise historical data stored in a key-value cache in a sparse tensor format; calculating a product between the query matrix Q and a transpose of the key matrix K to obtain an attention score by executing a sparse matrix multiplication instruction, wherein the sparse matrix multiplication instruction is configured to skip a multiplication operation related to a matrix element with a zero value or below a preset threshold; calculating an attention weight based on the attention score; and performing weighted summation on the value matrix V by using the attention weight to generate an attention output. The above-mentioned attention calculation method improves the overall inference performance of the system by accelerating based on the sparse matrix multiplication instruction and storing the key-value cache data in the sparse tensor format.

The application relates to the technical field of marine observation and artificial intelligence, in particular to a Transformer model accelerator for a marine drifting buoy, the accelerator comprises the following steps: multi-head self-attention is used to screen high-correlation positions through low-precision precalculation and generate a mask matrix; high-precision sparse matrix multiplication operation is performed based on the mask matrix; a Softmax operation is combined with a residual connection; and residual feature data is output; a layer normalization calculation module receives the residual feature data, adopts a dynamic Tanh function to replace a traditional layer normalization operation, performs normalization calculation on the residual feature data, and outputs normalized feature data; the high-energy-efficiency Transformer edge accelerator architecture provided by the application effectively improves the calculation energy efficiency of the Transformer model in a resource-limited edge scene, and the calculation energy efficiency and calculation accuracy are considered for a marine numerical calculation scene.

The invention discloses an attention calculation method, system and computer program product. The attention calculation method comprises the following steps: acquiring a query matrix Q; a key matrix K and a value matrix V are obtained, and the key matrix K and the value matrix V comprise historical data stored in a key value cache in a sparse tensor format; calculating a product between transposes of the query matrix Q and the key matrix K to obtain an attention score by executing a sparse matrix multiplication instruction configured to skip multiplication operations associated with matrix elements of a zero value or below a preset threshold; calculating an attention weight based on the attention score; and performing weighted summation on the value matrix V by utilizing an attention weight to generate attention output. According to the attention calculation method, the overall reasoning performance of the system is improved through acceleration based on a sparse matrix multiplication instruction and storage of key value cache data in a sparse tensor format.

The present application describes an accelerator, a computer system and a method for tensor product calculation for facilitating circuit design. The method can include: dividing a weight tensor into a block matrix; rearranging the block matrix in the weight tensor to obtain a rearranged weight tensor; calculating a bit mask, the bit mask including a bit matrix, bits in the bit matrix corresponding to elements in the rearranged weight tensor; removing zero elements in the rearranged weight tensor and packing non-zero elements in the rearranged weight tensor; generating a compact activation tensor based on the bit mask and an activation tensor; and performing tensor multiplication based on the compact weight tensor and the compact activation tensor. Wherein, the rearrangement step effectively reduces the fan-out between the activation and the corresponding weight, and reducing the fan-out can reduce the wiring length, thereby reducing the energy consumption of signal transmission.