42 results about "Sparse matrix multiplication" patented technology

The invention discloses an image super-resolution method and system based on semantic perception tokens, and relates to the technical field of computer vision, and the method comprises the steps: generating semantic confidence and grouping information through the aggregation of content perception tokens, and decoupling a basic residual error into a texture enhancement and degradation inhibition guidance graph; in combination with a static semantic constraint mask and a sparse matrix multiplication mechanism, progressive focusing of attention is realized; a diffusion time step embedding and cooperative modulator is introduced, semantic guidance information is dynamically injected into a multi-step denoising process, adaptive attention features and diffusion reconstruction features are fused, and finally a high-fidelity and high-resolution image is output. According to the method, content-adaptive high-resolution image reconstruction is realized through collaborative modulation of a sparse attention mechanism guided by semantic grouping and diffusion denoising guided by semantic decoupling.

The invention discloses a hardware accelerator and equipment oriented to triple sparse matrix multiplication and an application method thereof.The hardware accelerator comprises a high-bandwidth memory HBM, a crossbar switch network and an on-chip processing unit which are connected in sequence, and the on-chip processing unit comprises a hierarchical cache module, a global controller and a plurality of computing chips; each calculation piece comprises an RA calculation array, a TP calculation array and a local controller, wherein the RA calculation array and the TP calculation array are respectively used for executing front-end operation T = R * A and rear-end operation C = T * P in triple sparse matrix multiplication. The method aims at solving the problem that when a traditional universal processor processes triple sparse matrix multiplication, due to irregular memory access, uneven calculation load and sharp increase of middle parts and results, huge off-chip data carrying is confronted with serious performance and energy efficiency bottlenecks, and the calculation performance and energy efficiency of triple sparse matrix multiplication are improved.

The invention provides a GPU acceleration system and method based on medium sparseness large language model sparse matrix multiplication, and belongs to the technical field of computers. The system cuts a large model weight matrix through an unstructured cutting module to obtain a sparse weight matrix; a sparse matrix is converted into an MLSF format through a sparse tensor kernel layer, a slot filling layer and a residual element layer in the sparse matrix preprocessing module so as to minimize storage and decoding overhead; a sparse matrix multiplication GPU operation module is used, an MLSF format and a dense matrix are used as input, a fragmentation strategy and asynchronous loading are combined, a pipeline mechanism of data loading, decoding and calculation is constructed, and high overlapping of calculation and memory access operation is achieved. According to the method, dense matrix calculation with sparse matrix multiplication performance exceeding highly optimized performance is realized under medium sparseness, performance improvement of 5.29 times at most is obtained, and technical support is provided for efficient deployment of a large language model.

The invention provides a sparse matrix multiplication acceleration method based on a tensor processor, which relates to the field of high-performance computing, and comprises the following steps: carrying out block processing on a sparse matrix to obtain sparse matrix blocks matched with the register granularity of the tensor processor; performing block processing on the dense matrix to obtain dense matrix blocks; generating a matrix multiplication task set based on the sparse matrix block and the dense matrix block; and distributing the task set to a plurality of threads, controlling the plurality of threads to respectively call the tensor processors to execute matrix multiplication operation, and aggregating calculation results of the threads to obtain an output matrix. Compared with the prior art, the sparse matrix blocks are processed into the sparse matrix blocks matched with the register granularity of the tensor processor, the index coding and decoding overhead, the reconstruction overhead and the communication overhead of the sparse matrix are reduced, and therefore the operation performance of sparse matrix multiplication based on the tensor processor is improved. The system has the same beneficial effects.

The invention discloses a sparse reasoning method, system, equipment and product for large language model reasoning, and relates to the technical field of artificial intelligence. The method comprises the following steps: firstly, acquiring a weight matrix of a large language model, and then carrying out pruning sparsification processing on the weight matrix to obtain a sparse matrix; the storage format of the sparse matrix is converted into a bitmap coding storage format which is suitable for tensor calculation core perception and adopts a multi-level block structure to respectively correspond to different calculation granularities in a GPU / NPU hardware architecture so as to obtain a sparse model, and then the sparse model is deployed to respond to a reasoning request to perform reasoning service; sparse matrix multiplication is completed through a sparse matrix multiplication kernel based on on-chip storage bitmap coding so as to perform reasoning, so that through an innovative sparse matrix storage format and calculation optimization, the storage efficiency and the calculation performance in the reasoning process are remarkably improved, and especially in a low-sparseness scene, the reasoning efficiency is greatly improved. And the performance blank of the existing sparse reasoning framework in the field is filled.

The present disclosure provides a neural network training method and a computing device, which relates to the technical field of artificial intelligence. The scheme includes: splitting a weight matrix into В weight blocks each having the same size and satisfying N:M sparsity constraint; and then by using В sparse mask sets defined for the В weight blocks respectively and В random number indices generated for the В weight blocks respectively, generating a mask matrix for implementing sparse matrix multiplication operation. There is no need to pre-train a dense weight matrix of huge-scale in advance nor to fine-tune a trained model again for the N:M sparsity constraint, which saves a lot of training time and improves training efficiency.

The invention provides a hybrid expert model reasoning optimization method and device and electronic equipment. The method comprises the following steps: receiving a plurality of expert models of a hybrid expert model distributed by a terminal, and receiving operation weights of the plurality of expert models, gating weights of the hybrid expert model and feature vectors of text data sent by the terminal; based on the feature vector and the gating weight of the text data, calculating selection weights of a plurality of expert models, determining a plurality of target expert models from the plurality of expert models, and generating masks of the plurality of target expert models; based on the feature vector of the text data, the masks of the multiple target expert models and the operation weights of the multiple target expert models, sparse matrix multiplication is carried out, and multiple calculation results are obtained; and summarizing the plurality of calculation results to obtain a final calculation result. According to the hybrid expert model reasoning optimization method provided by the invention, the target expert model is determined by adopting the equipment side, and sparse matrix multiplication is performed, so that the data transmission time delay is reduced, and the calculation efficiency is improved.

The application discloses sparse matrix multiplication acceleration hardware, a recommendation system acceleration method and an AI chip. A data loading unit in the hardware establishes a data carrying task for dense data in a sparse matrix multiplication task, carries the dense data to a register, a normalization acceleration unit group sends each left and right matrix block to a multiplication calculation unit group after performing a pre-multiplication activation operation according to the left and right matrix blocks obtained from the register, the multiplication calculation unit group is used for selecting a multiplication calculation unit to perform multiplication calculation according to the data size of the left and right matrix blocks, and providing a calculation result to an addition calculation unit group to perform addition calculation, and the normalization acceleration unit group is also used for performing normalization calculation and / or activation operation according to the addition calculation result when the post-multiplication normalization and / or post-multiplication activation operation is needed, and obtaining a final result. The technical scheme of the embodiment can realize fast sparse matrix multiplication calculation under limited calculation resources.

The application discloses a kind of semi-precision sparse matrix multiplication multicore parallel method and device for vector processor.There are three kinds of multicore parallel modes according to the dimension of matrix and the number of computing core, suitable for a variety of computing scenarios, make full use of the multicore architecture of vector processor.At the same time, it reduces the calculation redundancy under part of matrix dimension specification, improves the parallelism of sparse matrix multiplication calculation, helps to play the computing performance of vector processor.Each multicore parallel mode is to parallel multiple computing cores in the dimension of weight matrix and dense input matrix, and to realize sparse matrix multiplication in different dimensions.The theoretical calculation efficiency of sparse matrix multiplication calculation in each multicore parallel mode is obtained based on the dimension specification of two matrices.Then the multicore parallel mode with the maximum theoretical calculation efficiency is selected for sparse matrix multiplication calculation.This can automatically adapt the optimal mode to perform calculation, with high versatility and improved calculation efficiency.

Apparatuses, systems, and methods to enable matrix multiplication acceleration by modifying an input to apply sparsity through sparse activation filtering. In at least one embodiment, a neural network modifies pixels within an image through sparse activation filtering to enable use of one or more matrix multiplication acceleration units to perform a sparse patch embedding operation.

The invention relates to an N: M sparse matrix multiplication operator optimization method and device, equipment and a storage medium. The method comprises the following steps of: performing task decomposition on N: M sparse matrix multiplication by using a multi-layer partitioning algorithm based on a multi-level scheduling abstraction and storage structure of a GPU (Graphics Processing Unit); carrying out sparsity perception preprocessing on the N: M sparse matrix multiplication after task decomposition, and carrying out optimization processing on input matrixes in different sparsity scenes by using a packaging strategy and a non-packaging strategy respectively; and optimizing the N: M sparse matrix multiplication workflow by adopting prefetching, and calculating the optimized input matrix according to the optimized N: M sparse matrix multiplication workflow to obtain a final calculation result. According to the embodiment of the invention, performance optimization of N: M sparse matrix multiplication is realized, wide input scenes can be adapted, and efficient reasoning on the premise of not influencing the model effect is realized.

The invention provides a sparse matrix multiplication device and method and a control method, and the method comprises the steps: a mask generation module reads a first matrix and a second matrix from a memory, sets the position corresponding to each non-zero element of the first matrix in a first register to be 1, and sets the position corresponding to each zero element to be 0; setting a position corresponding to each non-zero element of a second matrix in a second register to be 1, and setting a position corresponding to each zero element to be 0; the data compression module compresses each non-zero element value of the first matrix and the second matrix and the position index of each non-zero element value; the mask analysis module performs logic AND operation on values of corresponding positions of the first register and the second register to obtain an activation signal of each multiplication module; and the multiplication module performs multiplication and addition operation on corresponding non-zero element values of the first matrix and the second matrix when the activation signal is 1, and enters a dormant state when the activation signal is 0. According to the invention, hardware power consumption of sparse matrix multiplication can be reduced.

The present disclosure relates to sparse matrix multiplication in hardware. Methods, systems, and apparatuses, including computer-readable storage media, are provided for sparse matrix multiplication. A system for matrix multiplication includes an array of sparse tiles. Each sparse tile can be configured to receive an input submatrix and an input subvector, where the input submatrix has a number of non-zero values that is equal to or less than a predetermined maximum non-zero threshold. The sparse tile can compute, through a plurality of multiplier circuits, one or more products of a vector value multiplied by respective non-zero values of the input submatrix. The sparse tile can generate a tile output vector that is an output of the sparse tile and uses the one or more products, the tile output vector being a product of applying the tile input vector to the tile input matrix.

This application relates to a memory-aware sparse matrix multiplication method suitable for edge embedded platforms. The method includes: first, dividing the sparse matrix into row blocks and storing them in column-major order as column segments; second, dividing the dense matrix and the result matrix into column blocks, with the layout as continuous row segments. Blocking parameters are determined based on the on-chip cache capacity to ensure that each block of the result matrix can reside in the cache. Active column segments of the sparse matrix are traversed, and matching dense matrix row segments are preloaded into registers for element reuse. Non-zero elements of the column segments are multiplied and added to elements of the dense row segments, and the results are accumulated into the corresponding blocks of the result matrix. After completing all column segment operations, the result matrix format is restored, and the final multiplication result is output. This method can reduce computational latency and memory power consumption.

This application discloses an apparatus, method, and program product for accelerating unstructured sparse matrix multiplication computation. The apparatus includes a preprocessing unit, a partial sum generation unit, and a merging unit. The preprocessing unit is configured to convert the unstructured sparse weight matrix to be multiplied into multiple column groups by compressing non-zero elements within the matrix rows while retaining their original column indices, and aggregating non-zero elements from different rows along the column direction. The partial sum generation unit is configured to, for each column group, perform a scalar-vector multiplication operation on each non-zero element within the group with the corresponding row data of the dense matrix to obtain the product result, and combine all product results of each column group as a partial sum data block. The merging unit is configured to accumulate all partial sum data blocks to obtain a merged result. This application significantly improves the execution efficiency of unstructured sparse SpMM in large language models.

The application provides a linear scale electronic structure calculation method and system based on a dyeing superposition state and a terminal, based on the dyeing superposition state, and is used for solving sparse operator or matrix functions with spatial locality or limited correlation length, such as a density matrix, an inverse square root of an overlap matrix, and an operator required by an orthogonal representation transformation, which are involved in a Kohn-Sham self-consistent process under a local basis representation. On the premise of maintaining linear scale complexity, the application breaks through the dependence of a traditional algorithm on sparse matrix-sparse matrix multiplication (SpMSpM), and reconstructs core calculation into regular sparse matrix-dense matrix multiplication (SpMM). Through transformation of a bottom layer calculation paradigm, the regularity of Kohn-Sham electronic structure self-consistent calculation and the parallel adaptation ability of a bottom layer hardware are significantly improved, and inherent bottlenecks, such as a large pre-factor and low hardware execution efficiency, caused by the limitation of a core operator (SpMSpM) in a traditional linear scale algorithm are fundamentally relieved.

The embodiment of the present disclosure provides a model parameter determination method, a model inference system, a decoding device and electronic equipment, which relates to the technical field of model inference, and comprises the following steps: determining a to-be-decoded code stream in a first weight code stream based on a weight parameter, the first weight code stream being obtained by performing prefix encoding processing on an initial weight matrix of a current network layer; and performing prefix decoding processing on the to-be-decoded code stream to obtain target weight data, the target weight data being used for sparse matrix multiplication calculation of the current network layer. The method provided by the present disclosure compresses the data transmission amount through prefix encoding processing, reduces the bandwidth requirement, and simultaneously reduces the bandwidth overhead during model network inference through the cooperative processing of prefix decoding of the to-be-decoded code stream determined by the weight parameter part, thereby improving the model processing efficiency.

The present disclosure relates to sparse matrix multiplication in hardware. Methods, systems, and apparatuses, including computer-readable storage media, are provided for sparse matrix multiplication. A system for matrix multiplication includes an array of sparse tiles. Each sparse tile can be configured to receive an input submatrix and an input subvector, where the input submatrix has a number of non-zero values that is equal to or less than a predetermined maximum non-zero threshold. The sparse tile can compute, through a plurality of multiplier circuits, one or more products of a vector value multiplied by respective non-zero values of the input submatrix. The sparse tile can generate a tile output vector that is an output of the sparse tile and uses the one or more products, the tile output vector being a product of applying the tile input vector to the tile input matrix.

The invention provides a data processing method for sparse matrix multiplication, and the method comprises the following steps: determining a first target accumulator based on the number of non-zero elements of a processing unit of a first sparse matrix, the number of non-zero elements of a corresponding processing unit in a second sparse matrix, and a first preset selection strategy, performing analog matrix multiplication based on the first target accumulator to obtain the number of non-zero elements of a corresponding output unit of the output matrix; determining a second target accumulator based on the number of non-zero elements of a processing unit of the first sparse matrix, the number of non-zero elements of a corresponding output unit of the output matrix and a second preset selection strategy; the target accumulator is a hybrid accumulator at least comprising a merging accumulator; and using a second target accumulator to accumulate and sum the products of the non-zero elements in the processing units of the first sparse matrix and the non-zero elements in the corresponding processing units in the second sparse matrix to obtain the corresponding output units of the output matrix. According to the invention, the operation performance of sparse matrix multiplication is improved.

The application discloses a method for accelerating arbitrary precision sparse matrix multiplication and addition operation based on a tensor core, which is in combination with deep learning and the program characteristics of parallel processing of sparse matrices, and is in combination of an arbitrary precision sparse matrix multiplication and addition operation operator and a parallel computing acceleration device, decomposes a general matrix into a sparse matrix of a set format, blocks the sparse matrix and specifies a corresponding parallel computing hardware unit for the sparse matrix, uses shared memory to reduce memory conflict and access delay, calls a high-level language API function library to execute the loading, calculation and storage process of the calculation data, and finally reduces the output data of the function, so that the final result of the arbitrary precision sparse matrix multiplication and addition operation on the parallel computing acceleration device is obtained. The application can fully utilize the parallel computing resources such as the tensor core on the A100, effectively reduce the program calculation amount and communication overhead, and greatly improve the program running performance.

The application relates to the field of electronic information technology, and particularly discloses a co-occurrence matrix construction method based on multi-GPU coding, equipment and a medium. The method adopts a binary perception sparse compression matrix to perform structured coding on an original sparse vector, divides binary data into fixed-length bit segments and converts the binary data into an integer form, and then performs reconstruction calculation of sparse matrix multiplication on a GPU through bitwise AND and single-bit counting. Compared with the prior art, the method converts originally scattered sparse index operations into batch and dense bit segment operations, so that the SIMD characteristics of the GPU are fully utilized, and the memory access efficiency and the calculation throughput are improved.

This invention discloses an image super-resolution method and system based on semantically aware tokens, belonging to the field of computer vision technology. The method includes: generating semantic confidence and grouping information through content-aware token aggregation; decoupling the basic residual into a texture enhancement and degradation suppression guided map; achieving progressive attention focusing by combining static semantic constraint masks and sparse matrix multiplication mechanisms; introducing diffusion time-step embedding and a collaborative modulator to dynamically inject semantically guided information into a multi-step denoising process; fusing adaptive attention features and diffusion reconstruction features; and finally outputting a high-fidelity, high-resolution image. This invention achieves content-adaptive high-resolution image reconstruction through a semantically grouped guided sparse attention mechanism and semantically decoupled guided diffusion denoising collaborative modulation.

The invention relates to the technical field of electronic information, and particularly discloses a co-occurrence matrix construction method and device based on multi-GPU (Graphics Processing Unit) coding and a medium. According to the method, a binary sensing sparse compression matrix is adopted to carry out structured coding on an original sparse vector, binary data are segmented into fixed-length bit segments and are converted into integer forms, and the integer forms of the fixed-length bit segments are obtained; and reconstruction calculation of sparse matrix multiplication is realized on the GPU through bitwise and single-bit counting. Compared with the prior art, the method provided by the invention has the advantages that the original scattered sparse index operation is converted into batch and dense bit segment operation, so that the SIMD (Single Instruction Multiple Data) characteristic of the GPU is fully utilized, and the memory access efficiency and the calculation throughput rate are improved.

The present invention provides a data processing acceleration method and device for a sparse matrix multiplication operator, comprising: obtaining a sparse matrix multiplication operator to be processed during data processing, compressing the sparse matrix, and storing the compressed sparse matrix in a form including a non-zero element array, a row pointer array, a global column index array, and a local column index array. The global column index records the column index of the non-zero elements in a set of consecutive preset number of sparse rows divided by the sparse matrix, and the global column index is not repeated in each column index array. The local column index records the position index of each non-zero element in the global column index array. The global column index is used to access the corresponding row data in the dense matrix and read it into a temporary buffer area. The local column index is used to access the corresponding row data in the dense matrix in the temporary buffer area. The product of the non-zero element and the corresponding row data in the dense matrix is ​​calculated and accumulated, and the result matrix is ​​output. The present invention can improve data processing speed and reduce memory access overhead.

Apparatuses, systems, and methods to enable matrix multiplication acceleration by modifying an input to apply sparsity through sparse activation filtering. In at least one embodiment, a neural network modifies pixels within an image through sparse activation filtering to enable use of one or more matrix multiplication acceleration units to perform a sparse patch embedding operation.

The application discloses a CPU double-path acceleration method for large-scale LLM sparse matrix multiplication, and relates to the technical field of high-performance computing. The method first constructs a fast matrix multiplication algorithm lookup table, generates an optimized algorithm through random walk and performs Hensel promotion to compensate for numerical errors; then performs two-stage division and zero block identification on the weight matrix, classifies it into complete zero blocks, near-zero blocks and dense blocks; then aggregates non-zero elements for the near-zero blocks and merges dense blocks into large blocks using dynamic programming; uses multi-core CPU to concurrently perform sparse path and dense path calculation, respectively calls sparse kernels and matching fast algorithms, and finally fuses the results based on position mapping using vector instructions. The application solves the problem of low utilization of unstructured sparse matrices through double-path shunting strategy and software and hardware co-optimization, significantly improves the calculation throughput and hardware resource utilization while ensuring the model accuracy.

Methods, systems, and apparatus, including computer-readable storage media for hardware-accelerated fine-grained sparse computation. The accelerator provides for improved performance for structured fine-grained sparse AI workloads, for example by accelerating sparse matrix multiplication required to execute or train AI models. Sparse data is compressed to remove zero-valued elements before being streamed into a matrix multiplication unit (MXU) of the accelerator. The accelerator stores a gains matrix, which can be the matrix for multiplying with the received input matrix. The accelerator uses an index array mapping locations of elements in the compressed matrix with locations of elements in the matrix's precompressed form, to generate a multiplier matrix from the gains matrix. Aspects of the disclosure also provide for accelerated gains matrix loading in a hardware accelerator or other type of processor. The accelerator can load the gains matrix more efficiently in a compressed form, and then un-compress the matrix once loaded.

The application relates to the technical field of artificial intelligence, in particular to a neural network inference method, system, device, medium and computer product based on dynamic sparsification, the method comprising the following steps: acquiring an input sequence, generating a query matrix, a key matrix and a value matrix according to the input sequence; calculating the product of the query matrix and the key matrix to generate an initial attention score matrix; performing dynamic N:M structured sparse pruning on the initial attention score matrix to generate a sparse matrix; performing sparse matrix multiplication operation on the sparse matrix and the value matrix in a hardware calculation unit to generate an intermediate result; wherein the hardware calculation unit is configured to skip the multiplication and addition operation on zero value elements according to the N:M sparse mode of the sparse matrix; and performing sparse-aware normalization processing on the intermediate result to generate an attention output. By adopting the scheme, lossless conversion from algorithm sparsification to efficient hardware execution is realized, and substantial end-to-end acceleration and energy efficiency improvement of neural network inference are finally achieved.

The embodiment of the invention provides a model parameter determination method, a model reasoning system, a decoding device and electronic equipment, and relates to the technical field of model reasoning, and the method comprises the steps: determining a to-be-decoded code stream in a first weight code stream based on a weight parameter, and the first weight code stream is obtained by carrying out prefix coding processing on an initial weight matrix of a current network layer; and performing prefix decoding processing on the to-be-decoded code stream to obtain target weight data, the target weight data being used for sparse matrix multiplication calculation of the current network layer. According to the method provided by the invention, the data transmission amount is compressed through prefix coding processing, the bandwidth requirement is reduced, and meanwhile, the to-be-decoded code stream of the weight parameter determination part is subjected to prefix decoding cooperative processing, so that the bandwidth overhead during model network reasoning is reduced, and the model processing efficiency is improved.

The present disclosure discloses a matrix multiplication circuit, a method for performing sparse matrix multiplication by using the matrix multiplication circuit, and related products. The matrix multiplication circuit can be implemented in a combined processing device including a computing device. The combined processing device can also include an interface device and other processing devices. The computing device interacts with the other processing devices to jointly complete a user-specified computing operation. The combined processing device can also include a storage device connected to the computing device and the other processing devices, respectively, for storing data of the computing device and the other processing devices. The scheme of the present disclosure provides a circuit supporting sparse matrix multiplication, which can simplify processing and improve the processing efficiency of the machine.