17 results about "General matrix" patented technology

The embodiment of the invention provides a kernel selection method and device during general matrix multiplication operation, equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring general matrix multiplication problem size data; preprocessing the general matrix multiplication problem size data to obtain general matrix multiplication problem size features; based on a pre-trained problem size encoder, mapping the general matrix multiplication problem size feature into a problem size embedded vector of a preset dimension; taking the problem size embedded vector as a query vector, and performing nearest neighbor vector search in a pre-configured vector database to obtain a kernel configuration feature with the highest similarity; and outputting the kernel configuration feature as a selection result. The method is used for improving the kernel selection efficiency and precision during the operation of the general matrix multiplication.

The invention relates to the field of data lookup, and discloses a locality perception lookup table lookup method, which comprises the following steps of: S1, reordering between rows: clustering and reordering the rows of an input vector and a weight matrix according to the same quantized value to enable elements with the same quantized value to be continuously distributed; for a common matrix GeMV scene, cache locality is improved through rearrangement and subsequent blocking operation, and repeated loading of a lookup table LUT is reduced. And the data locality of the GeMV scene is remarkably improved through inter-line reordering and a virtual mapping strategy. On the basis of reordering of input vector element values, elements with the same value are aggregated, so that the LUT line loading times are reduced to 256 times at most from being in direct proportion to the input dimension d, and frequent replacement of the LUT lines in the WRAM is avoided; meanwhile, a weight matrix is divided into BR * BC sub-matrixes, the characteristic that vertical adjacent rows share an accumulator is utilized, multi-row sub-matrix calculation is loaded at a time, and virtual rearrangement maintains the row matching relation between an input vector and the weight matrix through index mapping.

This invention provides a method, apparatus, graphics processor, and storage medium for matrix multiplication operators, relating to the field of artificial intelligence technology. The method includes: reading two operation matrices from high-bandwidth memory; determining a first target input matrix in the general matrix buffer (GMB) based on the two operation matrices, and identifying the other operation matrix as a second target input matrix; the first target input matrix is ​​arranged in a transposed manner; and writing the first and second target input matrices into a tensor core for matrix multiplication to obtain the target output matrix from the tensor core. This invention, by identifying the data arrangement of the two operation matrices, prioritizes inputting the first target input matrix (with a transposed arrangement) into the GMB, effectively avoiding hardware performance limitations associated with non-transposed readings, ensuring the tensor core maintains full-bandwidth data input, and significantly improving the computational efficiency of the matrix multiplication operator.

A computing method and apparatus for general matrix multiplication (GEMM) are provided. The method comprises: in response to determining that a host needs to obtain a first calculation result of multiplication of first weight data and an input vector, performing a multiplication operation of second weight data and the input vector to obtain a second calculation result, wherein the first weight data satisfies a first arrangement rule for the host, and the second weight data is obtained by arranging the first weight data based on a second arrangement rule for a PIM apparatus different from the first arrangement rule; and obtaining the first calculation result based on the second calculation result.

Embodiments of the present disclosure provide a method, apparatus, computing device and storage medium for calculating matrix-vector multiplication, where the matrix is an M*K matrix and the vector is a K*1 vector, M and K are positive integers. The method comprises: converting the M*K matrix into an M*(K / N)*N first tensor, where N is a positive integer, N>1, and K is a multiple of N; converting the K*1 vector into an N*(K / N) second tensor; using a tensor computing core to batch-process the general matrix multiplication of the M*(K / N)*N first tensor and the N*(K / N) second tensor to obtain M result matrices; and adding the elements on the diagonal of each of the M result matrices to obtain each element on the M*1 result vector of the M*K matrix multiplied by the K*1 vector. The above scheme uses a tensor computing core to implement high-throughput, low-latency high-speed matrix-vector multiplication (MMV) operation.

A system and method for performing multiplication for a neural network, e.g. for data of one or more layers in a neural network, may include loading a portion of a compressed version of a sparse input matrix into a cache memory; uncompressing a subset of the data in the portion of the compressed version of the sparse input matrix; and multiplying a sparse kernel matrix by the subset of the data using a set of instructions which are themselves created based on the sparse kernel matrix.

A code optimization method and an optimization device are provided. The method is applied to the optimization device, and includes: determining M first general matrix vector multiplication (GEMV) calculations in a code to be optimized based on an identification rule; determining N second GEMV calculations in the M first GEMV calculations based on a merging rule; inserting general matrix multiplication (GEMM) calculations corresponding to the N second GEMV calculations into an insertion position of the GEMM calculations in the code, and deleting the N second GEMV calculations. This scheme can realize automatic identification and merging of GEMV calculations, and solve the problem of waste of matrix unit computing power.

A general matrix-matrix (GEMM) accelerator core includes first and second buffers, a control logic circuit, and a first processing element (PE). The first buffer receives a elements of a first matrix A of activation values. The second buffer receives b elements of a second matrix B of weight values. The control logic circuit replaces a zero-valued a element in a first column of the first buffer with a nonzero-valued a element that is within a maximum borrowing distance of a location of the zero-valued a element in the first column of the first buffer. The PE receives a elements from the first column of the first buffer including the nonzero-valued element a selected to replace the zero-valued a element and receives b elements from locations in the second buffer that correspond to locations in the first buffer from where the a elements have been received by the PE.

Disclosed in the embodiments of the present application are a data processing method and apparatus, a computer device and a storage medium. A neural network processor comprises a global memory, a transfer buffer and a plurality of computing units. The method comprises: partitioning a matrix to be updated, so as to obtain a plurality of sub-matrices to be updated, and equally allocating the plurality of sub-matrices to be updated to the computing units; determining, in a first matrix, a first sub-matrix corresponding to each sub-matrix to be updated, and transferring the first sub-matrix from the global memory to a preset buffer of the computing unit corresponding to the sub-matrix to be updated; determining, in a second matrix, a second sub-matrix corresponding to each sub-matrix to be updated, and transferring the second sub-matrix from the global memory to the transfer buffer; acquiring target first sub-matrices from the preset buffers, and acquiring target second sub-matrices from the transfer buffer; and performing, by means of each computing unit, general matrix multiplication on each sub-matrix to be updated, the target first sub-matrix and the target second sub-matrix, so as to obtain an updated sub-matrix.

The application belongs to the technical field of circuits and systems, and discloses an end-side-oriented self-attention reasoning method, an engine and a multi-head fast decoding circuit. The self-attention reasoning method realizes self-attention reasoning calculation by sequentially traversing the cache key-value pairs through a token-by-token pipeline, dynamically maintaining the maximum attention score statistics, the softmax normalization factor cumulative value and the attention result cumulative vector. In the process, each key-value pair is only accessed and processed once, without the need to store the intermediate attention scores or to perform block-shaped softmax or secondary traversal. Through the above method and in combination with the multi-head fast decoding circuit designed in cooperation with the self-attention reasoning method, the storage consumption of self-attention calculation can be reduced without additional hardware parallelism, high-precision Attention calculation and low-precision general matrix operation can be uniformly executed, and multi-head fast decoding can be supported, so that the end device can also efficiently realize LLM decoding.

A kind of tensor computing unit convolution operator optimization implementation method faces, convolution operator is represented by the DSL of deep learning compiler, implicit general matrix multiplication is obtained by coordinate transformation to convolution calculation;Then after scheduling optimization is carried out to the convolution operator, the optimal search parameter is obtained by searching and the CUDA C code is generated through the back end of deep learning compiler, then the generated CUDA C code is integrated into neural network, the inference speed of convolutional neural network on NVIDIAGPU platform is improved.The application can improve the performance of automatic code generation of convolution operator in half-precision calculation, and provide guarantee for the performance of automatic code generation of fusion operator in neural network inference calculation.

The embodiment of the invention discloses an indexing method and device for a large language model, computer equipment and a storage medium, and the method comprises the steps: generating an expert mask table based on a comparison result of a generalized coding sequence outputted by a router and a preset numerical value; executing a first type of prefix sum operation on the expert mask table to generate a first auxiliary index table, and integrating tail row data of the first auxiliary index table under a plurality of batch processing scenes to generate a second auxiliary index table; executing a second type of prefix sum operation on the tail row of the second auxiliary index table to generate a third auxiliary index table; according to the first auxiliary index table, the second auxiliary index table and the third auxiliary index table, rearranging input feature data to obtain rearranged feature data; performing expert-level batch general matrix multiplication on the rearranged feature data; and based on the gating weight output by the router, performing weighted fusion operation on results of the batch general matrix multiplication operation to obtain a final output feature of a big language model MOE architecture reasoning stage.

This disclosure relates to accelerating the processing of distributed matrix multiplication operations. The method proposed herein can efficiently perform operations such as General Matrix Multiplication (GEMM). The data required for such operations can be prefetched as needed, for example, immediately before a specific computation is performed on a pair of data blocks. Prefetching for subsequent computations can be performed during the current computation. After the current computation, the results can be accumulated in subsequent computations; for example, the accumulation operation can be offloaded using a data processing unit, thus freeing up one or more processing cores for other computations. Such operations can be performed in parallel until each block of the resulting matrix has been processed. In at least one embodiment, a single computational task can also be partitioned among multiple worker processes (e.g., processing units).

Architectures and instruction sets for non-general matrix multiplication operations are described. An example programmable processor, which implements these architectures and instruction sets, includes multiple processing engines, multiple buffers (with at least one buffer storing a multidimensional array), and multiple iterator tables stored in one of the multiple buffers. The multiple processing engines and multiple buffers are tightly coupled within an area of the programmable processor to enable high-bandwidth data communications between components of the programmable processor. Furthermore, the multiple iterator tables include an iterator table for each of the multiple buffers, and an iterator table enables read or write access to a respective buffer based on using multiple offset-stride tuples stored in the iterator table to derive addresses to access the multidimensional array stored in the respective buffer. Memory elements used by the multiple processing engines to perform operations using the multiple iterator tables consist of the multiple buffers.

A general matrix-matrix (GEMM) accelerator core includes first and second buffers, and a processing element (PE). The first buffer receives a elements of a matrix A of activation values. The second buffer receives b elements of a matrix B of weight values. The matrix B is preprocessed with a nonzero-valued b element replacing a zero-valued b element in a first row of the second buffer based on the zero-valued b element being in the first row of the second buffer. Metadata is generated that includes movement information of the nonzero-valued b element to replace the zero-valued b element. The PE receives b elements from a first row of the second buffer and a elements from the first buffer from locations in the first buffer that correspond to locations in the second buffer from where the b elements have been received by the PE as indicated by the metadata.

The invention discloses an ill-conditioned matrix SVD decomposition preprocessing method and device based on deep learning, and a medium, and belongs to the technical field of numerical algebra and deep learning. The method comprises the following steps: constructing an iterative deep neural network learning framework, extracting matrix features by using a convolutional layer, and constructing an orthogonal matrix through a House holder reflection decomposition method; training the network by using a mixed loss function, forcing the network to output an approximate diagonal matrix by punishing off-diagonal elements, and performing multiple rounds of iterative optimization by taking a learning result as the input of a new round of training; a precondition is constructed based on a matrix obtained through training, an original ill-conditioned linear equation set is preprocessed, and a preconditioned equation set is solved through an iterative algorithm. The method can be adapted to an ill-conditioned general matrix without a special structure, effectively solves the problems of insufficient approximation precision and poor generalization ability of a traditional method, and remarkably improves the numerical stability, convergence efficiency and calculation precision of high-dimensional ill-conditioned matrix solution.

The application belongs to the technical field of high-performance computing, and aims at solving the problems of limited computing power and access efficiency, and provides a double-precision general matrix multiplication optimization method and system for a memory architecture, wherein a main core receives an operation request, determines a unified blocking rule by using a multi-constraint optimization model, and decomposes a matrix into a plurality of sub-matrix blocks according to a core array topology; a slave core receives the sub-matrix blocks issued by the main core, divides the sub-matrix blocks into sub-matrix small blocks based on the unified blocking rule, loads the sub-matrix small blocks into local data memory independent buffers based on a DMA double-buffer protocol, divides the sub-matrix small blocks in the buffer into SIMD vectors according to the characteristics of a SIMD unit, and alternately switches vectorization calculation and cache operation through different independent buffers according to an iteration period; after all the slave cores complete the calculation, the main core collects the results written back to the main memory by the slave cores to obtain the final operation result, and the application realizes a double breakthrough of computing power and memory access efficiency.