56 results about "General matrix" patented technology

The invention belongs to the field of parallel computing architecture optimization, and relates to a matrix multiplication acceleration method and system based on dynamic computing resource mapping, and the method comprises the steps: constructing a dynamic topology model driven by tensor dimension features, and generating a thread block distribution mode according to matrix parameters and GPU hardware information; constructing a multi-dimensional resource scheduling strategy library, dynamically selecting an optimal thread block distribution strategy from the multi-dimensional resource scheduling strategy library, and generating a binding relationship between the thread blocks and the data blocks; calculating collaborative access logic of thread blocks and storage hierarchies based on block parameters and dynamic mapping function optimization; distributed calculation is carried out, calculation and data transmission are parallelized through pipelining and a double-buffering mechanism, and result aggregation across calculation units is completed synchronously through atomic operation and a barrier. According to the method, discontinuous memory access conflicts can be effectively reduced, the execution efficiency of the calculation instruction and the utilization rate of the cache space are improved, the parallel calculation process of accelerating and optimizing the general matrix multiplication is realized, and the data processing efficiency is improved.

The invention discloses a tensor core-based method for accelerating arbitrary precision sparse matrix multiplication and addition operation, which comprises the following steps of: aiming at an architecture of a tensor core on A100 and a programming model thereof, combining program characteristics of deep learning parallel processing sparse matrix, combining an arbitrary precision sparse matrix multiplication and addition operation operator with parallel computing acceleration equipment, and performing parallel computing acceleration on the sparse matrix. The method comprises the following steps: decomposing a general matrix into a sparse matrix in a set format, partitioning the sparse matrix and assigning a corresponding parallel computing hardware unit for the sparse matrix, reducing memory conflicts and access delay by using a shared memory, calling a high-level language API function library to execute loading, computing and storing processes of a computing data volume, and finally performing specification on output data of a function. Therefore, the final result of the random-precision sparse matrix multiplication and addition operation on the parallel computing acceleration equipment is obtained. According to the method, parallel computing resources such as the tensor core on the A100 can be fully utilized, the program calculation amount and the communication overhead are effectively reduced, and the program operation performance is greatly improved.

The invention discloses a task scheduling method and device, equipment and a medium, and relates to the field of task processing.The task scheduling method comprises the steps that when a universal graphics processor writes a current calculation task into a current task queue, a normalized task priority and task execution estimated duration corresponding to the current calculation task are determined, and the task execution estimated duration is determined; the queue utilization rate of the current task queue is determined; determining a current load rate based on the task priority, the task execution estimated duration and the queue usage rate, and determining a target computing core from computing cores of the universal graphics processor according to the current load rate; each computing core comprises a tensor computing core and a computing core based on a unified computing device architecture; according to the method, the current computing task is scheduled to the target computing core, and the target computing core is used for executing the general matrix multiplication operation on the target matrix contained in the current computing task, so that each computing core in the general graphics processor is fully utilized, and hardware resources in the general graphics processor are utilized in a balanced and efficient manner.

The invention discloses an information processing method and device, electronic equipment and a storage medium, and is applied to the field of information processing. The information processing method comprises the steps that a first utilization rate is determined in combination with a data flow between a memory and a shared memory when a processor executes a general matrix multiplication operator, and the first utilization rate is used for indicating performance evaluation of the general matrix multiplication operator at a processor level; determining a second utilization rate in combination with a data stream between a shared memory and a register file in a single computing unit when the processor executes the general matrix multiplication operator, the second utilization rate being used for indicating performance evaluation of the general matrix multiplication operator at the computing unit level; and based on the smaller one of the first utilization rate and the second utilization rate, determining the theoretical performance of the processor when executing the universal matrix multiplication operator. At present, only chip-level performance evaluation is considered in performance evaluation of a GEMM operator, and double-level-dimension theoretical performance evaluation is provided, so that the evaluation result is more accurate.

The invention belongs to the related technical field of high-performance computing, and provides a memory architecture-oriented dual-precision general matrix multiplication optimization method and system in order to solve the problems of limited computing power and access efficiency and the like in the prior art. Decomposing the matrix into a plurality of sub-matrix blocks according to the slave core array topology; the slave core receives the sub-matrix blocks issued by the master core, divides the sub-matrix blocks into small sub-matrix blocks based on a uniform blocking rule, loads the small sub-matrix blocks to an independent buffer area of a local data memory based on a DMA double-buffer protocol, divides the small sub-matrix blocks in the buffer area into SIMD vectors according to the SIMD unit characteristics of the slave core, and sends the SIMD vectors to the slave core; vectorization calculation and caching operation are alternately switched according to an iteration period through different independent buffer areas; and after all the slave cores finish calculation, the master core collects results written back to the master memory by the slave cores to obtain a final operation result, and double breakthrough of calculation power and memory access efficiency is realized.

Provided in the present invention are a method and system for optimizing general matrix multiplication, and a device and a medium. The method comprises the following steps: on the basis of the influence of the size of a dimension slice on the degree of parallelism and a memory access ratio, obtaining a dimension slicing strategy; and on the basis of the dimension slicing strategy, slicing at least one target dimension in general matrix multiplication to obtain optimized dimension slices, so as to complete the optimization of matrix multiplication. In the present application, the degree of parallelism and a memory access ratio are both used as considerations for a dimension slicing strategy, and the size of a target dimension and the number of slices can be taken into comprehensive consideration, so that parallel computing units can be filled, thereby fully utilizing the degree of computing parallelism of hardware, and also increasing a computing-to-memory access ratio.

The invention provides a data processing method, and relates to the technical field of artificial intelligence, in particular to the technical fields of chips, general matrix multiplication, large models and the like. According to the specific implementation scheme, in the current simulated annealing period, candidate data block information of a calculation unit is determined according to evaluated data block information of the calculation unit; determining a candidate hardware performance evaluation result according to the candidate data block information; according to the candidate hardware performance evaluation result, the current hardware performance evaluation result and the current temperature information, to-be-evaluated data block information of the current simulated annealing period in a post-simulated annealing period is determined, and the data block information comprises first data scale information and second data scale information; the first data scale information is used for indicating the scale of to-be-processed data of the calculation unit, and the second data scale information is used for indicating the scale of to-be-processed data blocks of the calculation unit. The invention further provides a data processing device, equipment and a medium.

The invention relates to the technical field of large language model deployment, and discloses a large language model weight inverse quantization reasoning device and method.The method comprises the steps that low-precision weight data is transmitted to a high-bandwidth storage from a host and then transmitted to an on-chip storage through the high-bandwidth storage; data conversion from a low-precision format to a high-precision format is completed in the on-chip memory, the data is multiplied by an inverse quantization factor to obtain recovered high-precision weight data, and the functional unit is responsible for executing general matrix multiplication of input data and the high-precision weight data after inverse quantization. And pipeline parallel execution of the inverse quantization operation and the general matrix multiplication operation is realized through a double-buffering technology. According to the invention, on the basis of a dual-path inverse quantization architecture of the vector processing unit and a dual-buffer mechanism in the on-chip memory, the problem of hardware adaptation of low-precision calculation is solved, and efficient execution of a low-precision conversion algorithm is realized under the condition of limited hardware resources.

The embodiment of the invention provides a method and device for calculating a matrix multiplied by a vector, computing equipment and a storage medium, the matrix is a matrix of M * K, the vector is a vector of K * 1, and M and K are positive integers. The method comprises the following steps: converting an M * K matrix into a first tensor of M * (K / N) * N, N being a positive integer, Ngt; 1 and K are multiples of N; converting the vector of K * 1 into a second tensor of N * (K / N); using a tensor calculation kernel to carry out general matrix multiplication calculation of a first tensor of M * (K / N) * N and a second tensor of N * (K / N) in batches to obtain M result matrices; elements on diagonals of each of the M result matrices are added to result in a matrix of M * K multiplied by each element on a result vector of M * 1 of the vector of K * 1. According to the scheme, high-throughput and low-delay high-speed matrix vector multiplication (MMV) operation is realized by utilizing the tensor calculation kernel.

The invention discloses a model reasoning acceleration method and device, electronic equipment and a nonvolatile storage medium. The method comprises the steps that in the process that a natural language processing model carries out general matrix calculation, a first matrix and a second matrix are divided into a plurality of blocks respectively, general matrix calculation is executed through an image processing unit, and the first matrix and the second matrix are matrixes needing general matrix calculation; loading the blocks from a global memory of the image processing unit to a shared memory of the image processing unit, and performing matrix operation by adopting threads corresponding to the blocks in the shared memory; and writing a result obtained by calculation of each thread back to a corresponding position in the global memory to obtain a matrix calculation result corresponding to matrix calculation of the first matrix and the second matrix. The technical problem of low model reasoning calculation efficiency caused by high memory access delay and low data reuse rate of a global memory of an image processing unit during reasoning of a large model in related technologies is solved.

The invention discloses an iterative computation matrix inversion method and system based on Cholesky decomposition, and relates to the technical field of DSP system optimization, and the method comprises the steps: obtaining a target source matrix; based on Cholesky decomposition, performing first iteration processing on the target source matrix to generate an upper triangular matrix; performing second iteration processing on the upper triangular matrix to generate an inverse matrix of the upper triangular matrix; performing conjugate transpose processing on the inverse matrix of the upper triangular matrix to generate a lower triangular matrix; wherein the lower triangular matrix is stored in the form of array storage; converting the storage mode of the inverse matrix of the upper triangular matrix into a sequential storage mode; and performing matrix multiplication processing on the inverse matrix of the upper triangular matrix and the lower triangular matrix to generate an inverse matrix of the target source matrix. According to the method, accumulative summation is replaced by iteration, complex multiplication and addition optimization calculation is adopted, multi-parallelism operation and parallelization processing zero padding operation are supported, a general matrix multiplication module can be adapted, and calculation time and area overhead are reduced.

Examples described herein relate to offload circuitry comprising one or more compute engines that are configurable to perform a workload offloaded from a process executed by a processor based on a descriptor particular to the workload. In some examples, the offload circuitry is configurable to perform the workload, among multiple different workloads. In some examples, the multiple different workloads include one or more of: data transformation (DT) for data format conversion, Locality Sensitive Hashing (LSH) for neural network (NN), similarity search, sparse general matrix-matrix multiplication (SpGEMM) acceleration of hash based sparse matrix multiplication, data encode, data decode, or embedding lookup.

The invention provides a code optimization method and optimization equipment. The method is applied to optimization equipment, and comprises the following steps: determining M first general matrix vector multiplication (GEMV) calculations in a code to be optimized based on an identification rule; determining N second GEMV calculations in the M first GEMV calculations based on a merging rule; and inserting general matrix multiplication (GEMM) calculations corresponding to the N second GEMV calculations into insertion positions of the GEMM calculations in the code, and deleting the N second GEMV calculations. According to the scheme, automatic identification and GEMV calculation merging can be realized, and the problem of matrix unit calculation power waste is solved.

The invention discloses a communication computing parallel optimization method, a multiprocessor system, a medium and a program product, and the method comprises the steps: configuring a first computing core and a second computing core on a single task flow for a general matrix multiplication subtask allocated to a single processor; wherein the first calculation core is used for executing a general matrix multiplication subtask, and the second calculation core is used for executing a set communication task; the set communication task comprises a full accumulation operator or a protocol dispersion operator; then, starting scheduling is conducted on the first calculation core and the second calculation core according to a preset dependency mechanism, so that the general matrix multiplication subtask and the set communication task are executed asynchronously in an overlapped mode; according to the method, the problem of poor reusability of the original Kernel caused by intrusive modification of the GEMM or re-implementation of the Kernel can be effectively avoided through parallel optimization of GEMM calculation and ensemble communication operation, and the performance overhead of the processor is reduced.

The embodiment of the invention provides a kernel selection method and device during general matrix multiplication operation, equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring general matrix multiplication problem size data; preprocessing the general matrix multiplication problem size data to obtain general matrix multiplication problem size features; based on a pre-trained problem size encoder, mapping the general matrix multiplication problem size feature into a problem size embedded vector of a preset dimension; taking the problem size embedded vector as a query vector, and performing nearest neighbor vector search in a pre-configured vector database to obtain a kernel configuration feature with the highest similarity; and outputting the kernel configuration feature as a selection result. The method is used for improving the kernel selection efficiency and precision during the operation of the general matrix multiplication.

The invention relates to the field of data lookup, and discloses a locality perception lookup table lookup method, which comprises the following steps of: S1, reordering between rows: clustering and reordering the rows of an input vector and a weight matrix according to the same quantized value to enable elements with the same quantized value to be continuously distributed; for a common matrix GeMV scene, cache locality is improved through rearrangement and subsequent blocking operation, and repeated loading of a lookup table LUT is reduced. And the data locality of the GeMV scene is remarkably improved through inter-line reordering and a virtual mapping strategy. On the basis of reordering of input vector element values, elements with the same value are aggregated, so that the LUT line loading times are reduced to 256 times at most from being in direct proportion to the input dimension d, and frequent replacement of the LUT lines in the WRAM is avoided; meanwhile, a weight matrix is divided into BR * BC sub-matrixes, the characteristic that vertical adjacent rows share an accumulator is utilized, multi-row sub-matrix calculation is loaded at a time, and virtual rearrangement maintains the row matching relation between an input vector and the weight matrix through index mapping.

The invention discloses a calculation method of a rock mass fracture network seepage field containing a complex fracture form, and relates to the field of geotechnical engineering. The method comprises the following steps: for each node in a rock mass fracture network, determining a source-sink item at the node based on the flow of a plurality of fracture units connected with the node; substituting the source sink item at the node and the equivalent permeability coefficient matrix into an overall matrix equation of fracture network seepage for solving to obtain a pressure head value of the node; the overall matrix equation is obtained by assembling unit seepage matrix equations of all nodes in the rock mass fracture network; in the construction process of the fracture seepage unit equation, the influence of the on-way resistance coefficient on the equivalent seepage flow velocity of the fracture in the complex wall surface form and the influence of the local resistance coefficient on the equivalent seepage flow velocity of the fracture in the complex wall surface form are considered; and determining the pressure head values of all nodes in the rock mass fracture network as a rock mass fracture network seepage field. The method can improve the calculation precision of the fracture network seepage field.

This invention provides a method, apparatus, graphics processor, and storage medium for matrix multiplication operators, relating to the field of artificial intelligence technology. The method includes: reading two operation matrices from high-bandwidth memory; determining a first target input matrix in the general matrix buffer (GMB) based on the two operation matrices, and identifying the other operation matrix as a second target input matrix; the first target input matrix is ​​arranged in a transposed manner; and writing the first and second target input matrices into a tensor core for matrix multiplication to obtain the target output matrix from the tensor core. This invention, by identifying the data arrangement of the two operation matrices, prioritizes inputting the first target input matrix (with a transposed arrangement) into the GMB, effectively avoiding hardware performance limitations associated with non-transposed readings, ensuring the tensor core maintains full-bandwidth data input, and significantly improving the computational efficiency of the matrix multiplication operator.

The present disclosure relates to a data processing method, apparatus, electronic device, and storage medium. The method comprises: in response to a data processing instruction for a target input feature map, obtaining a target input data block from an input data block arrangement corresponding to the target input feature map in a memory; the input data block arrangement is based on the data throughput of a general matrix processing engine in each clock cycle, and the target input feature map is divided into multiple input data blocks and stored in the memory according to a preset arrangement; the target input data blocks are cached; and based on the cached target input data blocks, matrix multiplication operations are performed using the general matrix processing engine to obtain data processing results. The embodiments of the present disclosure improve the data processing efficiency based on Img2Col, thereby improving the data processing efficiency of electronic devices based on deep learning networks.

The application relates to the field of computer architecture and parallel computing technology, and discloses an address mapping method and system for GPU convolution acceleration, which is applied to the process of performing convolution operation based on general matrix multiplication by a GPU, dynamically remaps the memory access address of a workspace matrix, and comprises the following steps: identifying a data repetition mode in the workspace matrix; based on the data repetition mode, remapping the original address of a current memory access request to a target address by using an address mapping strategy; performing a data access operation by using the target address; if the target data is cached in a cache level of the GPU, directly returning the same value in the original access address; otherwise, loading data from a global memory and storage. The application avoids frequent global memory and storage access caused by scattered storage of repeated data in a traditional method, significantly improves the cache hit rate, and further breaks through the bottleneck of low cache utilization of an existing strategy.

As general matrix multiply (GEMM) bottlenecks are ameliorated by tensor parallelism that is distributed to several processors, layer normalization (LN) surfaces as a latent bottleneck as it is not amenable to distribution. LN performance is linear to embedding size, which is extremely large in some AI models. Moreover, aggressive tiling prevents the use of internal pipelining. The disclosed implementation addresses this issue, composing LN from simpler operations and this composition is amenable to pipelining, facilitating efficient implementation of large AI models (e.g., GPTs). In both forward and backward propagation, the pipeline is stretched longer with improved balance across stages. This strategy improves throughput for larger batch-sizes as the workload benefits from pipelining operations for better performance. Furthermore, avoiding stochastic rounding further improves performance. In addition, LayerNorm checkpoints facilitate efficient computation of gradients during backward propagation.

A computing method and apparatus for general matrix multiplication (GEMM) are provided. The method comprises: in response to determining that a host needs to obtain a first calculation result of multiplication of first weight data and an input vector, performing a multiplication operation of second weight data and the input vector to obtain a second calculation result, wherein the first weight data satisfies a first arrangement rule for the host, and the second weight data is obtained by arranging the first weight data based on a second arrangement rule for a PIM apparatus different from the first arrangement rule; and obtaining the first calculation result based on the second calculation result.

The present invention discloses an automated optimization method for a mixed-precision operator with controllable error. The method includes: the first optimization, under a given input, optimizing the general matrix multiplication function through an ID conversion method, and recording the running order of the general matrix multiplication function and its corresponding dimensions; the second optimization, according to the optimized general matrix multiplication GEMM function ID list and the general matrix multiplication function order list, transmitting the input of each unoptimized general matrix multiplication call function to the fast matrix multiplication functions with preset different parameter combinations for performance timing and error calculation; the third optimization, analyzing the obtained data, replacing the single-precision matrix multiplication algorithm with a mixed-precision matrix multiplication operator, and outputting the optimized code. By using the present invention, the mixed-precision GEMM operator can help improve performance in more complex and extensive high-performance computing programs. The present invention can be widely applied to the field of compilers.

Embodiments of the present disclosure provide a method, apparatus, computing device and storage medium for calculating matrix-vector multiplication, where the matrix is an M*K matrix and the vector is a K*1 vector, M and K are positive integers. The method comprises: converting the M*K matrix into an M*(K / N)*N first tensor, where N is a positive integer, N>1, and K is a multiple of N; converting the K*1 vector into an N*(K / N) second tensor; using a tensor computing core to batch-process the general matrix multiplication of the M*(K / N)*N first tensor and the N*(K / N) second tensor to obtain M result matrices; and adding the elements on the diagonal of each of the M result matrices to obtain each element on the M*1 result vector of the M*K matrix multiplied by the K*1 vector. The above scheme uses a tensor computing core to implement high-throughput, low-latency high-speed matrix-vector multiplication (MMV) operation.

A system and method for performing multiplication for a neural network, e.g. for data of one or more layers in a neural network, may include loading a portion of a compressed version of a sparse input matrix into a cache memory; uncompressing a subset of the data in the portion of the compressed version of the sparse input matrix; and multiplying a sparse kernel matrix by the subset of the data using a set of instructions which are themselves created based on the sparse kernel matrix.

The present invention relates to a computer-implemented method, computer program, non-transitory, computer-readable medium comprising a program code and computer system for accelerating a General Matrix Multiplication (GEMM) on a computer system, and to methods, systems and computer programs / computer-readable media using such a computer-implemented method. A computer-implemented method for accelerating a General Matrix Multiplication, GEMM, on a computer system, comprises obtaining (310) a data structure comprising a representation of a first set of matrix layout variants known to be more efficient than a second set of matrix layout variants when being used to perform, by the computer system, a GEMM, with each matrix layout variant defining whether at least one of the matrices involved in the GEMM are to be transposed, and selecting (350) a matrix layout variant of the first set of matrix layout variants based on matrices involved in a GEMM to be performed, and performing (370) the GEMM on the matrices using the selected matrix layout variant.

The embodiment of the invention discloses a data processing method and device, computer equipment and a storage medium, a neural network processor comprises a global memory, a transfer cache region and a plurality of calculation units, a to-be-updated matrix is subjected to block processing to obtain a plurality of to-be-updated sub-matrixes, and the plurality of to-be-updated sub-matrixes are evenly distributed to each calculation unit; determining a first sub-matrix corresponding to each to-be-updated sub-matrix in the first matrix, and carrying the first sub-matrix from the global memory to a preset cache region of a calculation unit corresponding to each to-be-updated sub-matrix; determining a second sub-matrix corresponding to each to-be-updated sub-matrix in the second matrix, and carrying the second sub-matrixes from the global memory to a transfer cache region; obtaining a target first sub-matrix from the preset cache region, and obtaining a target second sub-matrix from the transfer cache region; and performing general matrix multiplication operation on each to-be-updated sub-matrix, the target first sub-matrix and the target second sub-matrix through a calculation unit to obtain an updated sub-matrix.

The invention provides a multi-low-rank adaptive fine tuning model reasoning optimization method and device. The method comprises a step of constructing a fusion input matrix and obtaining basic output by using a pre-training model, a step of carrying out fusion and feature extraction on input side low-rank matrixes of a plurality of LoRA adapters, and a step of carrying out fusion on output side low-rank moments of the plurality of LoRA adapters to obtain a correction term. The device comprises a corresponding processing module, namely, according to the method and the device, a plurality of small matrix multiplications of a plurality of LoRA fine tuning models are fused into a single general matrix multiplication (GEMM) operation, so that the calculation intensity is improved, the memory addressing operation is reduced, and the calculation efficiency is improved. The utilization rate of the GPU computing unit can be remarkably improved, so that the efficiency of multi-LoRA fine-tuning model reasoning is improved, and the multi-LoRA fine-tuning model reasoning can have better real-time performance and expandability.

The application discloses a method for accelerating arbitrary precision sparse matrix multiplication and addition operation based on a tensor core, which is in combination with deep learning and the program characteristics of parallel processing of sparse matrices, and is in combination of an arbitrary precision sparse matrix multiplication and addition operation operator and a parallel computing acceleration device, decomposes a general matrix into a sparse matrix of a set format, blocks the sparse matrix and specifies a corresponding parallel computing hardware unit for the sparse matrix, uses shared memory to reduce memory conflict and access delay, calls a high-level language API function library to execute the loading, calculation and storage process of the calculation data, and finally reduces the output data of the function, so that the final result of the arbitrary precision sparse matrix multiplication and addition operation on the parallel computing acceleration device is obtained. The application can fully utilize the parallel computing resources such as the tensor core on the A100, effectively reduce the program calculation amount and communication overhead, and greatly improve the program running performance.

A code optimization method and an optimization device are provided. The method is applied to the optimization device, and includes: determining M first general matrix vector multiplication (GEMV) calculations in a code to be optimized based on an identification rule; determining N second GEMV calculations in the M first GEMV calculations based on a merging rule; inserting general matrix multiplication (GEMM) calculations corresponding to the N second GEMV calculations into an insertion position of the GEMM calculations in the code, and deleting the N second GEMV calculations. This scheme can realize automatic identification and merging of GEMV calculations, and solve the problem of waste of matrix unit computing power.