657 results about "Matrix multiplication" patented technology

The invention discloses a processor and electronic equipment. The processor includes a computing unit including a tensor core configured to perform a matrix multiplication operation using a scaling factor, and a memory, the computing unit further including a scaling factor processing module configured to determine and cache a scaling factor for each tensor associated with the matrix multiplication operation, the computing unit further comprises at least one storage module arranged on a data path between the tensor core and the memory, and the at least one storage module is exclusively occupied by the tensor core when the tensor core executes tensor related operation. And the scaling factor processing module is arranged on the at least one storage module. At present, scaling factors and floating-point number quantization are completed by a vector calculation core, so that performance is reduced, and delay becomes high, and a scaling factor processing module arranged on a storage module in a calculation unit can improve the overall execution efficiency of low-precision matrix multiplication using the scaling factors.

Provided in the present invention is a parameter-efficient large-language fine-tuning federated learning framework, comprising the following steps: performing modeling on LoRA adapters of different edge clouds; since different weights exhibit different average performances on the LoRA adapters, using singular values to quantify the importance of the weights, and therefore, before each round of independent training of the LoRA adapters using N edge clouds, using a matrix singular value to decompose a BA matrix in the LoRA adapter for each trainable weight; configuring heterogeneous LoRA adapters on the basis of the importance of the weights; and using different numbers of quantization bits to quantize a pre-trained model, and performing high-precision inverse quantization on the pre-trained model only when matrix multiplication is executed, wherein the pre-trained model is quantized to the maximum number of quantization bits on the basis of the memory budget of the edge clouds. The present invention has the following beneficial effects: the present invention determines the optimal fine-tuning model structure, thereby improving the performance of LLM fine-tuning, and adapts to heterogeneous and resource-constrained edge clouds.

An apparatus providing tensor core matrix multiplication and accumulation (MMA) with hardware-based statistics collection and outlier suppression is disclosed. The apparatus includes processor circuitry comprising at least one processor core comprising matrix multiplication circuitry to: execute a matrix multiplication operation on first input data from a first set of registers and on second input data from a second set of registers; collect, as part of executing the matrix multiplication operation via statistics collection hardware circuitry of the matrix multiplication circuitry, output statistics data corresponding to the matrix multiplication operation; and output the output statistics data along with a result of the matrix multiplication operation; and output statistics storage to store the output statistics data.

Techniques for using a quantized and / or fused model are described. In some examples, a service is to receive a request to use a trained model, the request including input data; apply a trained model to the input data, the application of the trained model includes fusing one or more matrix multiplication operations with element-wise operations; and output a result from the trained model.

The embodiment of the invention provides an attention mechanism calculation optimization method and device, a storage medium and a program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: executing a forward re-calculation operator based on a query block and a key block, and obtaining a forward normalization result. Executing a first matrix multiplication operator based on the output gradient block and the forward normalization result to obtain a value intermediate result, and executing a point multiplication summation operator in parallel based on the forward output block and the output gradient block to obtain an intermediate tensor; executing a first type of transformation operator on the value intermediate result to obtain value gradient blocks; and meanwhile, a gradient back propagation operator is executed in parallel based on the output gradient block, the value block, the intermediate tensor and the forward normalization result, and a reverse normalization result is obtained. And finally, on the basis of a reverse normalization result, the key blocks and the query blocks, obtaining query gradient blocks and key gradient blocks, so that effective pipeline processing is performed on different operators, and the overall performance of attention mechanism calculation is improved.

The invention provides a mixed data precision matrix multiplication and addition unit and a calculation method, the matrix multiplication and addition unit comprises a calculation unit, and the calculation unit comprises a format division module, a multiplication array module, an addition tree module, an accumulator module, a normalization module and a shift register module. The calculation unit converts the first input matrix and the second input matrix into input data in a middle floating point format; executing parallel multiplication operation on the input data to generate an intermediate product result; performing index alignment and accumulation on the intermediate product result to generate an intermediate accumulated value; accumulating the intermediate product result and the value of the third input matrix in a form of accumulating an intermediate accumulated value, and outputting an accumulated result; and converting an accumulation result into a normalized result and outputting the normalized result. The format division module supports various precisions and converts data with different widths into an intermediate floating point format, so that other hardware units can be reused, and the problems that hardware resources are complex and different model reasoning scenes are difficult to meet are solved.

Apparatuses, systems, and techniques to determine a matrix multiplication algorithm for a matrix multiplication operation. In at least one embodiment, a matrix multiplication operation is analyzed to determine an appropriate matrix multiplication algorithm to perform the matrix multiplication algorithm.

The invention discloses a tensor core, a processor, a data processing method, electronic equipment and a storage medium, and is applied to the field of tensor processing. The tensor kernel comprises a first dot multiplication unit and a scaling factor matrix multiplication processing module, the scaling factor matrix multiplication processing module comprises a second dot multiplication unit, and the first dot multiplication unit and the second dot multiplication unit support dot multiplication operations of different floating-point number precisions. The tensor core is configured to receive a first tensor, a second tensor, a scaling factor of the first tensor, a scaling factor of the second tensor, an offset term of the first tensor, and an offset term of the second tensor, perform a matrix multiplication operation using the scaling factor and the offset term using a first dot multiplication unit and a scaling factor matrix multiplication processing module, and obtaining a matrix multiplication operation result of the first tensor and the second tensor. Matrix multiplication operation using scaling factors is executed by multiplexing dot multiplication units with different precisions in a tensor kernel, extra hardware area cost is reduced, and existing hardware resources are fully multiplexed.

The invention relates to the technical field of artificial intelligence compilers, in particular to an AI compiler and compiling method based on a multi-level intermediate representation framework, and the compiler comprises a high-level semantic retention layer which converts models of different AI frameworks into Lalg-on-Tensor IR intermediate representations; the hardware perception optimization layer comprises a tensor packaging and propagation module which is used for performing block packaging, layout propagation and folding of redundant packaging / unpackaging operation on the input tensor; the dynamic partitioning module is used for automatically selecting the partitioning size based on the cache capacity and the core number of the target hardware; the microkernel fusion module is used for fusing matrix multiplication, bias addition and an activation function into a single composite operator; and the microkernel collaboration layer is in butt joint with the hardware acceleration library through the XSMM dialect to generate a target hardware code. The hardware perception optimization layer can perform optimization according to different hardware characteristics, so that codes generated by the compiler can better adapt to target hardware, and the hardware utilization rate is improved.

The embodiment of the invention discloses a model reasoning calculation acceleration method and device, a medium and a program product, and the method comprises the steps: dividing the number of heads in multi-head attention processing into each processor core group according to the number of processor core groups in a processor in the multi-head attention processing of model reasoning; when the processor core group performs operation between a query vector and a key vector in multi-head attention processing, according to the total amount of the operation tasks and the number of the slave cores in the processor core group, the operation tasks are divided into the slave cores with uniform data volume; when an operation task is jointly processed by at least two slave cores, matrix segmentation is carried out on a query vector and a key vector in the operation task, and each slave core reads a corresponding vector according to a matrix segmentation result to carry out matrix multiplication operation; and caching the first matrix multiplication operation sub-result obtained by calculation of each slave core to a high-speed cache of a specified first target slave core to generate a first matrix multiplication operation result. The calculation efficiency can be improved by balancing the data volume of each slave core.

The invention relates to an optimization method of a hybrid expert system, computer equipment, a readable storage medium and a program product. A plurality of experts contained in the MOE are deployed in a plurality of artificial intelligence chips in groups, and the method comprises the following steps: carrying out routing calculation on an original input tensor to obtain a routing calculation result; determining input element grouping information corresponding to each expert based on expert index information and expert weight information in the routing calculation result; taking the expert dimension as a parallel dimension, executing rearrangement operation for the original input tensor in parallel based on the input element grouping information, taking a rearrangement result as input information of an expert, executing matrix multiplication and accumulation operation, and taking the expert dimension as the parallel dimension, and based on the input element grouping information, executing anti-rearrangement operation of the matrix multiplication and accumulation operation result in parallel to obtain a final operation result. By adopting the method, the MOE reasoning performance can be improved.

The invention discloses an attention mechanism calculation method and device, a storage medium and a product, and the method comprises the steps: carrying out matrix multiplication operation through employing a query matrix block of a first register block and a key matrix block of a shared memory, obtaining a first product matrix block, and writing the first product matrix block into a second register block; performing exponential operation by using the first product matrix blocks to obtain sub-matrix blocks, writing the sub-matrix blocks into a second register block in a covering manner, and writing the sub-matrix blocks into a third register block in the form of a target precision type; performing matrix multiplication operation by using the sub-matrix blocks of the third register block and the value matrix blocks of the shared memory to obtain second product matrix blocks, and writing the second product matrix blocks into a second register block; performing softmax operation by using the second product matrix blocks to obtain attention result matrix blocks, and writing the attention result matrix blocks into a fourth register block; and writing the attention result matrix of the fourth register group into the shared memory in blocks. According to the embodiment of the invention, overflow of the register can be avoided, and the utilization of hardware resources is maximized.

The invention discloses an attention mechanism calculation method and device, a medium and a product. The method comprises the steps that a second thread bundle group is controlled to load an ith query matrix block; controlling the second thread bundle group to perform matrix multiplication operation and exponential operation by using the ith query matrix block and the transposed jth key matrix block to obtain a jth attention score matrix block; and controlling the first thread bundle group and the second thread bundle group to alternately use different sub-blocks of the jth value matrix block to perform attention mechanism operation of the jth attention score matrix block until the last sub-block of the jth attention result matrix block is obtained. By adopting the embodiment of the invention, sufficient register resources can be provided for the calculation of the attention mechanism, so that the calculation efficiency is improved.

The embodiment of the invention provides a matrix multiplication and accumulation operation unit and method, a hardware accelerator and electronic equipment, and the matrix multiplication and accumulation operation unit comprises a data loading storage engine, a tensor register file and a matrix multiplication engine. The data loading and storage engine is used for loading data of a plurality of matrixes to be subjected to matrix multiplication and accumulation calculation; the tensor register file is used for storing data of a plurality of matrixes acquired from the data loading storage engine; the tensor register file comprises at least three tensor register groups, each tensor register group comprises a plurality of tensor registers, and different tensor register groups are used for storing data of different matrixes in the plurality of matrixes; and the matrix multiplication engine is used for carrying out matrix multiplication accumulation calculation based on the data of the plurality of matrixes stored in the tensor register file. According to the embodiment of the invention, more efficient MMA calculation is realized under the conditions of low cost, low power consumption and less occupied space.

The invention discloses a multi-precision matrix calculation unit and a use method thereof, and relates to the technical field of integrated circuits, the multi-precision matrix calculation unit comprises a control logic unit, a cache module and a calculation array; wherein the control logic unit is used for configuring a computing array and controlling the cache module to read in and read out data; the cache module comprises three first cache sub-modules and one second cache sub-module; wherein the three first cache sub-modules are respectively used for caching a first to-be-processed matrix, a second to-be-processed matrix and a third to-be-processed matrix, and the second cache sub-module is used for caching a result matrix; and the calculation array is used for carrying out multiplication and addition operation on different types of to-be-processed matrixes to obtain a result matrix and writing the result matrix back to the cache module. The method can serve as a basic operation core for integrated processing of large-scale matrix multiplication and can also be integrated in processors such as RISC-V for matrix operation, universality is higher, and the method can adapt to fast and efficient calculation scenes.

The invention discloses a multi-modal data driven dynamic pressure test scene generation method and system, and the method comprises the steps: carrying out the event extraction and matching of a user operation log and a full-link API call sequence, and carrying out the feature enhancement of each alignment event pair, and obtaining a local feature vector; statistical features are extracted from each user session, a graph attention network is constructed, and a group feature matrix is obtained; obtaining a global incidence matrix according to the mapping between the service load and the resource consumption; and splicing the user session feature matrix and the group feature matrix to generate group enhancement features, performing matrix multiplication on the group enhancement features and the global incidence matrix to obtain system-level risk features, splicing local feature vectors and the encoded user behavior logs, splicing the system-level risk features and the encoded performance indexes, and obtaining the user behavior log. And finally, carrying out multi-source fusion to generate a pressure measurement scene. The simulation precision, the dynamic adaptive capacity, the abnormal reproduction capacity, the resource utilization rate and the like are remarkably improved.

The invention discloses an input data sharing and cache optimization method in matrix multiplication calculation. The method comprises the steps of 1, segmenting and distributing an input data matrix and a weight matrix according to the number N of calculation cores on a chip; 2, sequentially connecting the plurality of calculation cores end to end to form a data transmission annular structure; step 3, calculating the distributed matrix multiplication by each calculation core, and transmitting the current input sub-matrix of the calculation core to the next calculation core; step 4, performing matrix multiplication operation on the transmitted input sub-matrix and the weight sub-matrix in the next calculation kernel; and 5, iterating transmission and calculation of the input sub-matrixes, and carrying out N rounds of matrix multiplication of the input sub-matrixes and the weight sub-matrixes to complete the whole operation process. The invention further discloses a system for implementing the method, and the system has wide application value.

The invention discloses a matrix multiplication task execution method and device, equipment, a medium and a program product, the method is applied to a processor unit in a graphics processor, and the method comprises the steps that in the process that the processor unit executes a matrix multiplication task, the processor unit executes the matrix multiplication task; distributing a target register for the matrix multiplication task in the processor unit, wherein the target register comprises a shared register; generating at least two types of second thread bundles for executing the matrix multiplication task; the at least two types of second thread bundles asynchronously execute the matrix multiplication task based on the common register; and the common register is used for storing an intermediate result of the matrix multiplication task.

According to the matrix multiplication operation method and device and the storage medium, an original input matrix of a client side is packaged and encrypted through homomorphic encryption, a server side is allowed to operate in a ciphertext mode, and therefore the data privacy of the client side is protected. Afterwards, verification is carried out on the result of matrix multiplication of the server side by using homomorphism of encryption operation through homomorphic hash, and the matrix multiplication and the result of matrix multiplication form a closed loop through a polynomial packaging technology, so that the data security of the client side and the verifiability of a reasoning result are effectively ensured. Moreover, under dual verification of commitment verification and result verification, the accuracy of the target operation result is ensured.

The invention relates to a tensor accelerator-oriented template calculation code generation method and system. The method comprises the following steps of: S1, calculating and converting a template into a field specific programming language designed by the tensor accelerator-oriented template calculation code generation method and the tensor accelerator-oriented template calculation code generation system; s2, carrying out row segmentation and column segmentation on the weight matrix calculated by the template; s3, stacking and rearranging the segmented weight rows / columns to form a weight matrix suitable for matrix multiplication; s4, starting an automatic tuning code generator, forming matrix multiplication cycle optimization configuration through evolutionary search, and screening candidate configuration by adopting a cost model; s5, the screened candidate configuration is used for actual measurement performance of a real machine, and an actual measurement result is used for updating a cost model; and S6, checking whether the number of evolution search iterations reaches an iteration number threshold or not, if so, ending the program, and selecting the optimal configuration, otherwise, turning to S4. According to the method, the multi-stage soft flow for memory access and calculation is constructed, so that the memory access overhead can be better masked.