339 results about "Weighing matrix" patented technology

The invention discloses a big language model-oriented inference operation acceleration method for processing an extremely long text, which belongs to the technical field of large-scale pre-training language model inference optimization, and specifically comprises the following steps of: selecting a big language model, starting a sparse attention mode, inputting different texts, recording execution time of each attention head and an attention mode type in different layers, and calculating the inference operation speed of the large language model; establishing a statistical database; solving a multi-head attention load balanced distribution strategy of the large language model under the actual text; establishing a weight index table by splitting the weight matrix; retrieving a weight sub-matrix corresponding to each attention head, and loading the weight sub-matrix to a corresponding GPU device; by asynchronously preloading the MHA calculation weight and the MLP calculation weight of the adjacent layers and combining KV cache management, load balancing in the reasoning process is realized. According to the method, dynamic segmentation, recombination and scheduling of the attention head are carried out in the reasoning stage, so that the problems of multi-GPU load imbalance and resource idling are effectively avoided, and the system throughput of long sequence processing is remarkably improved.

The invention discloses a strategy question-answering system and method based on a vertical domain large model, and the method comprises the steps: S1, receiving the natural language input of a user, and retrieving an open source data set to obtain an original data set with the comprehensive similarity meeting the requirements; s2, performing necessary data cleaning and preprocessing on the original data set to obtain a training set; s3, performing fine tuning on the pre-processed basic large language model by adopting the training data set, and introducing a low-rank structure to modify a weight matrix to obtain a final model; s4, the problem enters a final model for post-processing to generate prediction output; the system takes a large language basic model subjected to fine tuning training as a core, introduces a low-rank structure to perform fine tuning on model parameters, reduces parameter quantity required by training while keeping model performance, reduces model complexity, reduces an overfitting risk, improves generalization ability, supports deep fusion of a specific field knowledge base and universal field data, and has a wide application prospect. And the accuracy and correlation of questions and answers are improved.

The invention relates to the technical field of aircraft manufacturing, and discloses an aircraft part production quality optimization method based on data analysis. The method comprises the following steps: collecting multi-process real-time processing parameters and quality inspection data of a production line, and generating a dynamic quality weight matrix according to a process parameter coupling degree; extracting process path difference characteristics of qualified products and unqualified products in historical batches, and encoding the process path difference characteristics as a quality evolution chain matched with adjacent process parameter mutation relevance; optimizing process parameters at a quality evaluation node, driving a parameter combination to iterate to minimize quality fluctuation, generating an adjustment amount, updating an evolution chain constraint coefficient, and synchronously constructing a stability evaluation function of a correlation weight matrix and a defect propagation path; triggering process compensation according to an adjustment amount gradient, and verifying the cohesion through a tolerance rule by taking a reference parameter matched with the target evolution chain as compensation data; and generating a feedback matrix by using the quality fluctuation index and the compensation result, and correcting the mapping relation between the weight matrix and the evolution chain.

The invention discloses a large language model weight and activation combined quantification method and system, and belongs to the technical field of model quantification. The method comprises the following steps: collecting and preprocessing a calibration set, inputting a large language model to execute forward propagation, and recording an activation matrix; for each embedding dimension, counting maximum activation absolute values of all lexical elements on the dimension; generating a global threshold value by combining a quantile statistical method with the global sensitivity coefficient, and determining that the dimension with the maximum activation absolute value exceeding the global threshold value in the dimensions is an outlier dimension; respectively designing scaling factors for the normal dimension and the outlier dimension, and generating a reconstruction weight matrix; using Bayesian-gradient joint optimization to reconstruct a truncation threshold value of the weight; calculating a scaling factor of the reconstructed weight matrix to obtain a reconstructed quantized weight matrix; quantizing the activation matrix of the current layer according to the embedding dimension by applying a scaling factor; and performing multiplication calculation with the reconstructed quantization weight matrix to obtain a multiplication output result in an integer domain, and then performing unified inverse quantization recovery.

The invention relates to an automatic guided vehicle (AGV) adaptive path tracking method based on MPC. The method comprises the following steps: S1, establishing a kinematic discretization error model based on the kinematic characteristics of the two-wheel differential AGV, processing a continuous kinematic equation by adopting an Euler discretization method, expressing a dynamic change relationship between a transverse deviation distance and an angle deviation in a state-space equation form, and generating a kinematic discrete state-space model; the method has the advantages that the continuous equation is processed by establishing the kinematics discretization error model and adopting the Euler discretization method, the deviation dynamic relation is expressed through the state space, the calculation process is simplified, the precision is kept, sensor data are fused, the wheel type odometer, IMU and laser data are integrated through the extended Kalman filtering algorithm, and the precision is improved. The real-time position and angle deviation are calculated, the positioning accuracy is improved, a model prediction controller objective function is designed, a weight matrix and boundary constraint are introduced according to constraint conditions, and the effect of obstacle avoidance constraint is combined.

The invention discloses an input data sharing and cache optimization method in matrix multiplication calculation. The method comprises the steps of 1, segmenting and distributing an input data matrix and a weight matrix according to the number N of calculation cores on a chip; 2, sequentially connecting the plurality of calculation cores end to end to form a data transmission annular structure; step 3, calculating the distributed matrix multiplication by each calculation core, and transmitting the current input sub-matrix of the calculation core to the next calculation core; step 4, performing matrix multiplication operation on the transmitted input sub-matrix and the weight sub-matrix in the next calculation kernel; and 5, iterating transmission and calculation of the input sub-matrixes, and carrying out N rounds of matrix multiplication of the input sub-matrixes and the weight sub-matrixes to complete the whole operation process. The invention further discloses a system for implementing the method, and the system has wide application value.

The invention particularly discloses a model compression method based on a quantized large language model, and relates to the technical field of large language model compression. The method comprises the following steps: S1, performing quantization processing on a pre-trained large language model to obtain a quantized weight matrix, and calculating a pruning optimization objective function after quantization; s2, designing a post-quantization pruning measurement index based on the quantized weight matrix, and combining the pruning measurement index with an absolute value of the quantized weight and a pruning measurement regularization result of the original weight; and S3, based on a pruning measurement index, carrying out importance sorting on the quantized weight, generating a dynamic binary pruning mask, and minimizing quantization and pruning errors through a block optimization algorithm. The method adopts a block quantization post-pruning algorithm, minimizes quantization and pruning errors, and maximizes the model compression ratio on the premise of ensuring the model performance so as to meet the requirements of actual application on the model.

The invention discloses a collaborative filtering recommendation method and system based on logic rule reasoning, and the method comprises the steps: obtaining the original attribute characteristics of a user and a project or the negative characteristics of the original attribute characteristics based on the binary values of the continuous attributes and discrete attributes of the user and the project; performing logic conjunct or logic disjunction on the original attribute features or the negative features to obtain a logic rule matrix of a user-project pair; fusing the attribute feature information of the user and the item to obtain a unique rule weight matrix associated with the user-item pair; obtaining an interaction prediction rule factor based on the logic rule matrix and the rule weight matrix; and performing model training and prediction by using the interactive prediction rule factor. Through fusion of a neural symbol reasoning technology and a complete logic operator, dynamic mining of a logic rule between a user attribute and a project attribute is realized.

A method includes accessing a plurality of weight matrices of a machine learning model. The method also includes, for each weight matrix, decomposing the weight matrix into a U matrix, an S matrix, and a V matrix using singular value decomposition. The S matrix is a diagonal matrix, and a singular group corresponds to each element in the S matrix. The method further includes, for each weight matrix, determining an importance score of each singular group. The importance score of the singular group represents a change in loss if the singular group is removed from the machine learning model. The method also includes, for each weight matrix, ranking the singular groups across the plurality of weight matrices based on the importance scores. In addition, the method includes, for each weight matrix, identifying one or more of the singular groups to prune based on the ranking of the singular groups.

The embodiment of the invention provides a quantification method and reasoning method of an industry large model, equipment and a storage medium. When an original weight matrix of a network layer in an industry large model is quantified, values of matrix elements on diagonals of an original Hessian matrix are obtained based on an input data matrix of the network layer and can represent activation values, and the positions of columns in the original weight matrix and the original Hessian matrix of the network layer are reordered according to an activation value descending mode. Carrying out matrix partitioning processing on the reordered first weight matrix and the first Hessian matrix respectively, and quantizing the first sub-weight matrix based on the quantization parameter of each sub-weight matrix in the first weight matrix in sequence by taking the partitioned matrix as granularity; the method comprises the steps of quantizing a first weight matrix, updating other first weight sub-matrixes which are not quantized based on the quantized first weight sub-matrix, and finally determining a quantization result of an original weight matrix of a network layer based on a quantization result of each first weight sub-matrix in the first weight matrix, so that the size of an industry large model is compressed.

The invention belongs to the technical field of large models of deep learning, and provides a large-model low-rank adaptive layer weight matrix fine tuning method in order to reduce calculation complexity in a low-rank weight matrix multiplication process in a low-rank adaptive method and optimize calculation power and memory requirements required by a training model. The two low-rank weight matrixes are copied, spliced and then added, or the two low-rank weight matrixes are divided into the sub-matrixes and then the sub-matrixes are added one by one, so that compared with a mode of multiplying the two low-rank weight matrixes, the calculation complexity of matrix low-rank decomposition can be reduced, meanwhile, the rank of the low-rank weight matrixes can be improved, and the calculation efficiency of matrix low-rank decomposition is improved. When different downstream tasks are finely adjusted, the corresponding weight parameters are independently trained and switched, so that the training efficiency of different fine adjustment weights is improved, and the weight memory requirement is reduced.

The invention discloses a matrix decomposition and reconstruction method for a large model reasoning scene, and the method comprises the following steps: 1, carrying out the error statistics of conventional nodes of a network layer in a large model, and screening weight nodes capable of carrying out low-rank decomposition through a sensitivity analysis strategy; step 2, gradually performing low-rank setting, decomposing the weight matrixes in the weight nodes which can be subjected to low-rank decomposition and are obtained by screening in the step 1, and obtaining and recording respective optimal low-rank values of the decomposable weight matrixes; and step 3, performing network reconstruction on the original model based on the decomposable weight matrix and the corresponding low-rank value. The invention further discloses a system for implementing the method and a corresponding application, and the system and the application have a wide application scene.

The invention provides a federated training fine tuning method and device based on low-rank approximate quantization, and the method comprises the steps: carrying out the multi-round alternative optimization of an original weight matrix corresponding to an initial local generation type language model of a current iteration training round based on quantization and low-rank approximate processing, obtaining a target integer weight matrix corresponding to the original weight matrix and a target low-rank approximation matrix; fine-tuning the target low-rank approximation matrix by using local data to obtain a trained low-rank approximation matrix corresponding to the current iteration training round, and taking the target integer weight matrix and the trained low-rank approximation matrix as locally quantized generative language model parameters corresponding to the current iteration training round, and sending the locally quantized generative language model parameters to aggregation end equipment in a federated learning system where the local quantized generative language model parameters are located so as to perform global model aggregation. According to the method and the device, the problem that equipment computing resource consumption, computing precision, communication resource occupancy rate and privacy protection are difficult to balance in a federated learning scene of a current generative language model can be solved.

The invention provides a belief propagation LDPC decoding method and device based on deep learning, a medium and a program product. The method comprises the following steps: acquiring an LDPC code training sample set corresponding to a check matrix H; according to the check matrix H, determining a weight matrix and an activation function of each layer of a deep learning decoding model; based on the weight matrix and the activation function, using a belief propagation algorithm to establish a deep learning decoding model; training the deep learning decoding model by using the training sample set, and storing the trained deep learning decoding model as a decoder; and sending the LDPC code soft information which is corresponding to the check matrix H and needs to be decoded into the decoder to complete LDPC decoding. According to the method, iterative operation is completed by using deep learning in the error correction decoding process, the number of decoding iterations is reduced, and the sequence sent by the sending end is restored from the sequence containing noise and interference, so that the problem of decoding error codes caused by noise and other interference factors is solved.

The invention relates to the field of data processing, in particular to a fault analysis method for a three-phase intelligent electric energy meter, which comprises the following steps of: acquiring operation data of the three-phase intelligent electric energy meter in real time, and constructing a three-phase data sequence according to an acquisition time sequence; calculating correlation values between a target sequence and other sequences at a target moment by using a DTW algorithm, calculating respective weights of the three-phase sequences at the target moment based on all correlation values corresponding to the target moment, and constructing a collaborative weight matrix between the target moment and a reference moment according to all weights corresponding to the target moment and the reference moment; an existing Euclidean distance is improved by using the collaborative weight matrix, and an LOF value of data at each moment is acquired by using the improved Euclidean distance in combination with an LOF algorithm; and when the LOF value is greater than a preset LOF abnormal threshold value, judging that the electric energy meter has a fault at the moment. According to the method, the Euclidean distance is improved, the distance between the abnormal data point and the normal point is increased, the abnormal point is highlighted, and the accuracy of power consumption data anomaly detection is improved.

The embodiment of the invention provides a fine tuning method and device for a large language model. The method comprises the steps that a weight matrix of a pre-trained large language model is acquired, the weight matrix is decomposed into an amplitude vector and a first direction matrix, a first low-rank matrix and a second low-rank matrix are initialized, and the product of the first low-rank matrix and the second low-rank matrix is used for fitting an increment matrix of the first direction matrix; the first direction matrix is kept unchanged, multiple rounds of fine tuning are executed, and each round of fine tuning comprises the steps that a training text is input into a large language model, and calculation of a loss function is conducted; updating the amplitude vector based on the loss function; determining a first gradient matrix of the loss function relative to the current direction matrix; obtaining an updated first low-rank matrix and an updated second low-rank matrix according to the first gradient matrix; and determining the increment matrix of the round based on the product of the updated first low-rank matrix and the second low-rank matrix, and superposing the increment matrix of the round on the first direction matrix to serve as the updated direction matrix of the round.

The invention relates to a calculation device, a calculation method, and a program. This computing device (10) computes a matrix product of a sparse matrix and a weight matrix in a neural network, the sparse matrix having a structure in which a predetermined number of non-zero elements are included in each block of a predetermined size. The sparse matrix is defined by a local index representing the position of the non-zero element in each block and the value of the non-zero element, and the arithmetic device (10) performs: a process for acquiring, from the weight matrix, an element corresponding to the position of the non-zero element in the sparse matrix by referring to the local index; and multiplication and accumulation operation of non-zero elements of the sparse matrix and corresponding elements of the weight matrix is carried out.

The invention relates to the technical field of large model compression, and discloses a method, a device, equipment and a medium for compressing a large language model through singular value decomposition based on gradient attribution, and the method comprises the steps: calculating the gradient of an original weight matrix of each linear layer on an auxiliary calibration data set, during singular value decomposition of the original weight matrix, sparseness is distributed according to the gradient of the weight of each layer; collecting the gradient of each singular value when the original weight matrix passes through singular value decomposition on the auxiliary calibration data set; using an approximate loss item to represent the importance of each singular value when the original weight matrix passes through singular value decomposition, and filtering unimportant singular values according to a preset dynamic threshold value; and setting the filtered singular value as an index of the original weight matrix when the original weight matrix passes through singular value decomposition, and performing low-rank decomposition on the singular value related to the index in the original weight matrix layer by layer according to a gradient descent method of each singular value until the compression ratio of the large language model is met. And the compression ratio of the model can be obviously improved. The method is applied to application scenes such as financial science and technology large model compression.

Apparatus, data structure and method of adjusting weights of a neural network of a model, processing inputs of the model and outputting outputs of the model, in which the method comprises: providing training data, the training data comprising the inputs of the model and reference truth values for the outputs of the model, the reference truth values corresponding to the inputs of the model in the training data; providing an adjustment method set for adjusting the weight; determining principal component decomposition of a weight matrix including weights; determining a feature value of a covariance matrix corresponding to the feature vector; rearranging the eigenvectors in the matrix in an order that produces a monotonically decreasing order of eigenvalues associated with the eigenvectors; rearranging the weights in the weight matrix according to the rearrangement sequence of the feature vectors in the matrix; dividing the weight matrix into groups of weights; associating at least one of the groups with an adjustment method selected from the set of adjustment methods; and adjusting the weights in the at least one group on the training data using an adjustment method.

Disclosed are an inference model compression method and apparatus, which relate to the technical field of computers. The method comprises: inputting inference data into an inference model, and executing the inference model; then, for a first network layer in the inference model, acquiring a weight matrix of the first network layer and input data of the first network layer, and calculating the product of the weight matrix of the first network layer and the input data of the first network layer to obtain a result matrix that indicates the importance of weight values in the weight matrix; and next, in descending order of the importance, selecting the weight values that rank in the last a% in the weight matrix of the first network layer to perform a sparsification operation, and executing the sparsification operation on a plurality of network layers in the inference model to obtain a sparsified inference model. The first network layer is any one layer among the plurality of network layers, the result matrix and the weight matrix have the same size, and the magnitude of the value at the same position in the result matrix as in the weight matrix indicates the importance of the weight value at the same position in the weight matrix.

The invention relates to a multi-label text classification method based on positive and negative label learning and label correlation, and belongs to the field of multi-label classification. Comprising the following steps: constructing a feedforward neural network model with double hidden layers; initializing a model component; reading features and label information of samples in the training set, and generating a feature matrix and a label matrix; randomly initializing a weight matrix and an offset parameter of the model; inputting the feature matrix into an input layer of the model, and calculating neuron output layer by layer; calculating gradients of weight matrixes and bias parameters among layers in the model by adopting a composite error function, dynamically adjusting the gradients by utilizing an Adam optimization algorithm, and updating the weight matrixes and the bias parameters; when the error change amplitude is lower than a threshold value or reaches a preset number of iterations, stopping training; and after model convergence, predicting the test set to form a final multi-label classification result. According to the method, the limitation of traditional text classification is broken through through a deep learning technology, and high-precision and high-efficiency classification of complex text data is realized.

Provided are systems, methods, and apparatuses of data processing for machine learning. In one or more examples, the systems, devices, and methods include determining priority values for elements of a weight matrix based on a gradient of a loss function of an AI model and the weight matrix; determining an index value based on a number of elements in the weight importance matrix and a sparsity ratio; determining a threshold based on sorting the elements of the weight importance matrix in sequential order and determining a value of an element of the sorted weight importance matrix based on using the index value as an index of the sorted weight importance matrix; determining a pruned weights matrix based on the threshold; and processing a query using an updated AI model, the updated AI model being based on the pruned weights matrix being implemented in the AI model.

The invention discloses an improvement method of a large language model, electronic equipment and a storage medium, and the method comprises the steps: calculating a correlation matrix between middle layers through a public data set, and calculating mutual information redundancy between the layers based on the correlation matrix between the middle layers, an inter-layer redundancy matrix is constructed through mutual information redundancy among the layers; calculating the total redundancy of each layer based on the redundancy matrix, and generating a layer importance factor through function conversion; establishing an objective function based on the layer importance factors and constructing a linear optimization model of sparse rate distribution of each layer; solving the optimization model by adopting a linear programming algorithm to obtain an optimal sparse rate distribution scheme of each layer; differentiated pruning is carried out on each layer of weight matrix according to the distribution scheme, a weight parameter with the maximum amplitude in each layer of weight matrix is reserved, the rest parameters are set to be zero, and a sparse weight matrix is obtained; according to the method, the features can be selectively pruned in the middle layer, so that fine-grained optimization of a large language model is realized.

The invention discloses a feature selection method and a data reconstruction method for shoreline image feature selection. The method comprises the following steps: S1, integrating original data into a sample data set matrix X; s2, initializing a reconstruction transformation matrix W, a transformation matrix Q, an eigenvalue regression coefficient matrix A, a coordinate basis matrix B, a coding matrix E, an auxiliary matrix F and an auxiliary matrix D as unit matrixes, initializing a weight vector p and a mean vector v of a data set as unit vectors, and initializing a weight matrix P = diag (p); and S3, updating the coding matrix E, the auxiliary matrix F and the like based on the data set matrix X, the mean vector v of the data set, the transformation matrix Q, the eigenvalue regression coefficient matrix A, the coordinate basis matrix B and the weight matrix P. According to the invention, the accuracy of data reconstruction and the validity of feature selection can be improved.

The application provides an explainable sequence recommendation method, system and device fusing concepts and behaviors, relates to the technical field of recommendation systems, and the method comprises the following steps: obtaining a concept library and a historical behavior sequence of a user; obtaining a historical concept sequence and a concept embedding matrix according to the historical behavior sequence and the concept library; obtaining a concept weight matrix and a concept behavior sequence set according to the historical behavior sequence, the historical concept sequence and the concept embedding matrix in combination with an association strategy of concepts and behaviors; and performing sequence recommendation on the user according to the historical behavior sequence, the concept embedding matrix, the concept weight matrix and the concept behavior sequence set to obtain an explainable sequence recommendation result. By introducing the association strategy of concepts and behaviors, the application not only effectively fuses abstract concept information and specific behavior information, but also dynamically adjusts the understanding and explanation of user preferences through the concept embedding matrix and the concept weight matrix, thereby improving the accuracy and explainability of the sequence recommendation result.

The invention discloses an AI data processing method and system based on edge computing, and relates to the field of edge AI processing, and the method comprises the steps: defining a weight matrix of an AI model according to an edge computing node address, inputting structured data into the AI model for multi-dimensional feature analysis, and extracting high-order data representation; analyzing the processing mode through an instruction decoding method to obtain a sparseness level, and performing parameter importance evaluation through a dynamic importance scoring method based on distribution characteristics represented by high-order data and an aging weight value to generate a deformable sparse mask matrix; performing dynamic pruning fusion on a weight matrix of the AI model through sparse matrix multiplication based on the deformable sparse mask matrix to generate a sparse data processing graph; according to the method, through node resource sensing and feature mapping, the AI model weight matrix is accurately matched with the edge computing node capability, and the operation efficiency and stability of the AI model on the edge computing node are improved.

The invention discloses a memory enhancement action recognition method and system on a Riemannian manifold and a storage medium, and the method comprises the steps: 1, collecting human body action data, and representing the human body action data as a third-order tensor; 2, expanding along three modes to obtain three corresponding matrixes; 3, calculating by adopting a human short-term memory mechanism to obtain a memory enhanced weight matrix; 4, decomposing the weight matrix through a principal component analysis method to obtain a base vector with a weight; 5, recombining, normalizing and mapping to a unit hyper-sphere, and reserving an angle relation; 6, learning modal weight parameters through a Monte Carlo Markov algorithm, and calculating geometric differences between points on the hypersphere; and 7, carrying out human body action classification by adopting a K-nearest neighbor classifier. The method effectively solves the problem of time information loss in a complex scene, and is suitable for various application scenes such as medical health, virtual reality, physical training and the like.

The invention relates to the technical field of computers, and discloses a real-time calibration method and system for an edge-side weather prediction model, electronic equipment and a storage medium, and the method comprises the steps: obtaining multi-dimensional weather time sequence data, inputting the data into a neural network model, extracting spatial-temporal features, and outputting an initial prediction value; decomposing a weight matrix in the neural network model into an amplitude component and a directional component, performing low-rank matrix updating on the directional component to obtain an updated directional component, calculating a calibration weight according to the amplitude component and the updated directional component, performing deviation correction on the initial predicted value in combination with local observation data, and generating a calibrated predicted value; historical samples and current samples are dynamically managed based on prediction uncertainty indexes, an anti-forgetting training target is constructed in combination with weight regularization constraints, and low-rank matrix parameters are updated to cope with data distribution changes; according to the invention, the reliability of real-time weather prediction service can be improved.

The invention relates to a tax risk assessment method, device and model and a storage medium. The method comprises the following steps: constructing a tax risk assessment model based on a large language model; and performing fine tuning on the tax risk assessment model, including: introducing a first low-rank matrix and a second low-rank matrix into a middle layer of the tax risk assessment model, updating the first low-rank matrix and the second low-rank matrix according to an input data set, and performing fine tuning on the tax risk assessment model according to the updated first low-rank matrix and the second low-rank matrix. Updating the weight matrix of the middle layer; and according to the tax related data, performing tax risk assessment through the fine-tuned tax risk assessment model, and outputting tax risks and / or tax suggestions. According to the scheme provided by the invention, accurate, comprehensive, real-time and personalized evaluation can be carried out on the tax risk, and the accuracy, comprehensiveness, real-time performance and personalized degree of tax risk evaluation are improved.