Model processing method and apparatus, and computer device, readable storage medium and program product

Abstract

The present application relates to a model processing method and apparatus, and a computer device, a readable storage medium and a program product. The method comprises: on the basis of a diffusion kernel algorithm, performing low-dimensional conversion processing on input data, a target layer, and an activation function of a layer subsequent to the target layer, so as to obtain a converted feature vector; on the basis of a preset inter-layer similarity algorithm and the converted feature vector, constructing a similarity matrix between the target layer and the layer subsequent to the target layer; and determining similar layers from the similarity matrix, fusing the similar layers according to a preset fusion parameter so as to obtain fused similar layers, and on the basis of each fused similar layer, obtaining a compressed large language model. By means of the method, the performance of a large language model can be ensured, and the size of the model can also be effectively compressed.

Description

Model processing methods, apparatus, computer equipment, readable storage media, and program products

[0001] Related applications

[0002] This application incorporates Chinese Patent Application No. 2024117021085, filed on November 26, 2024, entitled “Model Processing Method, Apparatus, Computer Equipment, Readable Storage Medium and Program Product”, which is incorporated herein by reference in its entirety. Technical Field

[0003] This application relates to the field of natural language processing technology, and in particular to a model processing method, apparatus, computer device, readable storage medium, and program product. Background Technology

[0004] In the field of natural language processing, especially in the application of Large Language Models (LLMs), deep learning-based large language models can understand complex language semantics and generate coherent and relevant text content through pre-training on large-scale text data. Large language models often require enormous computing resources and storage space during deployment. In traditional techniques, model compression methods typically employ parameter pruning and quantization, leading to a performance degradation of large language models. Summary of the Invention

[0005] Therefore, it is necessary to provide a model processing method, apparatus, computer equipment, readable storage medium, and program product to address the above-mentioned technical problems, which can effectively compress the model size while ensuring the performance of large language models.

[0006] Firstly, this application provides a model processing method, including:

[0007] Based on the diffusion kernel algorithm, the input data, the target layer, and the activation function of the next layer of the target layer are subjected to low-dimensional transformation to obtain the transformed feature vector. The model includes the target layer and the next layer of the target layer.

[0008] Based on the preset inter-layer similarity algorithm and the transformed feature vector, a similarity matrix is ​​constructed between the target layer and the next layer of the target layer;

[0009] A similarity layer is determined in the similarity matrix, and the similarity layer is fused according to a preset fusion parameter to obtain a fused similarity layer. Based on each of the fused similarity layers, a compressed large language model is obtained.

[0010] In some embodiments, the low-dimensional transformation based on the diffusion kernel algorithm and the input data, the target layer, and the activation function of the next layer after the target layer to obtain the transformed feature vector includes:

[0011] A kernel matrix is ​​constructed based on the input data, the target layer, the activation function of the next layer after the target layer, and the kernel bandwidth parameter. A degree matrix is ​​then constructed based on the elements of the kernel matrix.

[0012] The difference between the degree matrix and the kernel matrix is ​​determined as the Laplace matrix;

[0013] Determine the eigenvectors corresponding to the target eigenvalues ​​of the Laplacian matrix;

[0014] The product of the eigenvector and the Laplacian matrix is ​​determined as the transformed eigenvector.

[0015] In some embodiments, constructing a similarity matrix between the target layer and the next layer of the target layer based on a preset inter-layer similarity algorithm and the transformed feature vector includes:

[0016] Calculate the joint probability distribution function of the target layer and the next layer below the target layer, and calculate the edge probability distribution function of the target layer and the edge probability distribution function of the next layer below the target layer;

[0017] Based on the joint probability distribution function, the edge probability distribution function of the target layer, and the edge probability distribution function of the next layer of the target layer, a similarity matrix between the target layer and the next layer of the target layer is constructed.

[0018] In some embodiments, the method further includes:

[0019] The sum of the multi-head attention value, feedforward value, and layer normalization value of the activation function of the layer preceding the target layer is determined as the activation function of the target layer.

[0020] In some embodiments, the method further includes:

[0021] Calculate the performance parameters of the compressed large language model;

[0022] If the performance parameters do not meet the preset performance conditions, the step of constructing the similarity matrix of the target layer and the next layer of the target layer based on the preset inter-layer similarity algorithm and the transformed feature vector continues until the performance parameters meet the preset performance conditions.

[0023] In some embodiments, the method further includes:

[0024] The initial inference tree of the large language model is constructed based on a preset inference tree inference algorithm, and the initial inference tree includes at least one target node;

[0025] For each target node, calculate the product of the confidence scores of all nodes along the path from the root node to the target node, and determine the product as the node value function of the target node; based on the node value function, the depth value of the target node, and the hyperparameter for adjusting the depth effect, determine the modified node value function of the target node; rearrange the path from the root node to the target node based on the modified node value function to obtain the rearranged path;

[0026] Based on the rearranged paths corresponding to each of the target nodes, a target inference tree is obtained.

[0027] Secondly, this application also provides a model processing apparatus, comprising:

[0028] The processing module is used to perform low-dimensional transformation processing on the input data, the target layer, and the activation function of the next layer of the target layer based on the diffusion kernel algorithm to obtain the transformed feature vector. The model includes the target layer and the next layer of the target layer.

[0029] A construction module is used to construct a similarity matrix between the target layer and the next layer of the target layer based on a preset inter-layer similarity algorithm and the transformed feature vector;

[0030] The fusion module is used to determine the similarity layer in the similarity matrix, and fuse the similarity layer according to the preset fusion parameters to obtain the fused similar layer, and to obtain the compressed model based on each of the fused similar layers.

[0031] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0032] Based on the diffusion kernel algorithm, the input data, the target layer, and the activation function of the next layer of the target layer are subjected to low-dimensional transformation to obtain the transformed feature vector. The model includes the target layer and the next layer of the target layer.

[0033] Based on the preset inter-layer similarity algorithm and the transformed feature vector, a similarity matrix is ​​constructed between the target layer and the next layer of the target layer;

[0034] A similarity layer is determined in the similarity matrix, and the similarity layer is fused according to a preset fusion parameter to obtain a fused similarity layer. Based on each of the fused similarity layers, a compressed large language model is obtained.

[0035] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0036] Based on the diffusion kernel algorithm, the input data, the target layer, and the activation function of the next layer of the target layer are subjected to low-dimensional transformation to obtain the transformed feature vector. The model includes the target layer and the next layer of the target layer.

[0037] Based on the preset inter-layer similarity algorithm and the transformed feature vector, a similarity matrix is ​​constructed between the target layer and the next layer of the target layer;

[0038] A similarity layer is determined in the similarity matrix, and the similarity layer is fused according to a preset fusion parameter to obtain a fused similarity layer. Based on each of the fused similarity layers, a compressed large language model is obtained.

[0039] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0040] Based on the diffusion kernel algorithm, the input data, the target layer, and the activation function of the next layer of the target layer are subjected to low-dimensional transformation to obtain the transformed feature vector. The model includes the target layer and the next layer of the target layer.

[0041] Based on the preset inter-layer similarity algorithm and the transformed feature vector, a similarity matrix is ​​constructed between the target layer and the next layer of the target layer;

[0042] A similarity layer is determined in the similarity matrix, and the similarity layer is fused according to a preset fusion parameter to obtain a fused similarity layer. Based on each of the fused similarity layers, a compressed large language model is obtained.

[0043] The aforementioned model processing method, apparatus, computer equipment, readable storage medium, and program product calculate the transformed feature vector using a diffusion kernel algorithm, calculate the similarity matrix between the target layer and the layer below the target layer based on the transformed feature vector and a preset inter-layer similarity algorithm, fuse the similar layers based on the similarity matrix, obtain the compressed model based on the fused similar layers, extract model information through streaming embedding technology, realize inter-layer fusion and merging, and effectively compress the model. This can reduce the model size while ensuring model performance, and better utilize redundant information in the model, thus achieving efficient model compression. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 is a flowchart illustrating a model processing method in one embodiment;

[0046] Figure 2 is a flowchart illustrating the model processing method in one embodiment;

[0047] Figure 3 is a flowchart illustrating the model processing method in one embodiment;

[0048] Figure 4 is a flowchart illustrating the model processing method in one embodiment;

[0049] Figure 5 is a structural block diagram of a model processing device in one embodiment;

[0050] Figure 6 is an internal structure diagram of a computer device in one embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] In an exemplary embodiment, as shown in FIG1, a model processing method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0053] Step 101: Based on the diffusion kernel algorithm, perform low-dimensional transformation on the input data, the target layer, and the activation function of the next layer after the target layer to obtain the transformed feature vector.

[0054] The diffusion kernel algorithm is used to transform high-dimensional layers into low-dimensional representations. Input data can be natural language text, images, etc. The input data, the function of the target layer, and the activation function of the next layer after the target layer can all be high-dimensional data. The model can be a large language model, which includes at least multiple target layers arranged sequentially. These target layers can be neural network layers, such as multiple convolutional layers, multiple pooling layers, etc. The terminal can determine the next layer after the target layer according to their arrangement.

[0055] Specifically, the target layer can be any layer among the multiple neural network layers contained in the large language model. After determining the target layer, the terminal can perform low-dimensional transformation processing on the target layer based on the diffusion kernel algorithm, the input data, the activation function of the target layer, and the activation function of the next layer after the target layer, to obtain the transformed feature vector of the target layer.

[0056] Step 102: Based on the preset inter-layer similarity algorithm and the transformed feature vector, construct the similarity matrix between the target layer and the next layer of the target layer.

[0057] The preset inter-layer similarity algorithm can be a normalized pairwise information similarity measurement algorithm. The similarity matrix can represent the similarity between the target layer and the next layer below it.

[0058] Specifically, the terminal can calculate the similarity matrix between the target layer and the next layer of the target layer by normalizing the information similarity measurement algorithm, the transformed feature vector of the target layer, and the transformed feature vector of the next layer of the target layer.

[0059] Step 103: Determine the similarity layer in the similarity matrix, and fuse the similarity layer according to the preset fusion parameters to obtain the fused similarity layer, and obtain the compressed model based on each fused similarity layer.

[0060] Among them, the fused similarity layer can be the target layer and the next layer after the target layer, where the similarity matrix satisfies the preset similarity conditions.

[0061] Specifically, the terminal can determine the relationship between the similarity matrix and preset similarity conditions. The terminal can identify the target layer and the next layer that meet the preset similarity conditions as similar layers. The terminal can then fuse these similar layers according to preset fusion parameters to obtain a fused similar layer consisting of the target layer and the next layer. The terminal can calculate the preset fusion parameters based on the parameters at the same position in the target layer and the next layer, as well as the parameters during the m-th compression. The specific formula for calculating the preset fusion parameters can be: θ m =λ m θ i +(1-λ m )θ j

[0062] Where, θ m λ represents the parameters for the merged position of the target layer and the layer below it; m Let θ be the coefficient for the m-th compression. i θ represents the parameters at the same location in the target layer and the layer below it. j The parameter is located at the same position as the next layer below the target layer.

[0063] In addition, the terminal can also use a weight allocation function to fuse the target layer and the next layer of the target layer that meet the preset similarity conditions to obtain the fused similar layer.

[0064] The terminal can execute the method described in the above embodiments for each layer in the model as a target layer to obtain fused similar layers corresponding to multiple target layers, and obtain a compressed model based on the fused similar layers corresponding to each target layer; optionally, the terminal can also determine the fused similar layers corresponding to each target layer, as well as the layers that have not been fused, as the compressed model.

[0065] The above model processing method calculates the transformed feature vector using the diffusion kernel algorithm, calculates the similarity matrix between the target layer and the next layer based on the transformed feature vector and the preset inter-layer similarity algorithm, fuses the similar layers based on the similarity matrix, obtains the compressed model based on the fused similar layers, extracts model information through streaming embedding technology, realizes inter-layer fusion and merging, and effectively compresses the model. It can reduce the model size while ensuring the model performance, and better utilize the redundant information in the model to achieve efficient model compression.

[0066] In one exemplary embodiment, step 101 specifically includes:

[0067] A kernel matrix is ​​constructed based on the input data, the target layer, and the activation function and kernel bandwidth parameters of the next layer after the target layer. A degree matrix is ​​then constructed based on the elements of the kernel matrix. The difference between the degree matrix and the kernel matrix is ​​determined as the Laplacian matrix. The eigenvectors corresponding to the target eigenvalues ​​of the Laplacian matrix are determined. The product of the eigenvectors and the Laplacian matrix is ​​determined as the transformed eigenvectors.

[0068] Specifically, the terminal constructs a kernel matrix based on the input data, the target layer, and the activation function and kernel bandwidth parameters of the layer below the target layer; the formula for expressing the kernel matrix can be:

[0069] Where K is the kernel matrix, H i H is the activation function of the target layer. j σ is the activation function of the next layer after the target layer, and σ is the kernel bandwidth parameter.

[0070] The terminal can perform diagonal processing based on the elements of the kernel matrix to obtain the degree matrix. The formula for expressing the degree matrix is ​​as follows:

[0071] The terminal can determine the Laplacian matrix by the difference between the degree matrix and the kernel matrix. The terminal can then determine the eigenvectors corresponding to the target eigenvalues ​​of the Laplacian matrix. Finally, the terminal can determine the transformed eigenvectors by multiplying the eigenvectors by the Laplacian matrix. The specific formula for expressing the transformed eigenvectors can be:

[0072] Where E is the transformed feature vector; EigVectords d This refers to the eigenvectors corresponding to the d smallest eigenvalues ​​of the Laplacian matrix L.

[0073] In this embodiment, converting high-dimensional data into low-dimensional representations helps to understand and compare the relationships between different layers.

[0074] In one exemplary embodiment, step 102 specifically includes:

[0075] Calculate the joint probability distribution function of the target layer and the next layer below the target layer, as well as the marginal probability distribution function of the target layer and the next layer below the target layer; based on the joint probability distribution function, the marginal probability distribution function of the target layer, and the marginal probability distribution function of the next layer below the target layer, construct the similarity matrix of the target layer and the next layer below the target layer.

[0076] Specifically, the terminal can calculate the joint probability distribution function of the target layer and the next layer below it, as well as the marginal probability distribution function of the target layer and the marginal probability distribution function of the next layer below it. The terminal can calculate the shared information between the joint probability distribution function of the target layer and the next layer below it, and the marginal probability distribution functions of the target layer and the next layer below it. The terminal can also calculate the normalized value of the target layer and the next layer below it, and determine the ratio of the shared information to the normalized value as the similarity matrix between the target layer and the next layer below it. The specific formula for calculating the similarity matrix can be:

[0077] Among them, S ij Let p(x,y) be the similarity matrix between the target layer and the next layer after the target layer. i and E j The joint probability distribution, p(x) and p(y) represent E, respectively. i and E j The marginal probability distribution of E. i E is the transformed feature vector of the target layer. j This is the transformed feature vector of the next layer after the target layer.

[0078] In this implementation, a similarity matrix between the target layer and the next layer of the target layer is calculated using a preset inter-layer similarity algorithm. Considering the shared information between layers, as well as the information content of each layer itself, the accuracy of the similarity calculation between layers is improved, which facilitates subsequent compression between layers.

[0079] In one exemplary embodiment, the model processing method further includes:

[0080] The sum of the multi-head attention value, feedforward value, and layer normalization value of the activation function of the layer preceding the target layer is determined as the activation function of the target layer.

[0081] The model can also include the layer above the target layer.

[0082] Specifically, the terminal can perform a multi-head attention operation on the activation function of the layer preceding the target layer to obtain a multi-head attention value; the terminal can perform a feedforward operation on the activation function of the layer preceding the target layer to obtain a feedforward value; the terminal can add the multi-head attention value, the feedforward value, and the activation function of the layer preceding the target layer (residual connection) to obtain a sum, and then perform a layer normalization operation on the sum to obtain a layer normalized value. The terminal can determine the activation function of the target layer as the sum of the multi-head attention value, the feedforward value, and the layer normalized value of the activation function of the layer preceding the target layer. The specific formula for the activation function of the target layer can be: H l =LayerNorm(H l-1 )+MultiHead(H l-1 )+FeedForward(H l-1 )

[0083] Among them, H l Indicates the activation of layer l, MultiHead(H) l -1) can be the multi-head attention value of the activation function of the previous layer. LayerNorm(H) l -1) can be the layer normalization value of the activation function of the previous layer. FeedForward(H l -1) can be the feedforward value of the activation function of the previous layer. The activation function represents the output of each layer of the model when processing the input data.

[0084] Additionally, if the target layer is the first layer of the model, the terminal can calculate the weights and biases of the first layer. Based on these weights, biases, and input data, the terminal can calculate the linear change value of the first layer. The terminal can also calculate the activation function of the first layer based on a preset activation function and the linear change value. The preset activation function can be a Rectified Linear Unit (ReLU), etc., and is not specifically limited here.

[0085] In this embodiment, by calculating the activation function of each layer, it is easier to compress and merge the layers in the future.

[0086] In one exemplary embodiment, the model processing method further includes:

[0087] Calculate the performance parameters of the compressed model; if the performance parameters do not meet the preset performance conditions, continue to execute the steps of constructing the similarity matrix of the target layer and the next layer of the target layer based on the preset inter-layer similarity algorithm and the transformed feature vector, until the performance parameters meet the preset performance conditions.

[0088] Among them, performance parameters can reflect the performance of a large language model, such as running speed and resource consumption. Preset performance conditions can be predetermined compression targets or performance thresholds.

[0089] Specifically, the terminal can set preset performance conditions based on user-specified model performance conditions or the model's own performance threshold. The terminal can calculate the performance parameters of the model after each compression. The terminal can compare the performance parameters with the preset performance conditions. If the performance parameters of the currently compressed model exceed the parameter range set by the preset performance conditions, it is determined that the performance parameters do not meet the preset performance conditions. The terminal then compresses the model and continues execution from step 102 until the compressed performance parameters meet the preset performance conditions. If the terminal calculates that the performance parameters of the currently compressed model are within the parameter range set by the preset performance conditions, it is determined that the performance parameters meet the preset performance conditions. The terminal then terminates the model compression and performs real-time dialogue processing or other processing based on the compressed model.

[0090] In addition, the terminal can receive a termination command and end the compression of the model based on the termination command.

[0091] In this embodiment, by judging whether the model meets the preset performance conditions, the model performance is continuously monitored during the compression process to find the best balance between compression ratio and performance, thus ensuring the performance of the model.

[0092] In one exemplary embodiment, the model processing method further includes:

[0093] The initial inference tree of the model is constructed based on a pre-defined inference tree inference algorithm. For each target node, the product of the confidence scores of each node included in the path from the root node to the target node is calculated, and the product value is determined as the node value function of the target node. Based on the node value function, the depth value of the target node, and the hyperparameters that adjust the influence of the depth, the modified node value function of the target node is determined. Based on the modified node value function, the path from the root node to the target node is rearranged to obtain the rearranged path. Based on the rearranged path corresponding to each target node, the target inference tree is obtained.

[0094] The preset inference tree algorithm can be an adaptive dynamic inference tree inference acceleration algorithm. The initial inference tree includes at least one target node. The path from the root node to the target node can be the draft token path. The confidence score can be the model's prediction probability of the draft token.

[0095] Specifically, the terminal can construct a model based on a preset inference tree algorithm to determine the root node and at least one target node, as well as one or more paths from the root node to each target node. The terminal uses the root node, one target node, and one or more paths as the initial inference tree. For each target node, the terminal can calculate the product of the confidence scores of each node included in the path from the root node to the target node, and use this product as the node value function of the target node. The specific formula for calculating the node value function can be:

[0096] Where Value(t) i ) can be a node-valued function for the target node, Path(root,t) i ) represents the path from the root node to the target node, c j This represents the confidence level of the nodes from the root node to the target node.

[0097] The terminal can determine the modified node value function of the target node based on the node value function, the depth value of the target node, and the hyperparameters adjusting the depth effect. The specific formula for the modified node value function of the target node can be: Value * (t i ) = Value(t i )×depth(t i ) -α

[0098] Where, Value * (t i The function depth(t) can be the modified node value of the target node. i ) represents the depth of the target node, and α is a hyperparameter that adjusts the influence of depth.

[0099] The terminal can rearrange the path from the root node to the target node based on the modified node value function to obtain the rearranged path; based on the rearranged path corresponding to each target node, the target inference tree is obtained.

[0100] In addition, to ensure consistency between the generation process and the original autoregressive decoding, the terminal can determine all ancestor nodes of each target node. If the current node is an ancestor of the target node, its mask value is set to 1; otherwise, it is set to 0. A mask value of 1 indicates that the current node is of interest when processing the target node, while a mask value of 0 indicates that the current node is not of interest when processing the target node. The formula for calculating the mask M is: M[i,j]=1 if j∈Ancestors(i) else 0

[0101] Here, Ancestors(i) represents the set of all ancestor nodes of node i.

[0102] In this embodiment, the token acceptance rate is improved by dynamically adjusting the inference tree structure, thereby accelerating the inference process of the large language model. This ensures that each token only notices its ancestor nodes in the tree structure, thus maintaining consistency in the generation process. While guaranteeing generation quality, the inference speed of the large language model is significantly improved. An adaptive dynamic inference tree structure is introduced, and through steps such as expansion, rearrangement, and attention mask adjustment, the traditional autoregressive decoding process is optimized, improving inference efficiency.

[0103] In an exemplary embodiment, as shown in FIG2, the model processing method specifically includes the following steps:

[0104] Step 201: Extract the activation function of each layer in the large language model from the terminal.

[0105] Step 202: The terminal performs low-dimensional transformation processing on the input data, the target layer, and the activation function of the next layer of the target layer based on the diffusion kernel algorithm to obtain the transformed feature vector.

[0106] Step 203: The terminal constructs a similarity matrix between the target layer and the next layer of the target layer based on the normalized pairwise information similarity measurement algorithm and the transformed feature vector.

[0107] Step 204: The terminal selects the target layer and the next layer of similarity layers according to the similarity matrix, and fuses the similar layers according to the preset fusion parameters to obtain the fused similar layers, and obtains the compressed model based on each fused similar layer.

[0108] Step 205: The terminal calculates the performance parameters of the compressed model. It determines whether the performance parameters meet the preset performance conditions. If the performance parameters of the compressed model do not meet the preset performance conditions, then proceed to step 203.

[0109] Step 206: If the performance parameters of the compressed model meet the preset performance conditions, then output the compressed model.

[0110] In this embodiment, by mapping the model to a low-dimensional manifold space, the essential relationships and structure between parameters are effectively captured. This makes the model compression process more accurate and efficient, especially when dealing with ultra-large-scale language models. Manifold embedding technology reduces model parameters while better maintaining the model's performance and expressive power, thereby effectively improving the quality and efficiency of the compressed model. During the model compression stage, similar layers are merged by analyzing the similarity between layers in the manifold space. By constructing a similarity matrix and selecting the most similar layer pairs for merging, the compressed model structure becomes more compact and efficient, especially in processing deep networks, where it can more accurately preserve the model's key features.

[0111] In an exemplary embodiment, as shown in Figure 3, the model processing method specifically includes the following steps:

[0112] Step 301: The terminal constructs the initial inference tree of the model based on the preset inference tree inference algorithm.

[0113] Step 302: For each target node, the terminal calculates the product of the confidence scores of each node in the path from the root node to the target node, and determines the product as the node value function of the target node.

[0114] Step 303: The terminal determines the modified node value function of the target node based on the node value function, the depth value of the target node, and the hyperparameters that adjust the depth effect.

[0115] Step 304: The terminal rearranges the path from the root node to the target node based on the modified node value function to obtain the rearranged path.

[0116] Step 305: The terminal obtains the target inference tree based on the rearranged paths corresponding to each target node.

[0117] Step 306: The terminal adjusts the attention mask based on whether the current node is an ancestor node of the target node. The attention mask value of the current node that is an ancestor node of the target node is set to 1, otherwise it is set to 0.

[0118] In this embodiment, the token acceptance rate is improved by dynamically adjusting the inference tree structure, thereby accelerating the inference process of a large language model. It mainly includes two core steps: an expansion stage and a reordering stage. Optimizing these two stages achieves efficient inference acceleration. By constructing a dynamic inference tree structure and combining steps such as expansion, reordering, and attention mask adjustment, the model's inference efficiency is significantly improved, enabling the large language model to process multiple possible output paths in parallel during generation, greatly reducing inference latency. Compared to traditional autoregressive decoding methods, the adaptive dynamic inference tree algorithm has a significant advantage in improving inference speed, making it particularly suitable for applications requiring real-time response.

[0119] In an exemplary embodiment, as shown in Figure 4, the model processing method can specifically include two parts: a knowledge extraction and layer merging compression part based on manifold embedding and a reasoning acceleration part based on adaptive dynamic reasoning trees. The knowledge extraction and layer merging compression part based on manifold embedding includes knowledge extraction via manifold embedding and layer merging based on similarity. Knowledge extraction via manifold embedding may include extracting layer activation and diffusion kernel algorithms; layer merging based on similarity may include constructing a similarity matrix and merging similar layers. The reasoning acceleration part based on adaptive dynamic reasoning trees may include reasoning tree expansion, tree node rearrangement, and attention mask adjustment.

[0120] In this embodiment, by combining manifold embedding-based model compression with an adaptive dynamic inference tree acceleration algorithm, not only is the model size significantly reduced, but inference speed is also greatly improved while maintaining model performance. Manifold embedding technology achieves efficient model compression, while the adaptive dynamic inference tree structure significantly improves inference efficiency, achieving the technical effect of significantly compressing model size and accelerating the inference process. This provides a new solution for related applications in the field of natural language processing, especially for the deployment and application of large models in resource-constrained environments. It is particularly suitable for application scenarios requiring the deployment of large-scale language models in resource-constrained environments. This application can be applied to application scenarios of fast-response real-time text generation and dialogue systems.

[0121] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0122] Based on the same inventive concept, this application also provides a model processing apparatus for implementing the model processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more model processing apparatus embodiments provided below can be found in the limitations of the model processing method described above, and will not be repeated here.

[0123] In an exemplary embodiment, as shown in FIG5, a model processing apparatus 50 is provided, including: a processing module 51, a construction module 52, and a fusion module 53, wherein:

[0124] Processing module 51 is used to perform low-dimensional transformation processing on input data, target layer and activation function of the next layer of target layer based on diffusion kernel algorithm to obtain transformed feature vector. The model includes target layer and the next layer of target layer.

[0125] Module 52 is used to construct a similarity matrix between the target layer and the next layer of the target layer based on a preset inter-layer similarity algorithm and the transformed feature vector;

[0126] The fusion module 53 is used to determine the similarity layer in the similarity matrix and fuse the similarity layer according to the preset fusion parameters to obtain the fused similar layer, and to obtain the compressed model based on each fused similar layer.

[0127] In some embodiments, the processing module 51 is configured to construct a kernel matrix based on the input data, the target layer, and the activation function and kernel bandwidth parameter of the next layer after the target layer, and to construct a degree matrix based on the elements of the kernel matrix;

[0128] The difference between the degree matrix and the kernel matrix is ​​determined as the Laplace matrix;

[0129] Determine the eigenvectors corresponding to the target eigenvalues ​​of the Laplacian matrix;

[0130] The product of the eigenvector and the Laplacian matrix is ​​used to determine the transformed eigenvector.

[0131] In some embodiments, the construction module 52 is used to calculate the joint probability distribution function of the target layer and the next layer below the target layer, and to calculate the edge probability distribution function of the target layer and the edge probability distribution function of the next layer below the target layer;

[0132] Based on the joint probability distribution function, the marginal probability distribution function of the target layer, and the marginal probability distribution function of the next layer of the target layer, a similarity matrix between the target layer and the next layer of the target layer is constructed.

[0133] In some embodiments, the processing module 51 is further configured to determine the sum of the multi-head attention value, feedforward value, and layer normalization value of the activation function of the layer preceding the target layer as the activation function of the target layer.

[0134] In some embodiments, the fusion module 53 is further configured to calculate the performance parameters of the compressed model;

[0135] If the performance parameters do not meet the preset performance conditions, continue to execute the steps of constructing the similarity matrix of the target layer and the next layer of the target layer based on the preset inter-layer similarity algorithm and the transformed feature vector, until the performance parameters meet the preset performance conditions.

[0136] In some embodiments, the fusion module 53 is further configured to construct an initial inference tree of the model based on a preset inference tree inference algorithm, wherein the initial inference tree includes at least one target node;

[0137] For each target node, calculate the product of the confidence scores of all nodes included in the path from the root node to the target node, and use this product as the node value function of the target node. Based on the node value function, the depth value of the target node, and the hyperparameters for adjusting the depth effect, determine the modified node value function of the target node. Rearrange the path from the root node to the target node based on the modified node value function to obtain the rearranged path.

[0138] Based on the rearranged paths corresponding to each target node, the target inference tree is obtained.

[0139] Each module in the aforementioned model device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0140] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram is shown in Figure 6. The computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a model processing method. The display unit of the computer device is used to form a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0141] Those skilled in the art will understand that the structure shown in Figure 6 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0143] Based on the diffusion kernel algorithm, the input data, the target layer and the activation function of the next layer of the target layer are transformed into low-dimensional features to obtain the transformed feature vector. The model includes the target layer and the next layer of the target layer.

[0144] Based on the preset inter-layer similarity algorithm and the transformed feature vector, a similarity matrix is ​​constructed between the target layer and the next layer of the target layer;

[0145] The similarity layers are determined in the similarity matrix and then fused according to the preset fusion parameters to obtain the fused similarity layers. Based on each fused similarity layer, the compressed model is obtained.

[0146] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0147] A kernel matrix is ​​constructed based on the input data, the target layer, the activation function of the next layer after the target layer, and the kernel bandwidth parameter. A degree matrix is ​​then constructed based on the elements of the kernel matrix.

[0148] The difference between the degree matrix and the kernel matrix is ​​determined as the Laplace matrix;

[0149] Determine the eigenvectors corresponding to the target eigenvalues ​​of the Laplacian matrix;

[0150] The product of the eigenvector and the Laplacian matrix is ​​used to determine the transformed eigenvector.

[0151] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0152] Calculate the joint probability distribution function of the target layer and the layer below the target layer, as well as the marginal probability distribution function of the target layer and the marginal probability distribution function of the layer below the target layer;

[0153] Based on the joint probability distribution function, the marginal probability distribution function of the target layer, and the marginal probability distribution function of the next layer of the target layer, a similarity matrix between the target layer and the next layer of the target layer is constructed.

[0154] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0155] The sum of the multi-head attention value, feedforward value, and layer normalization value of the activation function of the layer preceding the target layer is determined as the activation function of the target layer.

[0156] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0157] Calculate the performance parameters of the compressed model;

[0158] If the performance parameters do not meet the preset performance conditions, continue to execute the steps of constructing the similarity matrix of the target layer and the next layer of the target layer based on the preset inter-layer similarity algorithm and the transformed feature vector, until the performance parameters meet the preset performance conditions.

[0159] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0160] The initial inference tree of the model is constructed based on a preset inference tree inference algorithm, and the initial inference tree includes at least one target node;

[0161] For each target node, calculate the product of the confidence scores of all nodes included in the path from the root node to the target node, and use this product as the node value function of the target node. Based on the node value function, the depth value of the target node, and the hyperparameters for adjusting the depth effect, determine the modified node value function of the target node. Rearrange the path from the root node to the target node based on the modified node value function to obtain the rearranged path.

[0162] Based on the rearranged paths corresponding to each target node, the target inference tree is obtained.

[0163] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0164] Based on the diffusion kernel algorithm, the input data, the target layer and the activation function of the next layer of the target layer are transformed into low-dimensional features to obtain the transformed feature vector. The model includes the target layer and the next layer of the target layer.

[0165] Based on the preset inter-layer similarity algorithm and the transformed feature vector, a similarity matrix is ​​constructed between the target layer and the next layer of the target layer;

[0166] The similarity layers are determined in the similarity matrix and then fused according to the preset fusion parameters to obtain the fused similarity layers. Based on each fused similarity layer, the compressed model is obtained.

[0167] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0168] Based on the diffusion kernel algorithm, the input data, the target layer and the activation function of the next layer of the target layer are transformed into low-dimensional features to obtain the transformed feature vector. The model includes the target layer and the next layer of the target layer.

[0169] Based on the preset inter-layer similarity algorithm and the transformed feature vector, a similarity matrix is ​​constructed between the target layer and the next layer of the target layer;

[0170] The similarity layers are determined in the similarity matrix and then fused according to the preset fusion parameters to obtain the fused similarity layers. Based on each fused similarity layer, the compressed model is obtained.

[0171] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0172] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0173] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A model processing method, wherein, The method includes: Based on the diffusion kernel algorithm, the input data, the target layer, and the activation function of the next layer of the target layer are subjected to low-dimensional transformation to obtain the transformed feature vector. The model includes the target layer and the next layer of the target layer. Based on the preset inter-layer similarity algorithm and the transformed feature vector, a similarity matrix is ​​constructed between the target layer and the next layer of the target layer; A similarity layer is determined in the similarity matrix, and the similarity layer is fused according to a preset fusion parameter to obtain a fused similarity layer. Based on each of the fused similarity layers, a compressed model is obtained.

2. The method according to claim 1, wherein, The low-dimensional transformation based on the diffusion kernel algorithm and the activation function of the input data, the target layer, and the next layer after the target layer yields the transformed feature vector, including: A kernel matrix is ​​constructed based on the input data, the target layer, the activation function of the next layer after the target layer, and the kernel bandwidth parameter. A degree matrix is ​​then constructed based on the elements of the kernel matrix. The difference between the degree matrix and the kernel matrix is ​​determined as the Laplace matrix; Determine the eigenvectors corresponding to the target eigenvalues ​​of the Laplacian matrix; The product of the eigenvector and the Laplacian matrix is ​​determined as the transformed eigenvector.

3. The method according to claim 1, wherein, The step of constructing a similarity matrix between the target layer and the next layer of the target layer based on a preset inter-layer similarity algorithm and the transformed feature vector includes: Calculate the joint probability distribution function of the target layer and the next layer below the target layer, and calculate the edge probability distribution function of the target layer and the edge probability distribution function of the next layer below the target layer; Based on the joint probability distribution function, the edge probability distribution function of the target layer, and the edge probability distribution function of the next layer of the target layer, a similarity matrix between the target layer and the next layer of the target layer is constructed.

4. The method according to claim 1, wherein, The method further includes: The sum of the multi-head attention value, feedforward value, and layer normalization value of the activation function of the layer preceding the target layer is determined as the activation function of the target layer.

5. The method according to claim 1, wherein, The method further includes: Calculate the performance parameters of the compressed model; If the performance parameters do not meet the preset performance conditions, the step of constructing the similarity matrix of the target layer and the next layer of the target layer based on the preset inter-layer similarity algorithm and the transformed feature vector continues until the performance parameters meet the preset performance conditions.

6. The method according to claim 1, wherein, The method further includes: An initial inference tree for the model is constructed based on a preset inference tree inference algorithm, and the initial inference tree includes at least one target node; For each target node, calculate the product of the confidence scores of all nodes along the path from the root node to the target node, and determine the product as the node value function of the target node; based on the node value function, the depth value of the target node, and the hyperparameter for adjusting the depth effect, determine the modified node value function of the target node; rearrange the path from the root node to the target node based on the modified node value function to obtain the rearranged path; Based on the rearranged paths corresponding to each of the target nodes, a target inference tree is obtained.

7. A model processing apparatus, wherein, The device includes: The processing module is used to perform low-dimensional transformation processing on the input data, the target layer, and the activation function of the next layer of the target layer based on the diffusion kernel algorithm to obtain the transformed feature vector. The model includes the target layer and the next layer of the target layer. A construction module is used to construct a similarity matrix between the target layer and the next layer of the target layer based on a preset inter-layer similarity algorithm and the transformed feature vector; The fusion module is used to determine the similarity layer in the similarity matrix, and fuse the similarity layer according to the preset fusion parameters to obtain the fused similar layer, and to obtain the compressed model based on each of the fused similar layers.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein... When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images