Model inference verification method and apparatus, electronic device, and storage medium
By combining multi-head and tree-like attention mechanisms with dynamic pruning, the problem of low efficiency in large model inference is solved, achieving faster and more efficient inference verification.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- CHINA TELECOM ARTIFICIAL INTELLIGENCE TECHNOLOGY (BEIJING) CO LTD
- Filing Date
- 2025-09-15
- Publication Date
- 2026-05-07
AI Technical Summary
In existing large-scale model inference processes, serial decoding is inefficient, especially in long-text inference, and it is difficult to meet the requirements. Although methods such as quantization or distillation can improve speed, they lead to a decrease in performance.
A multi-head attention mechanism network is used for parallel decoding, combined with a tree-like attention mechanism network for tree-like inference decoding, and a dynamic pruning mechanism is introduced to reduce the number of model validations.
It improves the speed and efficiency of model inference, enhances the model inference effect, and enables more efficient model inference verification.
Smart Images
Figure CN2025121350_07052026_PF_FP_ABST
Abstract
Description
Model inference verification methods, devices, electronic equipment and storage media
[0001] This application claims priority to Chinese Patent Application No. 2024115155598, filed on October 28, 2024, entitled “A Model Reasoning Verification Method, Apparatus, Electronic Device and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of large model application technology, and in particular to a model reasoning verification method, device, electronic device and storage medium. Background Technology
[0003] Currently, large-scale model inference decoding is mostly one-way decoding, meaning that the current decoding depends on the result of the previous decoding, which is a serial structure. This greatly reduces decoding efficiency, especially when it comes to long text inference, where a single request often needs to generate articles of hundreds or thousands of words. Traditional serial decoding can no longer meet this requirement.
[0004] In large model inference, methods such as quantization and distillation are often used to reduce the model size, thereby increasing the inference speed. However, this can lead to a decrease in the inference performance. For example, methods such as flash attention, which use fusion operators to alleviate I / O, achieve model inference verification by sacrificing some of the softmax precision. On the other hand, methods such as key-value caching and page attention trade off between caching and read speed, reducing redundancy in the inference process by sacrificing caching, thereby accelerating the inference speed of the model. Summary of the Invention
[0005] The main objective of this application is to provide a model reasoning verification method, apparatus, electronic device, and storage medium that can realize model reasoning verification, improve model reasoning speed, enhance model reasoning verification efficiency, and improve model reasoning performance.
[0006] On the one hand, embodiments of this application propose a model inference verification method, the method comprising the following steps:
[0007] Get the text prompt words;
[0008] A large model is used to perform dynamic pruning inference on the text prompts to generate multiple candidate text sequences;
[0009] Based on each of the candidate text sequences, construct a corresponding candidate sequence combination tree;
[0010] The large model is used to perform tree-based decoding on the candidate sequence combination tree to determine the tree-based decoding output sequence;
[0011] Based on the tree-structured decoding output sequence, consistency verification is performed on each of the candidate text sequences to determine the optimal text sequence from among the multiple candidate text sequences.
[0012] In some embodiments, the step of using a large model to perform dynamic pruning inference on the text prompt words to generate multiple candidate text sequences includes:
[0013] The text prompt words are divided into text characters to determine the position of each text character and the unit text corresponding to each text character position;
[0014] The unit text corresponding to each of the text character positions is input into the multi-head attention mechanism network contained in the large model, and the multiple inference texts corresponding to each of the unit texts and the cumulative attention probability corresponding to each of the inference texts are output.
[0015] Based on the attention accumulation probability corresponding to each of the unit texts, the sampling threshold corresponding to each of the unit texts is dynamically determined, and multiple candidate inference texts corresponding to the sampling thresholds are determined from multiple inference texts;
[0016] Based on the multiple candidate inference texts corresponding to each unit text and the positions of the text characters, the texts are arranged and combined to generate multiple candidate text sequences.
[0017] In some embodiments, establishing a corresponding candidate sequence combination tree based on each of the candidate text sequences includes:
[0018] Construct the candidate sequence combination tree;
[0019] Based on the position of the text character corresponding to each unit text, determine the node level of each unit text in the candidate sequence combination tree;
[0020] Based on the node level corresponding to each unit text, determine the node position of each candidate inference text corresponding to each unit text in the candidate sequence combination tree, and fill each candidate inference text into the corresponding node position;
[0021] Based on each candidate text sequence, determine the textual association relationship between each candidate inference text, and establish association markers in the candidate sequence combination tree based on the textual association relationship.
[0022] In some embodiments, the step of using the large model to perform tree-based decoding on the candidate sequence combination tree to determine the tree-based decoding output sequence includes:
[0023] The candidate sequence combination tree is input into the tree-like attention mechanism network in the large model for decoding, and multiple target inference texts and the sequence positions corresponding to each target inference text are output.
[0024] Based on the sequence position of each target inference text, the target inference texts are concatenated to generate the tree-shaped decoding output sequence.
[0025] In some embodiments, the step of performing consistency verification on each of the candidate text sequences based on the tree-structured decoding output sequence, and determining the optimal text sequence from the plurality of candidate text sequences, includes:
[0026] For each of the candidate text sequences, a consistent text sequence that matches the tree-decoding output sequence is determined from the candidate text sequences, and the text length corresponding to the consistent text sequence is determined. The consistent text sequence is the sequence corresponding to the text overlap between the candidate text sequence and the tree-decoding output sequence.
[0027] Based on the text length corresponding to each of the consistent text sequences, a target consistent text sequence is determined from each of the consistent text sequences, wherein the text length corresponding to the target consistent text sequence is the longest;
[0028] The candidate text sequence to which the target consistent text sequence belongs is determined as the optimal text sequence.
[0029] In some embodiments, dynamically determining a sampling threshold corresponding to each of the unit texts based on the attention accumulation probability corresponding to each of the inference texts corresponding to each of the unit texts, and determining multiple candidate inference texts corresponding to the sampling thresholds from multiple inference texts, includes:
[0030] Obtain the preset attention accumulation probability threshold;
[0031] Based on the attention accumulation probability corresponding to each of the inference texts corresponding to each of the unit texts, a number of target texts are determined from the multiple inference texts, wherein the attention accumulation probability corresponding to the target texts is greater than the attention accumulation probability threshold;
[0032] The number of target texts is counted, and the sampling threshold is dynamically set based on the number of target texts, wherein the sampling threshold is less than or equal to the number of target texts;
[0033] Based on the attention accumulation probability corresponding to each target text, the attention accumulation probability is sorted, and several candidate inference texts are determined from the multiple target texts, wherein the number of candidate inference texts corresponds to the sampling threshold.
[0034] In some embodiments, the step of arranging and combining texts according to the multiple candidate inference texts corresponding to each unit text and the text character positions to generate multiple candidate text sequences includes:
[0035] Based on the text character positions corresponding to each of the aforementioned unit texts, determine the candidate text sequence positions corresponding to each of the aforementioned unit texts;
[0036] Each candidate inference text corresponding to each unit text is sequentially filled into the candidate text sequence position corresponding to each unit text in a permutation and combination form to generate each candidate text sequence.
[0037] In some embodiments, the text character segmentation of the text prompt word includes:
[0038] The text prompt words are divided using the N-gram model to form a sequence of byte segments of length N, where each byte segment serves as the unit text.
[0039] In some embodiments, the tree-like attention mechanism network organizes sequential data into a tree structure, such that each element in the network only pays attention to other elements within a certain range of its neighbors.
[0040] In some embodiments, the method further includes:
[0041] After determining the optimal text sequence, the large model is updated using the model inference parameters corresponding to the optimal text sequence.
[0042] On the other hand, embodiments of this application propose a model inference verification device, the device comprising:
[0043] The first module is configured to retrieve text prompts;
[0044] The second module is configured to use a large model to perform dynamic pruning inference on the text prompt words and generate multiple candidate text sequences.
[0045] The third module is configured to build a corresponding candidate sequence combination tree based on each of the candidate text sequences.
[0046] The fourth module is configured to use the large model to perform tree-based decoding on the candidate sequence combination tree to determine the tree-based decoding output sequence;
[0047] The fifth module is configured to perform consistency verification on each of the candidate text sequences based on the tree-structured decoding output sequence, and determine the optimal text sequence from the multiple candidate text sequences.
[0048] On the other hand, embodiments of this application propose an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the model inference verification method described above.
[0049] On the other hand, embodiments of this application propose a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned model inference verification method.
[0050] The embodiments of this application include at least the following beneficial effects: The model inference verification method, apparatus, electronic device, and storage medium provided in this application acquire text prompt words, use a large model to perform dynamic pruning inference on the text prompt words to generate multiple candidate text sequences, establish a corresponding candidate sequence combination tree based on each candidate text sequence, use the large model to perform tree-shaped decoding on the candidate sequence combination tree, determine the tree-shaped decoding output sequence, perform consistency verification on each candidate text sequence based on the tree-shaped decoding output sequence, and determine the optimal text sequence from multiple candidate text sequences. This application can realize model inference verification, introduce dynamic pruning inference, reduce the number of model verifications, improve model inference verification efficiency, enhance model inference performance, and increase model inference speed. Attached Figure Description
[0051] Figure 1 is a flowchart of a model reasoning verification method provided in an embodiment of this application;
[0052] Figure 2 is a flowchart of step S102 in an embodiment of this application;
[0053] Figure 3 is a schematic diagram of text character segmentation for text prompt words in an embodiment of this application;
[0054] Figure 4 is a flowchart of step S203 in an embodiment of this application;
[0055] Figure 5 is a flowchart of step S204 in an embodiment of this application;
[0056] Figure 6 is a flowchart of step S103 in an embodiment of this application;
[0057] Figure 7 is a flowchart of step S104 in an embodiment of this application;
[0058] Figure 8 is a flowchart of step S105 in an embodiment of this application;
[0059] Figure 9 is a schematic diagram of the structure of a model reasoning verification device provided in an embodiment of this application;
[0060] Figure 10 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.
[0062] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”
[0063] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.
[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0065] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.
[0066] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.
[0067] 1) Dynamic Pruning: During the decoding process of a large model, the logit needs to be sampled and decoded. If the number of tokens truncated each time is fixed, it is static pruning; if it varies according to specific circumstances, it is dynamic pruning. This patent adjusts based on the cumulative probability distribution value, therefore it is dynamic pruning.
[0068] 2) Logit: Logit is an important concept in deep learning. It represents the raw numerical values of the model output before they have been processed by the softmax function. These raw values may be interpreted as the model's confidence scores for each possible output class, but they are not necessarily normalized or probabilistic. During training, the model typically uses logits as input to the loss function, then transforms them into a probability distribution using the softmax function, which is ultimately used to calculate the loss. During the testing or inference phase, the softmax function is typically used to transform the logits into a probability distribution so that predictions can be made based on the class with the highest score.
[0069] 3) Token: This refers to breaking down input text into smaller units, such as words, letters, or characters. In natural language processing, breaking text down into tokens helps the model understand semantic and syntactic structure. When a Prompt is sent to a large model, it is broken down into multiple tokens; this process is called Tokenier. Generally, for an English word, four characters represent one token.
[0070] 4) Top-k Sampling: Top-k sampling is a commonly used decoding strategy in natural language generation tasks, aiming to balance the diversity and accuracy of the generated text. In each decoding step, the model selects the k most likely words and then randomly selects one of these words as the next word to be generated.
[0071] 5) Prompt: This refers to a text prompt, which is a piece of text input to the model. It is designed to guide the model to generate output with specific content or following a specific context. It can help AI models generate articles, scripts, and other content.
[0072] 6) N-gram model: The N-gram model is an algorithm based on statistical language models. Its basic idea is to perform a sliding window operation of size N on the text content according to bytes, forming a sequence of byte segments of length N. Each byte segment is called a gram. The frequency of occurrence of all grams is counted and filtered according to a set threshold to form a list of key grams, which is the vector feature space of the text.
[0073] 7) Vocabulary size: This refers to the size of the training set vocabulary. In Natural Language Processing (NLP), vocabulary size typically represents the number of distinct words or tokens in the training set. During training, the model learns the embedding representations of these words so that it can recognize and utilize them when processing new text.
[0074] 8) `head_logits`: In deep learning, this typically refers to the raw scores or ratings of the model's output layer, especially in classification problems. These scores are the model's linear output for each class, before being processed by activation functions such as softmax or sigmoid. In deep learning frameworks, `head_logits` is usually represented as a tensor or array, where each element corresponds to the logit value for a class. For example, when dealing with a classification problem with 10 classes, the network's `head_logits` output will be an array containing 10 elements.
[0075] 9) Softmax: The softmax function is mainly used in the output layer of multi-class classification problems in large models. It is essentially a normalization exponential function used to convert a set of arbitrary real numbers into real numbers representing a probability distribution. The softmax function maps the output values of the output nodes to the range [0,1] and guarantees that the sum of all output values is 1, which makes it possible to interpret it as a probability distribution.
[0076] Currently, large-scale model inference decoding is mostly one-way decoding, meaning that the current decoding depends on the result of the previous decoding, which is a serial structure. This greatly reduces decoding efficiency, especially when it comes to long text inference, where a single request often needs to generate articles of hundreds or thousands of words. Traditional serial decoding can no longer meet this requirement.
[0077] In large model inference, methods such as quantization and distillation are often used to reduce the model size, thereby increasing the inference speed. However, this can lead to a decrease in the inference performance. For example, methods such as flash attention, which use fusion operators to alleviate I / O, achieve model inference verification by sacrificing some of the softmax precision. On the other hand, methods such as key-value caching and page attention trade off between caching and read speed, reducing redundancy in the inference process by sacrificing caching, thereby accelerating the inference speed of the model.
[0078] Based on this, embodiments of this application propose a model inference verification method, apparatus, electronic device, and storage medium. It employs a multi-head attention mechanism network for parallel decoding and a tree-structured attention mechanism network for tree-structured inference decoding to improve the inference speed of large models. It also introduces a dynamic pruning mechanism to reduce the number of result verifications for large models, thereby improving the efficiency of model inference verification, enhancing the model inference effect, and realizing model inference verification.
[0079] Referring to Figure 1, which is an optional flowchart of a model inference verification method provided in an embodiment of this application, the method may include, but is not limited to, steps S101 to S105:
[0080] Step S101: Obtain text prompts;
[0081] Step S102: Use a large model to perform dynamic pruning inference on the text prompt words to generate multiple candidate text sequences;
[0082] Step S103: Based on each candidate text sequence, establish the corresponding candidate sequence combination tree;
[0083] Step S104: Use the large model to perform tree-based decoding on the candidate sequence combination tree to determine the tree-based decoding output sequence;
[0084] Step S105: Based on the tree-structured decoding output sequence, perform consistency verification on each candidate text sequence and determine the optimal text sequence from multiple candidate text sequences.
[0085] In some embodiments, the text prompt is called a Prompt. The Prompt can be input into a large model, which performs inference and verification based on the Prompt to generate output with specific content or following a specific context, i.e., the optimal text sequence mentioned above.
[0086] In some embodiments, referring to FIG2, which is an optional flowchart of step S102 in an embodiment of the present application, step S102 may include, but is not limited to, steps S201 to S204:
[0087] Step S201: Divide the text prompt words into text characters, determine the position of each text character and the unit text corresponding to each text character position;
[0088] Step S202: Input the unit text corresponding to each text character position into the multi-head attention mechanism network contained in the large model, and output the multiple inference texts corresponding to each unit text and the cumulative attention probability corresponding to each inference text;
[0089] Step S203: Based on the attention accumulation probability corresponding to each inference text corresponding to each unit text, dynamically determine the sampling threshold corresponding to each unit text, and determine multiple candidate inference texts corresponding to the sampling threshold from multiple inference texts;
[0090] Step S204: Based on the multiple candidate inference texts and text character positions corresponding to each unit text, perform text permutation and combination to generate multiple candidate text sequences.
[0091] In some embodiments, the Prompt is divided into multiple tokens (such as the unit text mentioned above), and each token has a fixed position in the Prompt (such as the text character position mentioned above). Referring to Figure 3, which is an optional schematic diagram of text character division for text prompts in an embodiment of this application, the Prompt is divided into multiple tokens, generating token1 to token5 and text character positions 1 to 5. The text character position corresponding to token1 is 1, the text character position corresponding to token2 is 2, and so on.
[0092] Alternatively, the Prompt can be divided using an N-gram model to form a sequence of byte segments of length N, with each byte segment being a token.
[0093] In some embodiments, a multi-head attention mechanism network is used to process each unit text in parallel to improve the inference speed of the model. Optionally, the multi-head attention mechanism network includes multiple self-attention structures, each of which corresponds one-to-one with the position of each text character. These structures are used to obtain the unit text at the corresponding text character position, perform inference and prediction on the unit text, and output the corresponding prediction result. The prediction result includes multiple inferred texts and the cumulative attention probability corresponding to each inferred text. Optionally, tokens at different text character positions are input to their respective attention structures, and inference and prediction are performed using the respective attention structures. The output includes multiple next tokens (such as the inferred texts mentioned above) corresponding to each token and the logit (such as the cumulative attention probability mentioned above) corresponding to each next token.
[0094] The multi-head attention mechanism network is fine-tuned by adjusting the loss function to improve its inference and prediction performance. The loss function is as follows:
[0095] Where K is the number of self-attention structures, k is the kth self-attention structure, λ is the constraint weight, which is the constraint on the value of K in the multi-head attention mechanism network. Referring to the regularization constraint, it is a penalty for excessively large K values. t is the text character position corresponding to the unit text processed by the kth self-attention structure, and L is the cross-entropy loss of the probability p of the text prediction value y of the associated text character position t+k+1 when using the kth self-attention structure to perform inference prediction on the unit text at text character position t.
[0096] In some embodiments, referring to FIG4, which is an optional flowchart of step S203 in an embodiment of the present application, step S203 may include, but is not limited to, steps S301 to S304:
[0097] Step S301: Obtain the preset attention accumulation probability threshold;
[0098] Step S302: Based on the attention accumulation probability corresponding to each inference text corresponding to each unit text, determine several target texts from multiple inference texts, wherein the attention accumulation probability corresponding to the target text is greater than the attention accumulation probability threshold.
[0099] Step S303: Count the number of target texts, and dynamically set a sampling threshold based on the number of target texts. The sampling threshold is less than or equal to the number of target texts.
[0100] Step S304: Sort the attention accumulation probabilities according to the attention accumulation probabilities corresponding to each target text, and determine a number of candidate inference texts from multiple target texts, wherein the number of candidate inference texts corresponds to the sampling threshold.
[0101] In some embodiments, a multi-head attention mechanism network is used to perform inference prediction on each unit text to obtain multiple inference texts corresponding to each unit text. Based on the cumulative attention probability corresponding to each inference text, Top-k sampling is performed, that is, the Top k candidate inference texts are extracted from the multiple inference texts, where k is the sampling threshold mentioned above. The k value can be dynamically set to achieve dynamic pruning. Specifically, based on the cumulative attention probability corresponding to each inference text, multiple target texts with a cumulative attention probability greater than the cumulative attention probability threshold are determined from the multiple inference texts. The k value is dynamically set according to the number of target texts. If the number of target texts is too large, the vocab size of the large model can be obtained, and the k value is set according to the vocab size, such as setting the k value to 60% or 50% of the vocab size, which can be set by the user. If the number of target texts is small, the number of target texts can be directly taken as the k value. Then, based on the sampling threshold k corresponding to the unit text, multiple candidate inference texts corresponding to the sampling threshold are collected.
[0102] In some embodiments, referring to FIG5, FIG5 is an optional flowchart of step S204 in an embodiment of the present application. Step S204 may include, but is not limited to, steps S401 to S402:
[0103] Step S401: Determine the candidate text sequence position corresponding to each unit text based on the text character position corresponding to each unit text.
[0104] Step S402: Fill the candidate reasoning texts corresponding to each unit text into the candidate text sequence positions corresponding to each unit text in a permutation and combination form to generate each candidate text sequence.
[0105] In some embodiments, multiple candidate text sequences are generated based on multiple candidate inference texts of each unit text and their corresponding text character positions. The number of candidate text sequences is: k1*k2*k3*...*k n , where k nLet n be the number of candidate inference texts corresponding to a unit text at text character position n. For example, assuming there are token1 to token3, the candidate inference texts corresponding to token1 include token11 and token12, the candidate inference texts corresponding to token2 include token21, and the candidate inference texts corresponding to token3 include token31 and token32. Then the generated candidate text sequence includes: token11token21token31, token11token21token32, token12token21token31, and token12token21token32.
[0106] In some embodiments, referring to FIG6, which is an optional flowchart of step S103 in an embodiment of the present application, step S103 may include, but is not limited to, steps S501 to S504:
[0107] Step S501: Construct a candidate sequence combination tree;
[0108] Step S502: Determine the node level of each unit text in the candidate sequence combination tree based on the text character position corresponding to each unit text.
[0109] Step S503: Based on the node level corresponding to each unit text, determine the node position of each candidate inference text in the candidate sequence combination tree corresponding to each unit text, and fill each candidate inference text into the corresponding node position.
[0110] Step S504: Based on each candidate text sequence, determine the textual association relationship between each candidate inference text, and establish association markers in the candidate sequence combination tree based on the textual association relationship.
[0111] In some embodiments, a candidate sequence combination tree is constructed based on each candidate text sequence. According to the node level corresponding to each unit text, the unit text with the earlier text character position has a higher node level. The node level corresponding to the unit text is determined as the node level of each candidate inference text. At this node level, the node position of each candidate inference text is determined. Each candidate inference text is filled into the corresponding node position to realize the construction of the candidate sequence combination tree. Using this candidate sequence combination tree, multiple candidate text sequences can be verified at once, which greatly improves the efficiency of model inference verification.
[0112] In some embodiments, referring to FIG7, which is an optional flowchart of step S104 in an embodiment of the present application, step S104 may include, but is not limited to, steps S601 to S602:
[0113] Step S601: Input the candidate sequence combination tree into the tree-like attention mechanism network in the large model, decode it, and output multiple target inference texts and the sequence positions corresponding to each target inference text;
[0114] Step S602: Based on the sequence position of each target inference text, the target inference texts are spliced together to generate a tree-shaped decoding output sequence.
[0115] In some embodiments, a tree attention mechanism network is used to decode the candidate sequence combination tree. Tree attention is an attention mechanism that can efficiently capture information when dealing with tasks with complex data structures. Its core idea is to reduce computational complexity and improve efficiency by limiting the scope of attention for each element. Tree attention organizes the sequence data into a tree structure, so that each element only pays attention to other elements within a certain range of its neighbors. This mechanism makes tree attention perform well when processing long sequence data, effectively capturing local information while reducing unnecessary computation.
[0116] Tree attention skips steps based on a tree-structured decoding method. For example, suppose there is a candidate text sequence "I am a teacher a student...". When decoding "I am", two candidate branches "I am a teacher" and "I am a student" are identified. When decoding "I am a", if the main model predicts "student", it can skip "teacher" and go directly to "student" to continue decoding. This reduces the number of decoding steps. Here, "I", "am", "a", and "student" are all target inference texts. Multiple target inference texts are concatenated to generate a tree-structured decoding output sequence, which is then used as the output of the main model.
[0117] In some embodiments, referring to FIG8, which is an optional flowchart of step S105 in an embodiment of the present application, step S105 may include, but is not limited to, steps S701 to S703:
[0118] Step S701: For each candidate text sequence, determine the consistent text sequence that matches the tree-decoding output sequence from the candidate text sequence, and determine the text length corresponding to the consistent text sequence, wherein the consistent text sequence is the sequence corresponding to the text overlap between the candidate text sequence and the tree-decoding output sequence;
[0119] Step S702: Based on the text length corresponding to each consistent text sequence, determine the target consistent text sequence from each consistent text sequence, wherein the text length corresponding to the target consistent text sequence is the longest;
[0120] Step S703: The candidate text sequence to which the target consistent text sequence belongs is determined as the optimal text sequence.
[0121] In some embodiments, each candidate text sequence is verified based on the tree-shaped decoding output sequence generated by the tree attention mechanism network, and the sequence with the longest consistent text length that overlaps with the tree-shaped decoding output sequence is determined from each candidate text sequence, which is the target consistent text sequence mentioned above.
[0122] The target consistent text sequence is determined as the optimal text sequence. Then, the model is updated using the relevant model inference parameters of the optimal text sequence. For example, the head_logits corresponding to the optimal text sequence are determined, and the corresponding key and value in the multi-head self-attention mechanism network of the large model are updated. The association between key, value and head_logits is a cache mechanism in the multi-head self-attention mechanism. In order to improve computational efficiency, the results of the self-attention layer calculated earlier can be cached and reused when processing subsequent parts. The result cache of the self-attention layer usually includes the following parts: key: used to look up the corresponding values; value: stores the actual data; head_logits: weights and scores in the self-attention calculation.
[0123] Referring to Figure 9, which is an optional structural schematic diagram of a model inference verification device provided in an embodiment of this application, the device can be used to implement the above-described model inference verification method and may include:
[0124] The first module is configured to retrieve text prompts;
[0125] The second module is configured to use a large model to perform dynamic pruning inference on text prompts and generate multiple candidate text sequences.
[0126] The third module is configured to build a corresponding candidate sequence combination tree based on each candidate text sequence;
[0127] The fourth module is configured to use a large model to perform tree-based decoding on the candidate sequence combination tree and determine the tree-based decoding output sequence.
[0128] The fifth module is configured to perform consistency verification on each candidate text sequence based on the tree-structured decoding output sequence, and determine the optimal text sequence from multiple candidate text sequences.
[0129] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0130] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described model reasoning and verification method. This electronic device can be any smart terminal, including a tablet computer.
[0131] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0132] Please refer to Figure 10, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes:
[0133] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.
[0134] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and called by the processor 901 to execute the model reasoning verification method of the embodiments of this application.
[0135] Input / output interface 903 is configured to enable information input and output;
[0136] Communication interface 904 is configured to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0137] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);
[0138] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.
[0139] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described model inference verification method.
[0140] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0141] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0142] This application provides a model inference verification method, apparatus, electronic device, and storage medium. It acquires text prompts, uses a large model to perform dynamic pruning inference on the prompts, generates multiple candidate text sequences, establishes a corresponding candidate sequence combination tree based on each candidate text sequence, performs tree-based decoding on the candidate sequence combination tree using the large model, determines the tree-based decoding output sequence, performs consistency verification on each candidate text sequence based on the tree-based decoding output sequence, and determines the optimal text sequence from the multiple candidate text sequences. This application enables model inference verification by introducing dynamic pruning inference, reducing the number of model verifications, improving model inference verification efficiency, enhancing model inference performance, and increasing model inference speed.
[0143] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0144] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0145] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0146] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0147] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0148] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0149] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0150] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0151] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0152] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The method can be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium is configured such that the computer operates in a specific and predefined manner—according to the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).
[0153] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0154] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A model inference verification method, comprising: Get the text prompt words; A large model is used to perform dynamic pruning inference on the text prompts to generate multiple candidate text sequences; Based on each of the candidate text sequences, construct a corresponding candidate sequence combination tree; The large model is used to perform tree-based decoding on the candidate sequence combination tree to determine the tree-based decoding output sequence; Based on the tree-structured decoding output sequence, consistency verification is performed on each of the candidate text sequences to determine the optimal text sequence from among the multiple candidate text sequences.
2. The model reasoning verification method according to claim 1, wherein, The method utilizes a large model to perform dynamic pruning and inference on the text prompts, generating multiple candidate text sequences, including: The text prompt words are divided into text characters to determine the position of each text character and the unit text corresponding to each text character position; The unit text corresponding to each of the text character positions is input into the multi-head attention mechanism network contained in the large model, and the multiple inference texts corresponding to each of the unit texts and the cumulative attention probability corresponding to each of the inference texts are output. Based on the attention accumulation probability corresponding to each of the unit texts, the sampling threshold corresponding to each of the unit texts is dynamically determined, and multiple candidate inference texts corresponding to the sampling thresholds are determined from multiple inference texts; Based on the multiple candidate inference texts corresponding to each unit text and the positions of the text characters, the texts are arranged and combined to generate multiple candidate text sequences.
3. The model reasoning verification method according to claim 2, wherein, The step of establishing a corresponding candidate sequence combination tree based on each of the candidate text sequences includes: Construct the candidate sequence combination tree; Based on the position of the text character corresponding to each unit text, determine the node level of each unit text in the candidate sequence combination tree; Based on the node level corresponding to each unit text, determine the node position of each candidate inference text corresponding to each unit text in the candidate sequence combination tree, and fill each candidate inference text into the corresponding node position; Based on each candidate text sequence, determine the textual association relationship between each candidate inference text, and establish association markers in the candidate sequence combination tree based on the textual association relationship.
4. The model reasoning verification method according to claim 3, wherein, The step of using the large model to perform tree-based decoding on the candidate sequence combination tree to determine the tree-based decoding output sequence includes: The candidate sequence combination tree is input into the tree-like attention mechanism network in the large model for decoding, and multiple target inference texts and the sequence positions corresponding to each target inference text are output. Based on the sequence position of each target inference text, the target inference texts are concatenated to generate the tree-shaped decoding output sequence.
5. The model inference verification method according to claim 1, wherein, The step of performing consistency verification on each of the candidate text sequences based on the tree-structured decoding output sequence, and determining the optimal text sequence from the multiple candidate text sequences, includes: For each of the candidate text sequences, a consistent text sequence that matches the tree-decoding output sequence is determined from the candidate text sequences, and the text length corresponding to the consistent text sequence is determined. The consistent text sequence is the sequence corresponding to the text overlap between the candidate text sequence and the tree-decoding output sequence. Based on the text length corresponding to each of the consistent text sequences, a target consistent text sequence is determined from each of the consistent text sequences, wherein the text length corresponding to the target consistent text sequence is the longest; The candidate text sequence to which the target consistent text sequence belongs is determined as the optimal text sequence.
6. The model inference verification method according to claim 2, wherein, The step of dynamically determining a sampling threshold corresponding to each of the unit texts based on the attention accumulation probability corresponding to each of the inference texts, and determining multiple candidate inference texts corresponding to the sampling thresholds from multiple inference texts, includes: Obtain the preset attention accumulation probability threshold; Based on the attention accumulation probability corresponding to each of the inference texts corresponding to each of the unit texts, a number of target texts are determined from the multiple inference texts, wherein the attention accumulation probability corresponding to the target texts is greater than the attention accumulation probability threshold; The number of target texts is counted, and the sampling threshold is dynamically set based on the number of target texts, wherein the sampling threshold is less than or equal to the number of target texts; Based on the attention accumulation probability corresponding to each target text, the attention accumulation probability is sorted, and several candidate inference texts are determined from the multiple target texts, wherein the number of candidate inference texts corresponds to the sampling threshold.
7. The model inference verification method according to claim 2, wherein, The step of arranging and combining texts according to the multiple candidate inference texts corresponding to each unit text and the text character positions to generate multiple candidate text sequences includes: Based on the text character positions corresponding to each of the aforementioned unit texts, determine the candidate text sequence positions corresponding to each of the aforementioned unit texts; Each candidate inference text corresponding to each unit text is sequentially filled into the candidate text sequence position corresponding to each unit text in a permutation and combination form to generate each candidate text sequence.
8. The model reasoning verification method according to claim 2, wherein, The step of segmenting the text prompt words into text characters includes: The text prompt words are divided using the N-gram model to form a sequence of byte segments of length N, where each byte segment serves as the unit text.
9. The model reasoning verification method according to claim 4, wherein, The tree-like attention mechanism network organizes sequential data into a tree structure, so that each element in the network only pays attention to other elements within a certain range of its neighbors.
10. The model inference verification method according to claim 5, further comprising: After determining the optimal text sequence, the large model is updated using the model inference parameters corresponding to the optimal text sequence.
11. A model reasoning verification device, comprising: The first module is configured to retrieve text prompts; The second module is configured to use a large model to perform dynamic pruning inference on the text prompt words and generate multiple candidate text sequences. The third module is configured to build a corresponding candidate sequence combination tree based on each of the candidate text sequences. The fourth module is configured to use the large model to perform tree-based decoding on the candidate sequence combination tree to determine the tree-based decoding output sequence; The fifth module is configured to perform consistency verification on each of the candidate text sequences based on the tree-structured decoding output sequence, and determine the optimal text sequence from the multiple candidate text sequences.
12. An electronic device, wherein, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the model reasoning verification method according to any one of claims 1 to 10.
13. A computer-readable storage medium storing a computer program, wherein, When the computer program is executed by a processor, it implements the model reasoning verification method according to any one of claims 1 to 10.