Inference method for artificial intelligence model, apparatus, electronic device and product

Abstract

The embodiments of the present disclosure relate to an inference method for an artificial intelligence model, an apparatus, an electronic device and a computer program product. The method comprises: acquiring an inference token for an inference operation. The method further comprises: on the basis of a first inference configuration parameter, acquiring a plurality of draft tokens. The method further comprises: on the basis of the inference token and the plurality of draft tokens, executing the inference operation in parallel. The method further comprises: after executing the inference operation, acquiring execution information of the inference operation. The method further comprises: when the execution information of the inference operation meets a preset condition, adjusting the first inference configuration parameter to a second inference configuration parameter. On the basis of the execution information of the inference operation, the present method dynamically adjusts the first inference configuration parameter, such that the inference operation can be better executed, thereby reducing the number of iterations of inference, reducing inference delay, and improving inference performance.

Description

Reasoning methods, devices, electronic devices, and products for artificial intelligence models.

[0001] This application claims priority to Chinese Patent Application No. 202411553081.8, filed on October 31, 2024, entitled "Reasoning Method, Apparatus, Electronic Device, and Product for Artificial Intelligence Models", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of computer technology, and more particularly to inference methods for artificial intelligence models, inference devices for artificial intelligence models, electronic devices, and computer program products. Background Technology

[0003] Currently, artificial intelligence (AI) technology is being applied in an increasing number of fields. In these fields, the need for reasoning based on user input is gradually increasing. Consequently, reasoning methods for AI models are also developing rapidly. Self-speculative decoding is gradually becoming a mainstream reasoning method. Self-speculative decoding does not require the participation of a small model; instead, it generates draft tokens through methods other than a small model, and then submits these draft tokens to a large language model (LLM) for verification to obtain accepted tokens. However, current self-speculative decoding schemes rely on pre-configured reasoning parameters, resulting in a relatively low acceptance rate for draft tokens. This affects the number of iterations and the speed of reasoning, and also wastes computational resources. Summary of the Invention

[0004] Embodiments of this disclosure provide a reasoning method for an artificial intelligence model, a reasoning device for an artificial intelligence model, an electronic device, and a computer program product.

[0005] According to a first aspect of this disclosure, a reasoning method for an artificial intelligence model is provided. The method includes acquiring reasoning lexical units for a reasoning operation. The method further includes acquiring multiple draft lexical units based on first reasoning configuration parameters. The method further includes executing the reasoning operation in parallel based on the reasoning lexical units and the multiple draft lexical units. The method further includes acquiring execution information of the reasoning operation after execution. The method further includes adjusting the first reasoning configuration parameters to second reasoning configuration parameters when the execution information of the reasoning operation meets preset conditions. This method dynamically adjusts the first reasoning configuration parameters based on the execution information of the reasoning operation, thus enabling better execution of the reasoning operation, thereby reducing the number of reasoning iterations, reducing reasoning latency, and improving reasoning performance.

[0006] In some embodiments of this disclosure, the plurality of draft words includes one or more sets of draft words. The execution information of the inference operation is the acceptance rate of the set of draft words with the highest acceptance rate among the set of draft words. If the execution information of the inference operation meets a preset condition, then during the process of adjusting the first inference configuration parameter to the second inference configuration parameter, after the inference operation is completed, the acceptance rate of the set of draft words is determined. When the acceptance rate of the set of draft words with the highest acceptance rate is less than a first threshold after K consecutive inference operations, the number of the plurality of draft words is reduced. Or when the acceptance rate of the set of draft words with the highest acceptance rate is greater than a second threshold after K consecutive inference operations, the number of the plurality of draft words is increased. Wherein the second threshold is greater than or equal to the first threshold, and K is greater than or equal to 1.

[0007] In some embodiments of this disclosure, in the process of reducing the number of multiple draft lexical units, the number G of the group or multiple groups of draft lexical units is reduced, and / or the number M of each group of draft lexical units is reduced. In the process of increasing the number of multiple draft lexical units, the number G of the group or multiple groups of draft lexical units is increased, and / or the number M of each group of draft lexical units is increased.

[0008] In these embodiments, the acceptance rate of the group with the highest acceptance rate among the one or more groups of draft lexics can be referred to as the total acceptance rate. The total acceptance rate is used to determine whether the multiple draft lexics (obtained according to the first inference configuration parameters) match the current inference scenario. If the number of draft lexics matching the current inference scenario increases, it indicates that the current inference performance is good; therefore, the number of draft lexicographical groups can be increased, or the length of each draft lexicographical group can be increased, or both can be increased. This can reduce the number of iterations and improve inference performance. If the number of draft lexics matching the current inference scenario decreases, it indicates that the current inference performance is poor; therefore, the number of draft lexicographical groups can be reduced, or the length of each draft lexicographical group can be reduced, or both can be reduced, to avoid excessive redundant calculations and wasted computational resources.

[0009] In some embodiments of this disclosure, the plurality of draft lexical units includes one or more sets of draft lexical units. The first inference configuration parameter includes the number G of the set of one or more sets of draft lexical units and the number M of lexical units in each set. During the process of obtaining the plurality of draft lexical units according to the first inference configuration parameter, X sets of static draft lexical units are obtained from at least one static N-gram library, and Y sets of dynamic draft lexical units are obtained from a dynamic N-gram library. The sum of X and Y is less than or equal to G, and the number of lexical units in each set is less than or equal to M. X is greater than or equal to 0. Y is greater than or equal to 0. G is greater than or equal to 1. M is greater than or equal to 1. Each static N-gram library in the at least one static N-gram library includes lexical units included in a specific scenario. The dynamic N-gram library includes draft lexical units accepted after each inference operation.

[0010] In this embodiment, the static N-gram library includes lexical units specific to a particular scenario, thus providing a basic draft lexical unit. The dynamic N-gram library, on the other hand, is dynamically updated and gradually enriched as the inference process progresses. The first inference configuration parameters are also dynamically updated. Based on the dynamically updated first inference configuration parameters, draft lexical units are dynamically selected from the static and / or dynamic N-gram libraries for parallel inference operations, improving the acceptance rate of draft lexical units and generating multiple accepted lexical units in a single inference iteration. Therefore, this method reduces the number of inference iterations, reduces inference latency, and improves inference performance.

[0011] In some embodiments of this disclosure, if the execution information of the inference operation meets preset conditions, then during the process of adjusting the first inference configuration parameter to the second inference configuration parameter, after the inference operation is completed, the acceptance rate of X groups of static draft words is determined. When the acceptance rate of the group with the highest acceptance rate among the X groups of static draft words is less than a first threshold after K consecutive inference operations, the proportion of the X groups of static draft words in that group or more draft words is reduced. Alternatively, or additionally, when the acceptance rate of the group with the highest acceptance rate among the X groups of static draft words is greater than a second threshold after K consecutive inference operations, the proportion of the X groups of static draft words in that group or more draft words is increased. Here, K is greater than or equal to 1.

[0012] In the above embodiments, the acceptance rate is used to determine whether static draft terms from the static N-gram lexicon match the current inference scenario. If the number of static draft terms from the static N-gram lexicon that match the current inference scenario increases, the proportion of draft terms selected from the static N-gram lexicon is increased to improve the acceptance rate of draft terms. This reduces the number of iterations and improves inference performance. If the number of draft terms from the static N-gram lexicon that match the current inference scenario decreases, the proportion of draft terms selected from the static N-gram lexicon is decreased to avoid a decrease in the acceptance rate of draft terms, which would affect inference performance.

[0013] In some embodiments of this disclosure, if the execution information of the inference operation meets preset conditions, then during the process of adjusting the first inference configuration parameter to the second inference configuration parameter, after the inference operation is completed, the acceptance rate of the Y groups of dynamic draft words is determined. When the acceptance rate of the group with the highest acceptance rate among the Y groups of dynamic draft words is greater than a second threshold after K consecutive inference operations, the proportion of the Y groups of dynamic draft words in that group or more draft words is increased. Alternatively, or additionally, when the acceptance rate of the group with the highest acceptance rate among the Y groups of dynamic draft words is less than a first threshold after K consecutive inference operations, the proportion of the Y groups of dynamic draft words in that group or more draft words is decreased. Here, the second threshold is greater than or equal to the first threshold. K is greater than or equal to 1.

[0014] In the above embodiments, the acceptance rate is used to determine whether draft terms from the dynamic N-gram lexicon match the current inference scenario. If the number of dynamic draft terms from the dynamic N-gram lexicon that match the current inference scenario increases, the proportion of draft terms selected from the dynamic N-gram lexicon is increased to improve the acceptance rate of draft terms. This reduces the number of iterations and improves inference performance. If the number of dynamic draft terms from the dynamic N-gram lexicon that match the current inference scenario decreases, the proportion of draft terms selected from the dynamic N-gram lexicon is decreased to avoid a decrease in the acceptance rate of draft terms, which would affect inference performance.

[0015] In some embodiments of this disclosure, the method further includes generating Z sets of Jacobi lexical units using the Jacobi method. During the parallel execution of inference operations based on inference lexical units and the plurality of draft lexical units, the inference operations are performed in parallel based on the inference lexical units, the plurality of draft lexical units, and the Z sets of Jacobi lexical units. The dynamic N-gram library also includes Jacobi lexical units accepted after each inference operation. If such Jacobi lexical units are accepted, the number of iterations can be reduced and inference performance improved. During the cold start phase, the number of N-grams in the dynamic N-gram library is relatively small. Randomly generating Jacobi lexical units can more quickly increase the number of N-grams in the dynamic N-gram library, thereby improving the acceptance rate of draft lexical units. Therefore, these embodiments are very helpful for the cold start phase.

[0016] In some further embodiments of this disclosure, the method further includes determining the acceptance rates of X groups of static draft lexical units and Y groups of dynamic draft lexical units after performing the inference operations. When the acceptance rate of the group with the highest acceptance rate among the X groups of static draft lexical units is less than a first threshold after K consecutive inference operations, or when the acceptance rate of the group with the highest acceptance rate among the Y groups of dynamic draft lexical units is greater than a second threshold after K consecutive inference operations, the number of Z groups of Jacobi lexical units is increased. Alternatively, or additionally, when the acceptance rate of the group with the highest acceptance rate among the X groups of static draft lexical units is greater than the second threshold after K consecutive inference operations, or when the acceptance rate of the group with the highest acceptance rate among the Y groups of dynamic draft lexical units is less than the first threshold after K consecutive inference operations, the number of Z groups of Jacobi lexical units is decreased. Here, K is greater than or equal to 1.

[0017] In these embodiments, the acceptance rate is used to determine whether static draft lexics from the static N-gram lexical library match the current inference scenario. If the number of static draft lexics from the static N-gram lexical library that match the current inference scenario increases, it indicates that the role of the dynamic N-gram lexical library can be weakened, thus reducing the number of Jacobian lexical groups and avoiding redundant draft lexics. If the number of draft lexics from the static N-gram lexical library that match the current inference scenario decreases, it indicates that the role of the dynamic N-gram lexical library needs to be strengthened, thus requiring an increase in the number of Jacobian lexical groups to expand the N-grams in the dynamic N-gram lexical library. This improves the acceptance rate of draft lexics and enhances inference performance.

[0018] In some embodiments of this disclosure, the inference operation is repeatedly executed until the inference termination condition is met. Each repeated execution of the inference operation is called a round of inference iteration. In the first round of inference iteration, the dynamic N-gram library can be a newly created empty N-gram library. In each round of inference iteration, the dynamic N-gram library is dynamically updated after each inference operation, and the first inference configuration parameter may also be dynamically updated. Based on the dynamically updated first inference configuration parameter, draft lexical units are dynamically selected for inference from a predetermined static N-gram library and / or a dynamically updated dynamic N-gram library, which can improve the acceptance rate of draft lexical units and generate multiple accepted lexical units in one inference iteration. Therefore, this method can reduce the number of inference iterations, reduce inference latency, and improve inference performance.

[0019] In some embodiments of this disclosure, the execution information of the inference operation may include at least one of the following: the acceptance rate of the group with the highest acceptance rate among X groups of static draft lexics (hereinafter referred to as the first acceptance rate), the acceptance rate of the group with the highest acceptance rate among Y groups of dynamic draft lexics (hereinafter referred to as the second acceptance rate), the total acceptance rate, computing power utilization, or bandwidth utilization. The method further includes reducing the number of draft lexics groups G and / or reducing the number of lexics M per group if the execution information is lower than a first execution information threshold. When the execution information is computing power utilization, if the computing power utilization is lower than a first computing power utilization threshold, the number of draft lexics groups is reduced, and / or the number of lexics M per group is reduced. When the execution information is bandwidth utilization, if the bandwidth utilization is lower than a first bandwidth utilization threshold, the number of draft lexics groups G and / or the number of lexics M per group is reduced.

[0020] In some embodiments of this disclosure, the execution information of the inference operation may include at least one of the following: a first acceptance rate, a second acceptance rate, a total acceptance rate, computing power utilization, or bandwidth utilization. The method further includes increasing the number of draft word groups and / or increasing the number of draft words in each draft word group if the execution information is higher than a second execution information threshold. When the execution information is computing power utilization, if the computing power utilization is higher than a second computing power utilization threshold, increasing the number of draft word groups and / or increasing the number of draft words in each draft word group. When the execution information is bandwidth utilization, if the bandwidth utilization is higher than a second bandwidth utilization threshold, increasing the number of draft word groups and / or increasing the number of draft words in each draft word group.

[0021] In the above embodiments, in addition to considering the acceptance rate of draft tokens, the hardware performance of the device executing the method is also taken into account. If the device has high computational power utilization or bandwidth utilization (good hardware performance), the hardware performance can be fully utilized for inference. Thus, with sufficient computing resources, even if the acceptance rate of a single draft token is not high, the acceptance rate of draft tokens can be improved by increasing the number of draft token groups, and the number of iterations can be reduced by increasing the length of draft tokens. If the device has low computational power utilization or bandwidth utilization (insufficient hardware performance), the number and length of draft tokens need to be determined based on available computing resources to adapt to specific application scenarios.

[0022] In some embodiments of this disclosure, the reasoning scenario can be identified based on the input lexical units. Then, a static N-gram library is obtained based on the reasoning scenario. The static N-gram library includes multiple N-grams associated with the reasoning scenario. The static N-gram library obtained in this manner can be considered a static N-gram library. Since each set of draft lexical units in the static N-gram library is associated with the reasoning scenario, the probability of draft lexical units selected from the static N-gram library being accepted is also higher during the cold start phase. Thus, by using the static N-gram library, especially during the cold start phase, the acceptance rate of draft lexical units can be improved, thereby improving reasoning performance.

[0023] In some embodiments of this disclosure, the reasoning scenario includes at least one of the following: code generation scenario, article summary scenario, online chat scenario, or intelligent question answering scenario.

[0024] In some embodiments of this disclosure, during the parallel execution of inference operations based on inference lexical units and multiple draft lexical units, the inference lexical units and the multiple draft lexical units are combined into corresponding sets of candidate lexical units. These sets of candidate lexical units are then decoded in parallel. Finally, the results of the parallel decoding are sampled to obtain the accepted draft lexical units.

[0025] In some embodiments of this disclosure, during the calculation of the acceptance rate of draft words, each group of draft words from the plurality of draft words is used as the target draft words. In this process, the accepted draft words in the target draft words are determined sequentially. If all draft words in the target draft words are accepted, the combination of the inference word and the target draft words is added to the dynamic N-gram library as a dynamic N-gram. If the first draft word in the target draft words is not accepted, the combination of the inference word and the specified word is added to the dynamic N-gram library as a dynamic N-gram. Here, the specified word includes the first word in the accepted words that corresponds to the first draft word and the words preceding the first word. Through this method, the dynamic N-gram library can be dynamically updated to improve the effectiveness of the next round of inference iterations.

[0026] In some embodiments of this disclosure, the accepted terms in each round are determined as new inference terms. Execution information is updated based on the hit rate of the plurality of draft terms relative to the accepted terms, thereby updating the first inference configuration parameters. In this manner, the first inference configuration parameters and inference terms can be dynamically updated to improve the effectiveness of the next round of inference iteration.

[0027] In some embodiments of this disclosure, the reasoning termination condition includes at least one of the following: the reasoning result reaches a preset word length; or the reasoning result includes a terminator. For example, if the reasoning result reaches the preset word length, the reasoning process terminates. Alternatively, if a terminator appears in the reasoning result, the reasoning process terminates.

[0028] In some embodiments of this disclosure, the method further includes: obtaining several sets of draft lexical units from a third N-gram library according to first inference configuration parameters. The third N-gram library includes N-grams retrieved via the web. The third N-gram library obtained via web retrieval serves as an additional alternative for draft lexical units, providing a richer selection of draft lexical units to improve their acceptance rate.

[0029] According to a second aspect of this disclosure, an inference apparatus for an artificial intelligence model is provided. The inference apparatus for the artificial intelligence model includes a first acquisition module, a second acquisition module, an execution module, a third acquisition module, and an adjustment module. The first acquisition module is configured to acquire inference lexical units for an inference operation. The second acquisition module is configured to acquire a plurality of draft lexical units based on first inference configuration parameters. The execution module is configured to execute the inference operation in parallel based on the inference lexical units and the plurality of draft lexical units. The third acquisition module is configured to acquire execution information of the inference operation after execution. The adjustment module is configured to adjust the first inference configuration parameters to second inference configuration parameters when the execution information of the inference operation meets preset conditions. This artificial intelligence model dynamically adjusts the first inference configuration parameters based on the execution information of the inference operation, thereby reducing the number of inference iterations, reducing inference latency, and improving inference performance.

[0030] In some embodiments of this disclosure, the plurality of draft words includes one or more sets of draft words. The execution information of the inference operation is the acceptance rate of the set of draft words with the highest acceptance rate among the set of draft words. The adjustment module includes a first determining module and a first sub-adjustment module. The first determining module is configured to determine the acceptance rate of one or more sets of draft words after the inference operation is completed. The first sub-adjustment module is configured to reduce the number of draft words when the acceptance rate of the set of draft words with the highest acceptance rate is less than a first threshold after K consecutive inference operations; or to increase the number of draft words when the acceptance rate of the set of draft words with the highest acceptance rate is greater than a second threshold after K consecutive inference operations. Wherein the second threshold is greater than or equal to the first threshold, and K is greater than or equal to 1.

[0031] In some embodiments of this disclosure, the first sub-adjustment module is configured to: when the acceptance rate of the group with the highest acceptance rate among one or more groups of draft words is less than a first threshold after K consecutive inference operations, perform at least one of the following: reduce the number of groups G of multiple draft words, or reduce the number of words M in each group of draft words.

[0032] In some embodiments of this disclosure, the first sub-adjustment module is configured to: when the acceptance rate of the group with the highest acceptance rate among one or more groups of draft lexical units is greater than a second threshold after K consecutive inference operations, perform at least one of the following: increase the number of groups of multiple draft lexical units G, or increase the number of lexical units M in each group of draft lexical units.

[0033] In some embodiments of this disclosure, the plurality of draft lexical units includes one or more sets of draft lexical units. The first inference configuration parameter includes the number G of the one or more sets of draft lexical units and the number M of lexical units in each set. The second acquisition module is configured to: acquire X sets of static draft lexical units from at least one static N-gram library and Y sets of dynamic draft lexical units from a dynamic N-gram library according to the first inference configuration parameter. Where X + Y ≤ G, and the number of lexical units in each set is less than or equal to M. Each static N-gram library in the at least one static N-gram library includes lexical units included in a specific scenario. The dynamic N-gram library includes draft lexical units accepted after each inference operation.

[0034] In some embodiments of this disclosure, the adjustment module includes a second determining module and a second sub-adjustment module. The second determining module is configured to determine, after performing inference operations, the acceptance rate of X groups of static draft words or the acceptance rate of Y groups of dynamic draft words. The second sub-adjustment module is configured to reduce the proportion of X groups of static draft words in one or more groups of draft words when the acceptance rate of the group with the highest acceptance rate among X groups of static draft words is less than a first threshold after K consecutive inference operations, or when the acceptance rate of the group with the highest acceptance rate among Y groups of dynamic draft words is greater than a second threshold after K consecutive inference operations; or to increase the proportion of X groups of static draft words in one or more groups of draft words when the acceptance rate of the group with the highest acceptance rate among X groups of static draft words is greater than the second threshold after K consecutive inference operations, or when the acceptance rate of the group with the highest acceptance rate among Y groups of dynamic draft words is less than the first threshold after K consecutive inference operations. Wherein, the second threshold is greater than or equal to the first threshold. K is greater than or equal to 1.

[0035] In some embodiments of this disclosure, the inference apparatus further includes a generation module. The generation module is configured to generate Z sets of Jacobian lexical units using the Jacobian method. The execution module is configured to perform inference operations in parallel based on the inference lexical units, multiple draft lexical units, and the Z sets of Jacobian lexical units. The dynamic N-gram library also includes Jacobian lexical units accepted after each inference operation.

[0036] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processor and a memory. The memory is coupled to the at least one processor and has instructions stored thereon. When executed by the at least one processor, the instructions cause the electronic device to perform the method according to a first aspect of this disclosure.

[0037] In a fourth aspect of this disclosure, a computer-readable storage medium is provided on which a computer program is stored. The computer program is executed by a processor to implement the method described according to a first aspect of this disclosure.

[0038] In a fifth aspect of this disclosure, a computer program product is provided, comprising computer-executable instructions. When executed by a processor, the instructions implement some or all of the steps of the method described according to a first aspect of this disclosure.

[0039] Understandably, the inference device for the artificial intelligence model of the second aspect, the electronic device of the third aspect, the computer storage medium of the fourth aspect, or the computer program product of the fifth aspect described above are all used to execute the method provided in the first aspect. Therefore, the explanations or descriptions regarding the first aspect also apply to the second, third, fourth, and fifth aspects. Furthermore, the beneficial effects achievable by the second, third, fourth, and fifth aspects can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0040] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0041] Figure 1 illustrates a schematic diagram of an example environment in which the apparatus and / or methods of embodiments of the present disclosure may be implemented.

[0042] Figure 2 shows an exemplary flowchart of a reasoning method for an artificial intelligence model according to an embodiment of the present disclosure.

[0043] Figure 3 illustrates an exemplary schematic diagram of a reasoning method for an artificial intelligence model according to an embodiment of the present disclosure.

[0044] Figure 4 illustrates an exemplary schematic diagram of the dynamic selection of draft terms according to an embodiment of the present disclosure.

[0045] Figure 5 illustrates an exemplary schematic diagram of dynamically adjusting draft lexical units according to an embodiment of the present disclosure.

[0046] Figure 6 illustrates an exemplary schematic diagram of dynamically updating a dynamic N-gram syntax library according to an embodiment of the present disclosure.

[0047] Figure 7 illustrates an exemplary schematic diagram of generating Jacobi morphemes according to an embodiment of the present disclosure.

[0048] Figure 8 illustrates an exemplary schematic diagram of dynamically adjusting Jacobi morphemes according to an embodiment of the present disclosure.

[0049] Figure 9 shows a schematic diagram of an inference apparatus for an artificial intelligence model according to an embodiment of the present disclosure.

[0050] In the various accompanying figures, the same or corresponding reference numerals indicate the same or corresponding parts. The elements in the accompanying figures are schematic and not drawn to scale. Detailed Implementation

[0051] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0052] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0053] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the specification and in the relevant art, and shall not be interpreted in an idealized or overly formal form unless otherwise explicitly defined herein. As used herein, the statement of “connecting” or “coupling” two or more parts together shall mean that these parts are directly joined together or joined through at least one intermediate component.

[0054] In the field of artificial intelligence, large language models (LLMs) can be used for inference. LLMs typically employ autoregressive decoding for inference. In autoregressive decoding, the input token for the next iteration depends on the output token of the current iteration. Thus, LLMs can perform serial decoding word by word. In AI, tokens are sometimes alternatively referred to as lexical symbols, identifiers, tags, or tokens. Each token represents a discrete element within the input text. For example, depending on the inference task, the discrete element used as a token can be a character, word, phrase, term, sentence, symbol, punctuation mark, or other meaningful element. However, serial decoding word by word results in a slow inference process and high inference latency. Furthermore, generating each token requires transferring parameters used for inference from storage to computation, making memory access bandwidth a bottleneck for improving inference speed.

[0055] One approach to improve inference speed is to generate several sets of draft lexical units outside the LLM (each set containing several draft lexical units). The LLM then performs parallel decoding and sampling verification on these draft lexical units to obtain the accepted lexical units (the units used at the end of the inference operation). The draft lexical units in each set typically have semantic relationships. This process can be viewed as one inference iteration (or one round of inference iteration). After multiple iterations, an inference text composed of multiple accepted lexical units can be obtained. This approach allows for the generation of multiple lexical units in a single inference iteration, thereby reducing the number of inference iterations, reducing memory access bandwidth usage, and improving inference performance. Furthermore, the LLM's verification ensures that the accuracy of the inference result does not decrease; therefore, parallel decoding technology using draft lexical units is becoming increasingly widely used. In this approach, how to generate draft lexical units becomes a crucial factor affecting inference speed.

[0056] As mentioned above, in current self-speculated inference schemes, the number and length of draft lexical units are fixed regardless of their acceptance rate. Therefore, the acceptance rate of draft lexical units in current self-speculated inference schemes is not very high. This affects the number of inference iterations and speed, and wastes computational resources. Embodiments of this disclosure propose an inference method for artificial intelligence models. This method proposes dynamically adjusting inference configuration parameters. In this method, inference lexical units for inference operations are obtained. Multiple draft lexical units are obtained according to first inference configuration parameters. Then, inference operations are executed in parallel based on the inference lexical units and the multiple draft lexical units. After the inference operations are completed, the execution information of the inference operations is obtained. When the execution information of the inference operations meets preset conditions, the first inference configuration parameters are adjusted to second inference configuration parameters. This method dynamically adjusts the first inference configuration parameters according to the execution information of the inference operations, thus enabling better execution of inference operations, thereby reducing the number of inference iterations, reducing inference latency, and improving inference performance.

[0057] The embodiments of this disclosure will now be described in further detail with reference to the accompanying drawings. Figure 1 shows a schematic diagram of an example environment 10 in which the apparatus and / or methods of the embodiments of this disclosure may be implemented. The apparatus and / or methods of the embodiments of this disclosure may be applied to an artificial intelligence (AI) cluster 12. The AI ​​cluster 12 may include a management node and multiple AI servers 120-1, ..., 120-n. The management node is used to manage the multiple AI servers 120-1, ..., 120-n. Each AI server 120-1, ..., 120-n includes a central processing unit (CPU) 121, one or more graphics processing units (GPUs) / neural network processing units (NPUs) 123-1, 123-2, and a bus 122. The bus 122 includes, for example, a high-speed serial computer expansion bus (PCIe bus). The CPU 121 and the GPU / NPU 123-1, 123-2 can communicate via the bus 122. CPU 121 and GPU / NPU 123-1, 123-2 can also share the same physical memory. CPU 121 and GPU / NPU 123-1, 123-2 can also interconnect via a proprietary protocol. Client 11 can send inference requests to AI service cluster 12. One or more AI servers in AI service cluster 12 can perform inference operations.

[0058] Figure 2 shows an exemplary flowchart of an inference method 200 for an artificial intelligence model according to an embodiment of the present disclosure. This method 200 can be executed by an inference device for an artificial intelligence model. Here, the inference device for the artificial intelligence model can be a CPU 121 or a cloud computing node, such as the AI ​​server 120-1 in Figure 1. The method 200 will now be illustrated schematically using the AI ​​server 120-1 as the execution entity. It should be noted that although two GPUs / NPUs are shown in Figure 1, the number of GPUs / NPUs can be more or less.

[0059] At box 202, CPU 121 in AI server 120-1 acquires the inference terms for the inference operation. In the first inference iteration, the inference terms are the input terms from the user. In each subsequent inference iteration, the inference terms are ordered combinations of the accepted terms from previous inference iterations. In this context, accepted terms refer to the terms accepted after the inference operation.

[0060] At box 204, CPU 121 acquires multiple draft tokens based on first inference configuration parameters. Here, the first inference configuration parameters refer to the configuration parameters used for inference operations. The number of first inference configuration parameters can be one or more. The acquired multiple draft tokens may include one or more sets of draft tokens, and each set of draft tokens may include one or more draft tokens.

[0061] At box 206, the GPU / NPU 123-1 and 123-2 in AI server 120-1 perform inference operations in parallel based on inference lexics and multiple draft lexics. In some embodiments of this disclosure, inference lexics can be combined with various groups of draft lexics, and inference operations can be performed in parallel on each combination through LLM.

[0062] At box 208, after CPU 121 completes the inference operation, it acquires the execution information of the inference operation. Here, the execution information is the information acquired after the inference operation is completed. The execution information can indicate the inference performance of the inference operation.

[0063] At box 210, when the execution information of the inference operation meets preset conditions, the CPU 121 adjusts the first inference configuration parameter to the second inference configuration parameter. Here, "adjusting the first inference configuration parameter to the second inference configuration parameter" means changing the parameter value to a parameter with a first value. When there are multiple first inference configuration parameters, only some parameters among the first inference configuration parameters can be adjusted.

[0064] Method 200 dynamically adjusts the first inference configuration parameters based on the execution information of the inference operation, thereby enabling better execution of the inference operation, reducing the number of inference iterations, reducing inference latency, and improving inference performance.

[0065] In some embodiments of this disclosure, the plurality of draft words includes one or more sets of draft words. The execution information of the inference operation is the acceptance rate of the set of draft words with the highest acceptance rate among the set of draft words. If the execution information of the inference operation meets a preset condition, then during the process of adjusting the first inference configuration parameter to the second inference configuration parameter, after the inference operation is completed, the acceptance rate of the set of draft words is determined. When the acceptance rate of the set of draft words with the highest acceptance rate is less than a first threshold after K consecutive inference operations, the number of the plurality of draft words is reduced. Or when the acceptance rate of the set of draft words with the highest acceptance rate is greater than a second threshold after K consecutive inference operations, the number of the plurality of draft words is increased. Wherein the second threshold is greater than or equal to the first threshold, and K is greater than or equal to 1.

[0066] In some embodiments of this disclosure, in the process of reducing the number of multiple draft lexical units, the number G of the group or multiple groups of draft lexical units is reduced, and / or the number M of each group of draft lexical units is reduced. In the process of increasing the number of multiple draft lexical units, the number G of the group or multiple groups of draft lexical units is increased, and / or the number M of each group of draft lexical units is increased.

[0067] In these embodiments, the acceptance rate of the group with the highest acceptance rate among the one or more groups of draft lexics can be referred to as the total acceptance rate. The total acceptance rate is used to determine whether the multiple draft lexics (obtained according to the first inference configuration parameters) match the current inference scenario. If the number of draft lexics matching the current inference scenario increases, it indicates that the current inference performance is good; therefore, the number of draft lexicographical groups can be increased, or the length of each draft lexicographical group can be increased, or both can be increased. This can reduce the number of iterations and improve inference performance. If the number of draft lexics matching the current inference scenario decreases, it indicates that the current inference performance is poor; therefore, the number of draft lexicographical groups can be reduced, or the length of each draft lexicographical group can be reduced, or both can be reduced, to avoid excessive redundant calculations and wasted computational resources.

[0068] In some embodiments of this disclosure, the plurality of draft lexical units includes one or more sets of draft lexical units. The first inference configuration parameter includes the number G of the set of one or more sets of draft lexical units and the number M of lexical units in each set. During the process of obtaining the plurality of draft lexical units according to the first inference configuration parameter, X sets of static draft lexical units are obtained from at least one static N-gram library, and Y sets of dynamic draft lexical units are obtained from a dynamic N-gram library. The sum of X and Y is less than or equal to G, and the number of lexical units in each set is less than or equal to M. X is greater than or equal to 0. Y is greater than or equal to 0. G is greater than or equal to 1. M is greater than or equal to 1. Each static N-gram library in the at least one static N-gram library includes lexical units included in a specific scenario. The dynamic N-gram library includes draft lexical units accepted after each inference operation. Here, the N-gram library can be represented as N-Grams. The N-gram library includes multiple N-grams. Where N represents the number of lexical units in a single N-gram, i.e., the length of a single N-gram.

[0069] In this embodiment, the static N-gram library includes lexical units specific to a particular scenario, thus providing a basic draft lexical unit. The dynamic N-gram library, on the other hand, is dynamically updated and gradually enriched as the inference process progresses. The first inference configuration parameters are also dynamically updated. Based on the dynamically updated first inference configuration parameters, draft lexical units are dynamically selected from the static and / or dynamic N-gram libraries for parallel inference operations, improving the acceptance rate of draft lexical units and generating multiple accepted lexical units in a single inference iteration. Therefore, this method reduces the number of inference iterations, reduces inference latency, and improves inference performance.

[0070] In some embodiments of this disclosure, if the execution information of the inference operation meets preset conditions, then during the process of adjusting the first inference configuration parameter to the second inference configuration parameter, after the inference operation is completed, the acceptance rate of X groups of static draft words is determined. When the acceptance rate of the group with the highest acceptance rate among the X groups of static draft words is less than a first threshold after K consecutive inference operations, the proportion of the X groups of static draft words in that group or more draft words is reduced. Alternatively, or additionally, when the acceptance rate of the group with the highest acceptance rate among the X groups of static draft words is greater than a second threshold after K consecutive inference operations, the proportion of the X groups of static draft words in that group or more draft words is increased. Here, K is greater than or equal to 1.

[0071] In the above embodiments, the acceptance rate is used to determine whether static draft terms from the static N-gram lexicon match the current inference scenario. If the number of static draft terms from the static N-gram lexicon that match the current inference scenario increases, the proportion of draft terms selected from the static N-gram lexicon is increased to improve the acceptance rate of draft terms. This reduces the number of iterations and improves inference performance. If the number of draft terms from the static N-gram lexicon that match the current inference scenario decreases, the proportion of draft terms selected from the static N-gram lexicon is decreased to avoid a decrease in the acceptance rate of draft terms, which would affect inference performance.

[0072] In some embodiments of this disclosure, if the execution information of the inference operation meets preset conditions, then during the process of adjusting the first inference configuration parameter to the second inference configuration parameter, after the inference operation is completed, the acceptance rate of the Y groups of dynamic draft words is determined. When the acceptance rate of the group with the highest acceptance rate among the Y groups of dynamic draft words is greater than a second threshold after K consecutive inference operations, the proportion of the Y groups of dynamic draft words in that group or more draft words is increased. Alternatively, or additionally, when the acceptance rate of the group with the highest acceptance rate among the Y groups of dynamic draft words is less than a first threshold after K consecutive inference operations, the proportion of the Y groups of dynamic draft words in that group or more draft words is decreased. Here, K is greater than or equal to 1.

[0073] In the above embodiments, the acceptance rate is used to determine whether draft terms from the dynamic N-gram lexicon match the current inference scenario. If the number of dynamic draft terms from the dynamic N-gram lexicon that match the current inference scenario increases, the proportion of draft terms selected from the dynamic N-gram lexicon is increased to improve the acceptance rate of draft terms. This reduces the number of iterations and improves inference performance. If the number of dynamic draft terms from the dynamic N-gram lexicon that match the current inference scenario decreases, the proportion of draft terms selected from the dynamic N-gram lexicon is decreased to avoid a decrease in the acceptance rate of draft terms, which would affect inference performance.

[0074] In some embodiments of this disclosure, the operations at boxes 202 to 210 can be repeatedly performed until the inference termination condition is met. Each repeated execution of the operations at boxes 202 to 210 is called a round of inference iteration. In the first round of inference iteration, the dynamic N-gram library can be a newly created empty N-gram library. In each round of inference iteration, the dynamic N-gram library is dynamically updated after each inference operation, and the first inference configuration parameter may also be dynamically updated. Based on the dynamically updated first inference configuration parameter, dynamically selecting draft lexicals from a predetermined (static) static N-gram library and / or a dynamically updated dynamic N-gram library for inference can improve the acceptance rate of draft lexicals, and multiple accepted lexicals can be generated in one inference iteration. Therefore, this method can reduce the number of inference iterations, reduce inference latency, and improve inference performance. The inference termination condition can be that the inference result reaches a preset lexical length. The inference termination condition can also be that the inference result includes a terminator. For example, if the inference result reaches the preset lexical length, the inference process ends. Alternatively, the reasoning process ends if a terminator appears in the result of the reasoning.

[0075] Figure 3 illustrates an exemplary schematic diagram of an inference method for an artificial intelligence model according to an embodiment of the present disclosure. As described above, a static N-gram library is static, and a dynamic N-gram library is dynamic. In this context, a static N-gram library may be alternatively referred to as static N-Grams, and a dynamic N-gram library may be alternatively referred to as dynamic N-Grams. Accordingly, a static N-gram may be alternatively referred to as a static N-Gram, and a dynamic N-gram may be alternatively referred to as a dynamic N-Gram. In the example of Figure 3, at box 320, CPU 121 may perform inference scenario recognition based on input token 301 input by the user. For example, the input token 301 may be used to perform intent recognition to determine the inference scenario desired by the user through natural language processing techniques. In some embodiments of the present disclosure, the inference scenario may be a code generation scenario, an article summary scenario, an online chat scenario, or an intelligent question answering scenario. The inference scenarios listed herein are merely exemplary, and the scenarios to which the embodiments of the present disclosure can be applied are not limited to the above scenarios. In one example, a user may, for example, input "Please help generate a piece of bubble sort code" as input token 301. Using natural language processing (NLP) technology, CPU 121 can identify the current code generation scenario from input word 301. In another example, a user could input, for instance, an article and "Please help summarize the above article," as input word 301. Using NLP technology, CPU 121 can identify the current article summarization scenario from input word 301. Similarly, in other reasoning scenarios, CPU 121 can also identify the current reasoning scenario from input word 301 using NLP technology.

[0076] After identifying the reasoning scenario, CPU 121 can obtain static N-Grams 311 based on the reasoning scenario. Static N-Grams 311 can include multiple static N-Grams associated with the reasoning scenario. Each static N-Gram includes one or more lexical units. In one example, CPU 121 can generate static N-Grams 311 locally based on input lexical unit 301. The static N-Grams in the static N-Grams 311 can be obtained, for example, from a local database or knowledge graph. CPU 121 can also receive static N-Grams 311 associated with input lexical unit 301 from other CPUs 121. Other CPUs 121 can, for example, pre-generate multiple static N-Grams and, upon receiving a reasoning scenario associated with input lexical unit 301, find the static N-Grams 311 for that reasoning scenario and send the static N-Grams 311 to that CPU 121. Since each static N-Gram in a static N-Grams is associated with the inference scenario, the probability of draft terms selected from static N-Grams being accepted remains high during the cold start phase. Thus, by leveraging static N-Grams, especially during the cold start phase, the acceptance rate of draft terms can be improved, thereby enhancing inference performance.

[0077] In parallel with the operation of obtaining the static N-Grams 311, an empty dynamic N-Grams 312 can be created. For example, a dedicated storage space for the dynamic N-Grams 312 can be reserved in the memory of CPU 121. The static N-Grams 311 and the dynamic N-Grams 312 can be viewed as an N-Grams set 310. In each round of inference iteration, at box 330, multiple draft terms can be selected from the N-Grams set 310.

[0078] In the first round of inference iteration, CPU 121 determines the last one or more words in input word 301 as inference words. The inference words are combined (concatenated) with each set of draft words selected from the N-Grams set 310 to form a set of candidate words. Assuming G sets of draft words are selected from the N-Grams set 310, the inference words are combined with each of the G sets of draft words to form G sets of candidate words. In this context, G is a positive integer. For example, the inference words are combined with the first set of draft words in the G sets to form the first set of candidate words, the inference words are combined with the second set of draft words in the G sets to form the second set of candidate words, and so on. Assuming input word 301 is "tomorrow", the inference word could be "day". If the first set of draft words is "is sunny", then the first set of candidate words is "day is sunny". If the second set of draft words is "is rainy", then the second set of candidate words is "day is rainy". Alternatively, assuming the input word 301 is "tomorrow", the inference word lexicon can be "tomorrow". If the first draft word lexicon is "is sunny", then the first candidate word lexicon is "tomorrow is sunny". If the second draft word lexicon is "is rainy", then the second candidate word lexicon is "tomorrow is rainy".

[0079] In the first round of inference iteration, the proportion of draft words selected from static N-Grams can be the default. Correspondingly, the proportion of draft words selected from dynamic N-Grams is also the default. The number of selected draft word groups can be the default, and the length of the selected draft words can also be the default. In one example, the default proportion of draft words selected from static N-Grams can be set to 100%, the number of selected draft word groups can be set to G, and the length of the selected draft words can be M. In this context, M is a positive integer. In this case, if the number of N-Grams matching the inference word in the static N-Grams is less than G, draft words are selected from the dynamic N-Grams to make up G draft words. If G draft words cannot be selected from either N-Grams, the current round of inference iteration uses the currently selectable draft words (i.e., uses fewer than G groups of draft words for the current round of inference iteration).

[0080] At box 350, the LLM performs parallel decoding on G groups of candidate words. The LLM can be deployed locally on CPU 121 or on an external server. If the LLM is deployed on an external server, the CPU 121 sends instructions to the external server to control the LLM to perform parallel decoding. During parallel decoding, G groups of candidate words are input into the LLM. Using a preset mask, the LLM can distinguish between the G groups of candidate words to obtain the decoding result for each group. Here, the LLM can use known parallel decoding techniques to decode the G groups of candidate words, or it can use parallel decoding techniques developed in the future. The embodiments of this disclosure do not limit the parallel decoding method.

[0081] At box 360, the LLM samples the decoding results of all candidate tokens and verifies the samples to determine the accepted tokens. The sampling method can be greedy sampling, selecting the K decoding results with the most accepted tokens, or randomly selecting K decoding results from the P decoding results with the most accepted tokens. In this context, K and P are positive integers, with K less than P. Here, accepted tokens refer to tokens that consecutively match the LLM's decoding results according to the order of natural language. Once an unaccepted token appears, all tokens following it are considered unaccepted. Assume a single draft token group consists of a first draft token, a second draft token, a third draft token, and a fourth draft token. If the first, second, and fourth draft tokens all match the LLM's decoding results, but the third draft token does not, then the accepted tokens are the first and second draft tokens. Since the third draft lexicon is an unaccepted lexicon, the fourth draft lexicon is also considered an unaccepted lexicon. If the lexicon in the LLM decoding result corresponding to the third draft lexicon is called the substitution lexicon, then the inference lexicon, the first draft lexicon, the second draft lexicon, and the substitution lexicon are combined (concatenated) into the inference result 370.

[0082] In some embodiments of the present disclosure, during the verification process, the CPU 121 may regard each group of draft tokens in the sampling result as target draft tokens and calculate the acceptance rate for the target draft tokens. During this process, the accepted draft tokens in the target draft tokens are determined in order. In the example where the first group of draft tokens is "It is sunny", assuming that the decoding result of the LLM for "Tomorrow is sunny" is "Tomorrow is rainy", then only one of the three draft tokens "is", "sunny", and "day" in the first group of draft tokens is an accepted token, and its acceptance rate is 1 / 3. In this example, the inference result 370 is "Tomorrow is rainy". In the example where the second group of draft tokens is "It is rainy", assuming that the decoding result of the LLM for "Tomorrow is rainy" is "Tomorrow is rainy", then all three of the three draft tokens "is", "rainy", and "day" in the second group of draft tokens are accepted tokens, and its acceptance rate is 3 / 3 = 100%. In this example, the inference result 370 is "Tomorrow is rainy".

[0083] As shown by the arrow ARR1 in FIG. 3, the inference result 370 can be used to generate a new dynamic N-Gram and add the new dynamic N-Gram to the dynamic N-Grams 312. In one example, if all the draft tokens in the target draft tokens are accepted, the candidate token corresponding to the target draft token is added as a dynamic N-Gram to the dynamic N-Grams. If the first draft token in the target draft tokens is not accepted, the inference token and the specified tokens are concatenated into a dynamic N-Gram, and the dynamic N-Gram is added to the dynamic N-Grams. Here, the specified tokens include the first token corresponding to the first draft token in the inference result and the tokens before the first token. In another example, depending on the length N of the new dynamic N-Gram, the new dynamic N-Gram may include one or more tokens in the inference result 370. In one example, assuming that the inference result 370 is "Tomorrow is rainy", if N = 3, "day is rainy" can be added as a dynamic N-Gram to the dynamic N-Grams 312. If N = 2, "day is" can be added as a dynamic N-Gram to the dynamic N-Grams 312. "is rainy" can also be added as a dynamic N-Gram to the dynamic N-Grams 312.

[0084] As shown by the arrow ARR2 in FIG. 3, the inference result 370 can be used as a new inference token. Assuming that the inference result 370 is "Tomorrow is rainy", the inference token can be updated to "Tomorrow is rainy". Assuming that the inference result 370 is "Tomorrow is rainy day", the inference token can be updated to "Tomorrow is rainy day".

[0085] As shown by arrow ARR3 in Figure 3, the acceptance rate of each group of draft terms determined based on the inference result 370 can be used as execution information for the inference operation, used to select one or more groups of draft terms from the N-Grams set 310 at box 330. Since the value of the execution information is redefined in each iteration, the selection of draft terms is dynamic. Figure 4 shows an exemplary schematic diagram of the dynamic selection of draft terms according to an embodiment of this disclosure.

[0086] In the example of Figure 4, the inference term 44 is concatenated with multiple draft terms selected from the N-Grams set 310 to form multiple candidate terms. Some of the draft terms in these multiple sets may come from static N-Grams 311, while others may come from dynamic N-Grams 312. At box 350, the multiple candidate terms are input into the LLM for parallel decoding. During decoding, the LLM uses masks to distinguish between candidate terms, ensuring that decoding of a single candidate term is not affected by other candidate terms. For example, when decoding a set of draft terms 46, the draft terms between the inference term 44 and that set of draft terms 46, as well as the draft terms after that set of draft terms 46, are masked. Thus, the LLM decodes the candidate term formed by combining the inference term 44 and that set of draft terms 46. In Figure 4, decoded term 45 represents the decoding result of the LLM on the inference term 44. Similarly, decoded lexicon 47 represents the result of LLM decoding of draft lexicon 46.

[0087] At box 360, LLM samples the decoding results of all candidate tokens and verifies the sampled results to determine the accepted tokens. The verification process is illustrated below using a set of draft tokens 46 as an example. Assume this set of draft tokens 46 includes draft tokens D1, D2, and D3. The decoding result for draft token D1 is decoded token d1, the decoding result for draft token D2 is decoded token d2, and the decoding result for draft token D3 is decoded token d3. Comparing draft token D1 with decoded token 45 determines whether draft token D1 is correct. Comparing draft token D2 with decoded token d1 determines whether draft token D2 is correct. Comparing draft token D3 with decoded token d2 determines whether draft token D3 is correct. Assuming draft lexical D1 is the same as decoded lexical 45, draft lexical D2 is the same as decoded lexical d1, and draft lexical D3 is different from decoded lexical d2, then draft lexical D1 and draft lexical D2 can be determined as accepting lexicals. Decoded lexical d2 can be determined as a replacement lexical for draft lexical D3. Thus, draft lexical D1, draft lexical D2, and decoded lexical d2 can be determined as a new dynamic N-Gram and added to dynamic N-Grams 312.

[0088] At box 480, CPU 121 calculates the acceptance rate of each draft term and updates the execution information used to select draft terms from the N-Grams set 310 based on the calculation results. In one example, the execution information may include a first acceptance rate of draft terms selected from static N-Grams. The first acceptance rate is the highest among the hit rates of each draft term selected from static N-Grams relative to the inference result. In another example, the execution information may include a second acceptance rate of draft terms selected from dynamic N-Grams. The second acceptance rate is the highest among the hit rates of each draft term selected from dynamic N-Grams relative to the inference result. In yet another example, the execution information may include the total acceptance rate of draft terms selected from the N-Grams set 310. The total acceptance rate is the highest among the hit rates of draft terms selected from the N-Grams set 310 relative to the inference result.

[0089] The following describes an embodiment where the execution information includes a first acceptance rate. In this embodiment, during the process of determining the number of static draft lexicons X and the number of dynamic draft lexicons Y, the CPU 121 can determine a first acceptance rate for draft lexicons selected from static N-Grams. If the first acceptance rate is lower than a first acceptance rate threshold, the CPU 121 reduces the proportion of the number of draft lexicons selected from static N-Grams among the plurality of draft lexicons. If the first acceptance rate is higher than a second acceptance rate threshold, the CPU 121 increases the proportion of the number of draft lexicons selected from static N-Grams among the plurality of draft lexicons. Here, the second acceptance rate threshold is higher than the first acceptance rate threshold. If the first acceptance rate is higher than the first acceptance rate threshold and lower than the second acceptance rate threshold, the CPU 121 keeps the proportion of the number of draft lexicons selected from static N-Grams unchanged.

[0090] Alternatively, in other embodiments, the proportion of draft lexical units selected from static N-Grams can be adjusted based on the results of multiple rounds of inference iterations. In these embodiments, during the determination of the number of static draft lexical units X and the number of dynamic draft lexical units Y, CPU 121 determines a first acceptance rate for draft lexical units selected from static N-Grams. If the first acceptance rate is lower than a first acceptance rate threshold, CPU 121 increments a first counter. If the first acceptance rate is higher than or equal to the first acceptance rate threshold, CPU 121 resets the first counter to zero. If the value of the first counter reaches a first counter threshold, CPU 121 reduces the proportion of draft lexical units selected from static N-Grams in the plurality of draft lexical units and resets the first counter to zero. If the first acceptance rate is higher than a second acceptance rate threshold, CPU 121 increments a second counter. If the first acceptance rate is lower than or equal to the second acceptance rate threshold, CPU 121 resets the second counter to zero. If the value of the second counter reaches the second counting threshold, the CPU 121 increases the proportion of draft words selected from static N-Grams among the plurality of draft words and then resets the second counter to zero. Here, the second acceptance rate threshold is higher than the first acceptance rate threshold. In other cases, the CPU 121 keeps the proportion of draft words selected from static N-Grams unchanged.

[0091] In the above embodiments, a first acceptance rate is used to determine whether draft terms from the static N-Grams match the current inference scenario. If the number of draft terms from the static N-Grams that match the current inference scenario increases, the proportion of draft terms selected from the static N-Grams is increased to improve the acceptance rate of the draft terms. This reduces the number of iterations and improves inference performance. If the number of draft terms from the static N-Grams that match the current inference scenario decreases, the proportion of draft terms selected from the static N-Grams is reduced to avoid a decrease in the acceptance rate of the draft terms, which could affect inference performance.

[0092] The following describes an embodiment where the execution information includes a second acceptance rate. In this embodiment, during the determination of the number of static draft lexicons X and the number of dynamic draft lexicons Y, the CPU 121 determines a second acceptance rate for draft lexicons selected from dynamic N-Grams. If the second acceptance rate is lower than a third acceptance rate threshold, the CPU 121 reduces the proportion of the number of draft lexicons selected from dynamic N-Grams among the plurality of draft lexicons. If the second acceptance rate is higher than a fourth acceptance rate threshold, the CPU 121 increases the proportion of the number of draft lexicons selected from dynamic N-Grams among the plurality of draft lexicons. Here, the fourth acceptance rate threshold is higher than the third acceptance rate threshold. If the second acceptance rate is higher than the third acceptance rate threshold but lower than the fourth acceptance rate threshold, the CPU 121 keeps the proportion of the number of draft lexicons selected from dynamic N-Grams unchanged.

[0093] Alternatively, in other embodiments, the proportion of draft lexical units selected from dynamic N-Grams can be adjusted based on the results of multiple rounds of inference iterations. In these embodiments, during the determination of the number of static draft lexical units X and the number of dynamic draft lexical units Y, CPU 121 determines a second acceptance rate for draft lexical units selected from dynamic N-Grams. If the second acceptance rate is lower than a third acceptance rate threshold, CPU 121 increments a third counter. If the second acceptance rate is higher than or equal to the third acceptance rate threshold, CPU 121 resets the third counter to zero. If the value of the third counter reaches the third counter threshold, CPU 121 reduces the proportion of draft lexical units selected from dynamic N-Grams in the plurality of draft lexical units and resets the third counter to zero. If the second acceptance rate is higher than a fourth acceptance rate threshold, CPU 121 increments a fourth counter. If the second acceptance rate is lower than or equal to the fourth acceptance rate threshold, CPU 121 resets the fourth counter to zero. If the value of the fourth counter reaches the fourth counting threshold, the CPU 121 increases the proportion of draft words selected from dynamic N-Grams among the plurality of draft words and then resets the fourth counter to zero. Here, the fourth acceptance rate threshold is higher than the third acceptance rate threshold. In other cases, the CPU 121 keeps the proportion of draft words selected from dynamic N-Grams unchanged.

[0094] In the above embodiments, a second acceptance rate is used to determine whether draft terms from the dynamic N-Grams match the current inference scenario. If the number of draft terms from the dynamic N-Grams that match the current inference scenario increases, the proportion of draft terms selected from the dynamic N-Grams is increased to improve the acceptance rate of the draft terms. This reduces the number of iterations and improves inference performance. If the number of draft terms from the dynamic N-Grams that match the current inference scenario decreases, the proportion of draft terms selected from the dynamic N-Grams is reduced to avoid a decrease in the acceptance rate of the draft terms, which could affect inference performance.

[0095] The following example, using Figure 5, illustrates an implementation of the execution information including the overall acceptance rate. In the example in Figure 5, the dark box represents inference token 44, and the striped box represents draft tokens. Each row of draft tokens represents a group of draft tokens. Assume a single group of draft tokens contains 5 draft tokens. The inference tokens are concatenated with each group of draft tokens to form a group of candidate tokens. At box 350, the LLM performs parallel decoding on the 3 groups of candidate tokens. Assume that one draft token in the first group matches the LLM's decoding result, while no draft tokens in the second and third groups match the LLM's decoding result. The acceptance rate for each group of draft tokens can be calculated at box 480. The acceptance rate for the first group of draft tokens is 1 / 5, while the acceptance rates for the other two groups are 0; therefore, the overall acceptance rate for the 3 groups of draft tokens is 1 / 5.

[0096] If the overall acceptance rate is below the fifth acceptance rate threshold, CPU 121 reduces the number of draft word groups and / or reduces the number of draft words in each draft word group. If the overall acceptance rate is above the sixth acceptance rate threshold, CPU 121 increases the number of draft word groups and / or increases the number of draft words in each draft word group. Here, the sixth acceptance rate threshold is higher than the fifth acceptance rate threshold. If the overall acceptance rate is higher than the fifth acceptance rate threshold but lower than the sixth acceptance rate threshold, CPU 121 keeps the number of draft word groups and the number of draft words in each draft word group unchanged.

[0097] Referring to the example in Figure 5, assuming the fifth acceptance rate threshold is greater than 1 / 5, then the total acceptance rate of the three draft word groups is lower than the fifth acceptance rate threshold. Therefore, in the next inference iteration, the length of a single draft word group can be reduced to 4. Alternatively, or additionally, the number of groups of these multiple draft word groups can be reduced in the next inference iteration, for example, from 3 to 2.

[0098] Alternatively, in other embodiments, the number and length of draft word groups can be adjusted based on the results of multiple rounds of inference iterations. In these embodiments, if the overall acceptance rate is below a fifth acceptance rate threshold, a fifth counter is incremented. If the overall acceptance rate is above or equal to the fifth acceptance rate threshold, the fifth counter is reset to zero. If the value of the fifth counter reaches the fifth counting threshold, the number of draft word groups is reduced, and / or the number of draft words in each draft word group is reduced, and the fifth counter is reset to zero. If the overall acceptance rate is above a sixth acceptance rate threshold, a sixth counter is incremented. If the overall acceptance rate is below or equal to the sixth acceptance rate threshold, the sixth counter is reset to zero. If the value of the sixth counter reaches the sixth counting threshold, the number of draft word groups is increased, and / or the number of draft words in each draft word group is increased, and the sixth counter is reset to zero.

[0099] In the above embodiments, the overall acceptance rate is used to determine whether the plurality of draft words (selected draft words) match the current inference scenario. If the number of draft words matching the current inference scenario increases, it indicates that the current inference effect is good. Therefore, the number of draft word groups can be increased, the length of each draft word group can be increased, or both can be increased. This can reduce the number of iterations and improve inference performance. If the number of draft words matching the current inference scenario decreases, it indicates that the current inference effect is poor. Therefore, the number of draft word groups can be reduced, the length of each draft word group can be reduced, or both can be reduced to avoid excessive redundant calculations and wasted computing resources. It should be noted that since the length of the draft words can change dynamically, the length of the new dynamic N-Gram can also change dynamically. Thus, the lengths of the dynamic N-Grams in the dynamic N-Grams can all be the same, partially the same and partially different, or all different.

[0100] To enrich dynamic N-Grams, some embodiments of this disclosure propose using a random approach to obtain dynamic N-Grams. Figure 6 illustrates an exemplary schematic diagram of dynamically updating dynamic N-Grams according to embodiments of this disclosure. These embodiments are described below with reference to Figure 6. In these embodiments, CPU 121 generates one or more sets of Jacobi lexical units using the Jacobi method, such as the three sets of Jacobi lexical units 61, 62, and 63 in Figure 6. Then, the inference lexical unit 44 is combined with each set of Jacobi lexical units 61, 62, and 63 to form one or more sets of candidate lexical units respectively. The three sets of Jacobi lexical units 61, 62, and 63, along with two sets of draft lexical units 64 and 65 selected from the N-Grams set 310, are input into the LLM for parallel decoding. The Jacobi lexical units and draft lexical units can be distinguished by means of a mask. The candidate lexical units obtained based on the Jacobi lexical units can then also be decoded at box 350 and sampled for verification at box 360. Thus, the inference result 370 may contain contributions from Jacobian lexical units 61, 62, and 63. If the candidate lexical units corresponding to these Jacobian lexical units are accepted, the number of iterations can be reduced and inference performance improved. During the cold start phase, the number of N-Grams in the dynamic N-Grams is still relatively small. Randomly generating Jacobian lexical units can more quickly increase the number of N-Grams in the dynamic N-Grams, thereby improving the acceptance rate of draft lexical units. Therefore, these embodiments are very helpful for the cold start phase.

[0101] Figure 7 illustrates an exemplary schematic diagram of generating Jacobian lexical units according to an embodiment of the present disclosure. Jacobian lexical units can be obtained through multiple iterations (e.g., Jacobian iterations). It should be noted that this iteration is a different process from the inference iteration described herein. In the first iteration, Jacobian lexical unit 1 can be randomly generated, and then inference lexical unit 0 is concatenated with Jacobian lexical unit 1 to form a first random input, which is then decoded in parallel at box 350. This yields Jacobian lexical unit 2. In the next iteration, inference lexical unit 0, Jacobian lexical unit 1, and Jacobian lexical unit 2 can be concatenated to form a second random input, which is then decoded in parallel at box 350. This yields Jacobian lexical unit 3. Assuming the length of a single set of Jacobian lexical units is 3, Jacobian lexical unit 1, Jacobian lexical unit 2, and Jacobian lexical unit 3 are concatenated into a set of Jacobian lexical units. Each of the three sets of Jacobian lexical units 61, 62, and 63 in Figure 6 can be generated in the manner shown in Figure 7.

[0102] Some embodiments of this disclosure propose that Jacobian lexical units can also be dynamically adjusted based on execution information. Figure 8 shows an exemplary schematic diagram of dynamically adjusting Jacobian lexical units according to an embodiment of this disclosure. In the example of Figure 8, five groups of Jacobian lexical units are shown, with the Jacobian lexical units in each group connected by solid lines, and the arrows on the solid lines indicating the order of the lexical units. Each row of Jacobian lexical units represents the Jacobian lexical units processed in the same iteration. Each group of Jacobian lexical units includes four Jacobian lexical units.

[0103] At box 480, CPU 121 calculates the acceptance rate of each group of draft lexicons. In some embodiments of this disclosure, the Jacobian lexicons can be dynamically adjusted using a first acceptance rate of draft lexicons selected from static N-Grams. If the first acceptance rate is lower than a first acceptance rate threshold, CPU 121 increases the number of Jacobian lexicons. If the first acceptance rate is higher than a second acceptance rate threshold, CPU 121 decreases the number of Jacobian lexicons. Here, the second acceptance rate threshold is higher than the first acceptance rate threshold. If the first acceptance rate is higher than the first acceptance rate threshold but lower than the second acceptance rate threshold, CPU 121 keeps the number of Jacobian lexicons unchanged. In the example of Figure 8, assuming the first acceptance rate is higher than the second acceptance rate threshold, CPU 121 reduces the number of Jacobian lexicons from 5 to 4.

[0104] In these embodiments, a first acceptance rate is used to determine whether draft terms from static N-Grams match the current inference scenario. If the number of draft terms from static N-Grams that match the current inference scenario increases, it indicates that the effect of dynamic N-Grams can be weakened, thus reducing the number of Jacobian term sets and avoiding redundant draft terms. If the number of draft terms from static N-Grams that match the current inference scenario decreases, it indicates that the effect of dynamic N-Grams needs to be enhanced, thus requiring an increase in the number of Jacobian term sets to expand the N-Grams in dynamic N-Grams. This improves the acceptance rate of draft terms and enhances inference performance.

[0105] In other embodiments of this disclosure, a second acceptance rate selected from draft lexicons of dynamic N-Grams can be used to dynamically adjust the number of Jacobi lexicons. If the second acceptance rate is lower than a third acceptance rate threshold, the CPU 121 reduces the number of Jacobi lexicons. If the second acceptance rate is higher than a fourth acceptance rate threshold, the CPU 121 increases the number of Jacobi lexicons. Here, the fourth acceptance rate threshold is higher than the third acceptance rate threshold. If the second acceptance rate is higher than the third acceptance rate threshold but lower than the fourth acceptance rate threshold, the CPU 121 keeps the number of Jacobi lexicons unchanged.

[0106] In these embodiments, a second acceptance rate is used to determine whether draft terms from the dynamic N-Grams match the current inference scenario. If the number of draft terms from the dynamic N-Grams that match the current inference scenario increases, it indicates that the effect of the dynamic N-Grams can be enhanced, and therefore the number of Jacobi terminology groups can be increased to expand the N-Grams in the dynamic N-Grams. This improves the acceptance rate of draft terms and enhances inference performance. If the number of draft terms from the dynamic N-Grams that match the current inference scenario decreases, it indicates that the effect of the dynamic N-Grams can be weakened, and therefore the number of Jacobi terminology groups can be reduced to avoid redundant draft terms.

[0107] Furthermore, some embodiments of this disclosure propose that computing power utilization or bandwidth utilization can be determined as execution information. If the execution information is lower than a first execution information threshold, the CPU 121 reduces the number of groups of the plurality of draft words and / or reduces the number of draft words in each group of draft words. When the execution information is computing power utilization, if the computing power utilization is lower than a first computing power utilization threshold, the CPU 121 reduces the number of groups of the plurality of draft words and / or reduces the number of draft words in each group of draft words. When the execution information is bandwidth utilization, if the bandwidth utilization is lower than a first bandwidth utilization threshold, the CPU 121 reduces the number of groups of the plurality of draft words and / or reduces the number of draft words in each group of draft words. If the execution information is higher than a second execution information threshold, the CPU 121 increases the number of groups of the plurality of draft words and / or increases the number of draft words in each group of draft words. When the execution information is computing power utilization, if the computing power utilization is higher than a second computing power utilization threshold, then CPU 121 increases the number of groups of the multiple draft tokens, and / or increases the number of draft tokens in each group of draft tokens. When the execution information is bandwidth utilization, if the bandwidth utilization is higher than a second bandwidth utilization threshold, then CPU 121 increases the number of groups of the multiple draft tokens, and / or increases the number of draft tokens in each group of draft tokens.

[0108] In the above embodiments, in addition to considering the acceptance rate of draft tokens, the hardware performance of the device executing the method is also taken into account. If the device has high computational power utilization or bandwidth utilization (good hardware performance), the hardware performance can be fully utilized for inference. Thus, with sufficient computing resources, even if the acceptance rate of a single draft token is not high, the acceptance rate can be improved by increasing the number of draft token groups, and the number of iterations can be reduced by increasing the length of the draft tokens. If the device has low computational power utilization or bandwidth utilization (insufficient hardware performance), the number and length of draft token groups need to be determined based on available computing resources to adapt to specific application scenarios.

[0109] As an extension, some embodiments of this disclosure propose that more N-Grams can be used to generate draft terms. In these embodiments, the technical device can establish third N-Grams. The third N-Grams include N-Grams retrieved via the web. The draft terms at box 340 in Figure 3 can also be selected from the third N-Grams. The third N-Grams obtained through web retrieval, as an additional alternative to draft terms, can provide a richer selection of draft terms, thereby improving the acceptance rate of draft terms.

[0110] Figure 9 shows a schematic diagram of an inference apparatus 900 for an artificial intelligence model according to an embodiment of the present disclosure. The apparatus 900 is, for example, arranged in the AI ​​server cluster 12 shown in Figure 1. The apparatus 900 may include multiple modules for performing corresponding steps in the method 200 discussed in Figure 2. As shown in Figure 9, the apparatus 900 includes a first acquisition module 902, a second acquisition module 904, an execution module 906, a third acquisition module 908, and an adjustment module 910. The first acquisition module 902 is configured to acquire inference terms for an inference operation. The second acquisition module 904 is configured to acquire multiple draft terms based on first inference configuration parameters. The execution module 906 is configured to execute the inference operation in parallel based on the inference terms and the multiple draft terms. The third acquisition module 908 is configured to acquire execution information of the inference operation after execution. The adjustment module 910 is configured to adjust the first inference configuration parameters to second inference configuration parameters when the execution information of the inference operation meets preset conditions. The device 900 dynamically adjusts the first inference configuration parameters based on the execution information of the inference operation, thereby reducing the number of inference iterations, reducing inference latency, and improving inference performance.

[0111] In some embodiments of this disclosure, the plurality of draft words includes one or more sets of draft words. The execution information of the inference operation is the acceptance rate of the set of draft words with the highest acceptance rate among the set of draft words. The adjustment module 910 includes a first determining module and a first sub-adjustment module. The first determining module is configured to determine the acceptance rate of one or more sets of draft words after the inference operation is completed. The first sub-adjustment module is configured to reduce the number of draft words when the acceptance rate of the set of draft words with the highest acceptance rate is less than a first threshold after K consecutive inference operations; or to increase the number of draft words when the acceptance rate of the set of draft words with the highest acceptance rate is greater than a second threshold after K consecutive inference operations. Wherein the second threshold is greater than or equal to the first threshold, and K is greater than or equal to 1.

[0112] In some embodiments of this disclosure, the first sub-adjustment module is configured to: when the acceptance rate of the group with the highest acceptance rate among one or more groups of draft words is less than a first threshold after K consecutive inference operations, perform at least one of the following: reduce the number of groups G of multiple draft words, or reduce the number of words M in each group of draft words.

[0113] In some embodiments of this disclosure, the first sub-adjustment module is configured to: when the acceptance rate of the group with the highest acceptance rate among one or more groups of draft lexical units is greater than a second threshold after K consecutive inference operations, perform at least one of the following: increase the number of groups of multiple draft lexical units G, or increase the number of lexical units M in each group of draft lexical units.

[0114] In some embodiments of this disclosure, the plurality of draft lexical units includes one or more sets of draft lexical units. The first inference configuration parameter includes the number G of the one or more sets of draft lexical units and the number M of lexical units in each set. The second acquisition module 904 is configured to: acquire X sets of static draft lexical units from at least one static N-gram library and Y sets of dynamic draft lexical units from a dynamic N-gram library according to the first inference configuration parameter. Where X + Y ≤ G, and the number of lexical units in each set of draft lexical units is less than or equal to M. Each static N-gram library in the at least one static N-gram library includes lexical units included in a specific scenario. The dynamic N-gram library includes draft lexical units accepted after each inference operation.

[0115] In some embodiments of this disclosure, the adjustment module 910 includes a second determining module and a second sub-adjustment module. The second determining module is configured to determine, after performing an inference operation, the acceptance rate of X groups of static draft words or the acceptance rate of Y groups of dynamic draft words. The second sub-adjustment module is configured to reduce the proportion of X groups of static draft words in one or more groups of draft words when the acceptance rate of the group with the highest acceptance rate among X groups of static draft words is less than a first threshold after K consecutive inference operations, or when the acceptance rate of the group with the highest acceptance rate among Y groups of dynamic draft words is greater than a second threshold after K consecutive inference operations; or to increase the proportion of X groups of static draft words in one or more groups of draft words when the acceptance rate of the group with the highest acceptance rate among X groups of static draft words is greater than the second threshold after K consecutive inference operations, or when the acceptance rate of the group with the highest acceptance rate among Y groups of dynamic draft words is less than the first threshold after K consecutive inference operations. The second threshold is greater than or equal to the first threshold. K is greater than or equal to 1.

[0116] In some embodiments of this disclosure, the inference apparatus further includes a generation module. The generation module is configured to generate Z sets of Jacobian lexical units using the Jacobian method. The execution module 906 is configured to perform inference operations in parallel based on the inference lexical units, multiple draft lexical units, and the Z sets of Jacobian lexical units. The dynamic N-gram library also includes Jacobian lexical units accepted after each inference operation.

[0117] In summary, the inference method, inference apparatus, electronic device, and computer program product for artificial intelligence models according to embodiments of this disclosure dynamically adjust the first inference configuration parameters based on the execution information of the inference operation and can generate multiple accepting terms in one inference iteration. Therefore, inference operations can be performed more effectively, thereby reducing the number of inference iterations, reducing inference latency, and improving inference performance.

[0118] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0119] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), and any suitable combination thereof. The computer-readable storage medium as used herein is not to be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0120] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0121] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0122] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0123] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0124] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0125] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0126] Unless otherwise expressly indicated by the context, the singular form of words used herein and in the appended claims includes the plural form, and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Where the term “example” is used herein, particularly when it follows a set of terms, the “example” is merely exemplary and illustrative and should not be considered exclusive or pervasive.

[0127] Further aspects and scope of adaptation become apparent from the description provided herein. It should be understood that various aspects of this application may be implemented individually or in combination with at least one other aspect. It should also be understood that the descriptions and specific embodiments herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0128] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A reasoning method for artificial intelligence models, characterized in that, The method includes: Obtain the reasoning lexical units for reasoning operations; Multiple draft lexical units are obtained based on the first inference configuration parameters; The reasoning operation is performed in parallel based on the inference lexical units and the multiple draft lexical units. After the inference operation is completed, the execution information of the inference operation is obtained; and When the execution information of the inference operation meets the preset conditions, the first inference configuration parameter is adjusted to the second inference configuration parameter.

2. The method according to claim 1, characterized in that, The plurality of draft lexical units includes one or more sets of draft lexical units, and the execution information of the inference operation is the acceptance rate of the set of draft lexical units with the highest acceptance rate. When the execution information of the inference operation meets the preset conditions, adjusting the first inference configuration parameter to the second inference configuration parameter includes: After performing the inference operation, determine the acceptance rate of the one or more sets of draft terms; When the acceptance rate of the group with the highest acceptance rate among the one or more groups of draft words is less than a first threshold after K consecutive inference operations, the number of the multiple draft words is reduced; or When the acceptance rate of the group with the highest acceptance rate among the group or multiple groups of draft words is greater than the second threshold after K consecutive inference operations, the number of the multiple draft words is increased. Wherein the second threshold is greater than or equal to the first threshold, and K is greater than or equal to 1.

3. The method according to claim 2, characterized in that, The reduction of the number of the plurality of draft lexical units includes at least one of the following: reducing the number G of the group or plurality of draft lexical units, or reducing the number M of each group of draft lexical units; and / or The increase in the number of the plurality of draft lexical units includes at least one of the following: increasing the number of groups G of the one or more groups of draft lexical units, or increasing the number of lexical units M of each group of draft lexical units.

4. The method according to claim 1, characterized in that, The plurality of draft lexical units includes one or more groups of draft lexical units, and the first inference configuration parameter includes the number of groups G of the one or more groups of draft lexical units and the number of lexical units M in each group of draft lexical units; The acquisition of multiple draft lexical units based on the first inference configuration parameters includes: According to the first inference configuration parameters, X sets of static draft lexical units are obtained from at least one static N-gram grammar library, and Y sets of dynamic draft lexical units are obtained from a dynamic N-gram grammar library, wherein X+Y≤G, the number of lexical units in each set of draft lexical units is less than or equal to M, each static N-gram grammar library in the at least one static N-gram grammar library includes lexical units included in a specific scenario, and the dynamic N-gram grammar library includes draft lexical units accepted after each inference operation is performed.

5. The method according to claim 4, characterized in that, When the execution information of the inference operation meets the preset conditions, adjusting the first inference configuration parameter to the second inference configuration parameter includes: After performing the inference operation, determine the acceptance rate of the X group of static draft words or the acceptance rate of the Y group of dynamic draft words. When the acceptance rate of the group with the highest acceptance rate among the X groups of static draft lexics is less than the first threshold after K consecutive inference operations, or when the acceptance rate of the group with the highest acceptance rate among the Y groups of dynamic draft lexics is greater than the second threshold after K consecutive inference operations, the proportion of the X groups of static draft lexics in the one or more groups of draft lexics is reduced; or When the acceptance rate of the group with the highest acceptance rate in the X groups of static draft words is greater than the second threshold after K consecutive inference operations, or when the acceptance rate of the group with the highest acceptance rate in the Y groups of dynamic draft words is less than the first threshold after K consecutive inference operations, the proportion of the X groups of static draft words in the one or more groups of draft words is increased. Wherein the second threshold is greater than or equal to the first threshold, and K is greater than or equal to 1.

6. The method according to claim 4 or 5, characterized in that, The method further includes: generating a Z group of Jacobi lexical units using the Jacobi method; The parallel execution of the inference operation based on the inference lexical and the plurality of draft lexical includes: performing the inference operation in parallel based on the inference lexical, the plurality of draft lexical and the Z group of Jacobi lexical; The dynamic N-gram library also includes Jacobi lexicals that are accepted after each inference operation.

7. An inference device for an artificial intelligence model, comprising: The first acquisition module is configured to acquire the reasoning lexical units of the reasoning operation; The second acquisition module is configured to acquire multiple draft lexical units based on the first inference configuration parameters; An execution module is configured to execute the inference operation in parallel based on the inference lexical and the plurality of draft lexical; The third acquisition module is configured to acquire the execution information of the inference operation after the inference operation is completed; as well as An adjustment module is configured to adjust the first inference configuration parameter to a second inference configuration parameter when the execution information of the inference operation meets preset conditions.

8. The apparatus according to claim 7, characterized in that, The plurality of draft lexical units includes one or more sets of draft lexical units, and the execution information of the inference operation is the acceptance rate of the set of draft lexical units with the highest acceptance rate. The adjustment module includes: The first determining module is configured to determine the acceptance rate of the one or more sets of draft words after the reasoning operation is performed; The first sub-adjustment module is configured to reduce the number of draft words when the acceptance rate of the group with the highest acceptance rate among the group or groups of draft words is less than a first threshold after K consecutive inference operations; or to increase the number of draft words when the acceptance rate of the group with the highest acceptance rate among the group or groups of draft words is greater than a second threshold after K consecutive inference operations. Wherein the second threshold is greater than or equal to the first threshold, and K is greater than or equal to 1.

9. The apparatus according to claim 8, characterized in that, The first sub-adjustment module is configured as follows: When the acceptance rate of the group with the highest acceptance rate among the one or more groups of draft words is less than the first threshold after K consecutive inference operations, perform at least one of the following: reduce the number of groups G of the multiple draft words, or reduce the number of words M in each group of draft words; and / or When the acceptance rate of the group with the highest acceptance rate among the group or groups of draft words is greater than the second threshold after K consecutive inference operations, perform at least one of the following: increase the number of groups of the plurality of draft words G, or increase the number of words M in each group of draft words.

10. The apparatus according to claim 7, characterized in that, The plurality of draft lexical units includes one or more groups of draft lexical units, and the first inference configuration parameter includes the number of groups G of the one or more groups of draft lexical units and the number of lexical units M in each group of draft lexical units; The second acquisition module is configured to: acquire X sets of static draft lexical units from at least one static N-gram grammar library according to the first inference configuration parameters, and acquire Y sets of dynamic draft lexical units from a dynamic N-gram grammar library; Where X+Y≤G, the number of lexical units in each draft lexical unit is less than or equal to M, each static N-gram lexical unit in the at least one static N-gram lexical unit includes lexical units included in a specific scenario, and the dynamic N-gram lexical unit includes draft lexical units accepted after each inference operation is performed.

11. The apparatus according to claim 10, characterized in that, The adjustment module includes: The second determining module is configured to determine the acceptance rate of the X group of static draft words or the acceptance rate of the Y group of dynamic draft words after the inference operation is completed. The second sub-adjustment module is configured to: reduce the proportion of the X group of static draft words in one or more groups of draft words when the acceptance rate of the group with the highest acceptance rate in the X group of static draft words is less than a first threshold after K consecutive inference operations, or when the acceptance rate of the group with the highest acceptance rate in the Y group of dynamic draft words is greater than a second threshold after K consecutive inference operations; or increase the proportion of the X group of static draft words in one or more groups of draft words when the acceptance rate of the group with the highest acceptance rate in the X group of static draft words is greater than the second threshold after K consecutive inference operations, or when the acceptance rate of the group with the highest acceptance rate in the Y group of dynamic draft words is less than the first threshold after K consecutive inference operations. Wherein the second threshold is greater than or equal to the first threshold, and K is greater than or equal to 1.

12. The apparatus according to claim 10 or 11, characterized in that, The reasoning device further includes: The generation module is configured to generate Z sets of Jacobian lexical units using the Jacobian method; The execution module is configured to perform the inference operation in parallel based on the inference lexical, the plurality of draft lexical, and the Z group of Jacobian lexical. The dynamic N-gram library also includes Jacobi lexicals that are accepted after each inference operation.

13. A computer program product tangibly stored on a non-transient computer-readable medium and comprising machine-executable instructions for performing the method according to any one of claims 1-6.

Images