Speculative inference method and electronic device

Abstract

The embodiments of the present application relate to the technical field of artificial intelligence, and particularly relate to a speculative inference method and an electronic device. By means of the method, the efficiency of speculative inference can be improved. The method is applied to an electronic device, wherein a processor of the electronic device comprises a first processing unit and a second processing unit, which are used for processing tasks in parallel. The method specifically comprises: during the first round of draft task, a first processing unit inputting into a draft model a token vector corresponding to a first token, so as to generate a plurality of first draft tokens to be verified, and transmitting to a second processing unit a first token sequence, which includes the plurality of first draft tokens, wherein the first token is generated by an original large model on the basis of a prompt in a prefill stage; during the second round of draft task, the first processing unit inputting into the draft model a token vector corresponding to the first token sequence, so as to generate a plurality of second draft tokens to be verified; and during the first round of verification task, the second processing unit inputting the token vector, which corresponds to the first token sequence, into the original large model corresponding to the draft model, verifying the plurality of first draft tokens by means of the original large model to obtain a verification result, and determining a third token sequence on the basis of the verification result, wherein the third token sequence is used as an input for the next round of draft task or verification task, and the first round of verification task and the second round of draft task are executed in parallel.

Description

Speculative reasoning methods and electronic devices

[0001] This application claims priority to Chinese patent application filed on January 15, 2025, with application number 202510066211.3 and entitled "Speculative Reasoning Method and Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of AI technology, and more particularly to a speculative reasoning method and electronic device. Background Technology

[0003] Speculative decoding, also known as speculative inference or speculative sampling, is used to improve the inference efficiency of natural language processing models such as Large Language Models (LLMs) during the decoding phase. The speculative inference process can include a draft phase and a verification phase. In the draft phase, a lighter model than the original LLM model is used as the draft model, and multiple draft tokens to be verified are continuously generated using autoregressive sampling. In the verification phase, the original LLM model is used to verify the multiple draft tokens to be verified, thereby improving the inference efficiency of the decoding phase.

[0004] Current speculative reasoning schemes are inefficient and need improvement. Summary of the Invention

[0005] This application provides a speculative reasoning method and electronic device. Based on a heterogeneous computing architecture, it enables the parallel execution of the verification stage and the drafting stage. The drafting stage and the verification stage do not need to wait for each other in terms of execution timing, thereby improving the efficiency of speculative reasoning.

[0006] In a first aspect, embodiments of this application provide a speculative reasoning method applied to an electronic device. The processor of the electronic device includes a first processing unit and a second processing unit for parallel processing tasks. The method includes: in a first draft task, the first processing unit inputs the word vector corresponding to the first token into a draft model to generate multiple first draft tokens to be verified; and transmitting a first token sequence including the multiple first draft tokens to the second processing unit; the first token is generated by the original large model in the prefill stage based on the prompt; in a second draft task, the first processing unit inputs the word vector corresponding to the first token sequence into the draft model to generate multiple second draft tokens to be verified; in a first verification task, the second processing unit inputs the word vector corresponding to the first token sequence into the original large model corresponding to the draft model, and verifies the multiple first draft tokens through the original large model to obtain a verification result; and determining a third token sequence based on the verification result; the third token sequence is used as input for the next draft task or verification task; the first verification task and the second draft task are executed in parallel.

[0007] The speculative reasoning method proposed in this application is based on a heterogeneous computing architecture and a mechanism that allows verification and drafting tasks to be executed in parallel. The verification and drafting stages do not need to wait for each other, thus improving the efficiency of speculative reasoning.

[0008] In one possible implementation, the verification of multiple first draft tokens using the original large model to obtain verification results includes: running the original large model to decode the multiple first draft tokens in parallel to obtain a probability sequence corresponding to the multiple first draft tokens; and verifying each draft token in the multiple first draft tokens sequentially according to the probability sequence to obtain a verification result.

[0009] During the verification phase, by leveraging the parallel decoding capability of the original large model's prefill phase, a single prediction can be made to obtain the predicted probability of the next token for each token in the token sequence. Thus, based on a single prediction result (a probability sequence, such as logit or a normalized probability distribution), the verification of multiple draft tokens can be achieved. In other words, the verification of multiple draft tokens can be achieved with a single prediction delay.

[0010] In one possible implementation, determining the third token sequence based on the verification result includes: if the verification result indicates that multiple first draft tokens have partially passed verification, taking the token sequence corresponding to the probability sequence as the third token sequence; transmitting the third token sequence to the first processing unit; after determining the third token sequence, the method further includes: in the third round of draft task, the first processing unit inputs the word vectors corresponding to the third token sequence into the draft model to generate multiple third draft tokens to be verified, and transmitting the fourth token sequence including the multiple third draft tokens to the second processing unit.

[0011] During verification, if a draft token that fails verification is found in the token sequence, the current verification round ends, and the token sequence predicted by the original large model is used as the input for the next round of draft model. If the first processing unit is currently executing the next round of draft task, upon receiving the third token sequence transmitted from the second processing unit, the current draft task is immediately stopped, and the third token sequence is re-inputted to generate draft tokens. In this way, while improving efficiency through parallel speculative inference mechanisms, the accuracy of prediction performance or inference performance can be guaranteed by leveraging the performance of the original large model.

[0012] In one possible implementation, after generating multiple second draft tokens to be verified in the second draft task, the method further includes: transmitting a second token sequence including multiple second draft tokens to a second processing unit; determining a third token sequence based on the verification result, including: if the verification result indicates that all multiple first draft tokens have passed verification, verifying the first second draft token in the second token sequence based on the last token in the probability sequence; if the first second draft token has passed verification, using the second token sequence as the third token sequence; after determining the third token sequence, the method further includes: in the second verification task, inputting the word vector corresponding to the third token sequence into the original large model, and verifying the multiple second draft tokens through the original large model to obtain the verification result.

[0013] During the validation of the original large model, the token sequence predicted in the prefill phase has one more token (the last token) than the input draft token sequence to be validated. This last token can be used to validate the first token among the multiple draft tokens generated in the next round of the draft task. This is because the draft task does not wait for the previous round of validation to finish. Therefore, one possible scenario is that during the t-th round of validation, the second processing unit can receive the draft token sequence to be validated from the draft task of the (t+1)-th round. If the last token predicted by the original large model during the t-th round of validation matches the first draft token of the (t+1)-th round, it indicates that the generation result of the draft model in the (t+1)-th round has a certain degree of accuracy and can be used as the input for validation in the (t+1)-th round. This mechanism can ensure that the accuracy of the draft tokens generated in each round is improved under the parallel operation mechanism.

[0014] In one possible implementation, after generating multiple second draft tokens to be verified in the second draft task, the method further includes: transmitting a second token sequence including multiple second draft tokens to a second processing unit; determining a third token sequence based on the verification result, including: if the verification result indicates that all multiple first draft tokens have passed verification, verifying the first second draft token in the second token sequence based on the last token in the probability sequence; if the first second draft token fails verification, using the token sequence corresponding to the probability sequence as the third token sequence; transmitting the third token sequence to a first processing unit; after determining the third token sequence, the method further includes: in the third draft task, the first processing unit inputs the word vector corresponding to the third token sequence into the draft model to generate multiple third draft tokens to be verified, and transmitting a fourth token sequence including multiple third draft tokens to the second processing unit.

[0015] Similarly, if the last token predicted by the original large model in round t is inconsistent with the first draft token generated by the draft task in round t+1, it means that the accuracy of the draft token generated in round t+1 is not high. In this case, the latest token sequence predicted by the current original large model is used as the input of the draft task in round t+1, and the draft task in round t+1 is re-executed to improve the accuracy of the token sequence predicted in each round.

[0016] In one possible implementation, before verifying multiple first draft tokens using the original large model, the method further includes: a second processing unit that reads the value of a verification position; the verification position is used to indicate the position of the first draft token among the multiple draft tokens to be verified in the first token sequence; verifying the N draft tokens includes: verifying from the position indicated by the verification position using the original large model; after determining the third token sequence, the method further includes: the second processing unit that updates the value of the verification position according to the third token sequence.

[0017] In practical applications, the length of a token sequence may reach hundreds or thousands of tokens. If each verification starts from the beginning, repetitive and useless verification operations are required. Updating the verification position allows verification to start only from the position that has not been verified before, avoiding invalid verification work and improving verification efficiency.

[0018] In one possible implementation, updating the value of the verification position according to the third token sequence includes: updating the value of the verification position to M+1 based on the number of tokens in the third token sequence being M.

[0019] In one possible implementation, before the first processing unit inputs the word vector corresponding to the first token into the draft model, the method further includes: the first processing unit reading the value of the draft length; the draft length is used to indicate the number of draft tokens to be verified generated by the draft model in this round of draft task; after obtaining the verification result, the method further includes: the second processing unit adjusting the value of the draft length according to the verification result.

[0020] In one possible implementation, adjusting the draft length value based on the verification result includes: when the verification result indicates that M out of N draft tokens have been verified successfully, and 0 ≤ M < N, adjusting the draft length value to X, where X ≤ N; and when the verification result indicates that all N draft tokens have been verified successfully, adjusting the draft length value to Y, where Y ≥ N.

[0021] The adjustment principle can be that the higher the validation pass rate, the longer the draft should be, and the lower the validation pass rate, the shorter the draft should be. For example, if the validation pass rate exceeds a preset upper threshold, the draft length is increased by 1; if it is below a preset lower threshold, the draft length is decreased by 1. The draft length can be adjusted according to the validation results, which can increase the flexibility and controllability of draft token generation.

[0022] In one possible implementation, before verifying multiple first draft tokens using the original large model, the method further includes: a second processing unit that reads the value of a confidence factor; the confidence factor is a threshold used to control whether the verification of a draft token passes; and verifying each draft token in the multiple first draft tokens sequentially according to the probability sequence, including: calculating the difference between the probability values ​​of tokens corresponding to the same position in the first probability sequence predicted by the original large model and the second probability sequence predicted by the draft model, and obtaining the verification result based on the difference and the value of the confidence factor; after obtaining the verification result, the method further includes: adjusting the value of the confidence factor according to the verification result.

[0023] Adjusting the confidence factor allows for adaptation to different task types. Some tasks are highly complex and require diverse responses; in these cases, the confidence factor can be adjusted downwards or upwards. Other tasks are more general, allowing for adjustments as well. The specific adjustment depends on how the confidence factor is determined. If the confidence factor is used to compare the difference between the confidence factor and the probability value, then for complex tasks, the confidence factor should be lowered, while for relatively simple tasks, it should be increased. If the confidence factor is directly compared to the probability value, then for complex tasks, the confidence factor should be increased, while for relatively simple tasks, it should be decreased. By adjusting the confidence factor, the applicability to different task types can be controlled.

[0024] In one possible implementation, the draft model is a self-draft model obtained by skipping specified intermediate layers based on the original large model; before executing each round of draft tasks, the method further includes: a first processing unit, reading the value of the skip vector; the skip vector is used to indicate the intermediate layers skipped by the draft model relative to the original large model; when executing each round of draft tasks, the method includes: skipping the corresponding intermediate layers in the original large model according to the skip vector, and executing the current round of draft tasks.

[0025] It should be noted that the implementation using its own draft model requires a skip vector. In the implementation using an independent draft model to achieve the parallel inference mechanism described above, a skip vector is not needed.

[0026] In one possible implementation, the first processing unit is an in-memory computing (CIM) unit; the second processing unit is a neural network processing (NPU) unit.

[0027] The heterogeneous computing architecture of the in-memory computing unit (CIM) and the NPU, compared to the traditional CPU architecture, can reduce the power consumption caused by accessing memory, thereby alleviating the efficiency bottleneck caused by the memory wall problem.

[0028] Secondly, embodiments of this application also provide an electronic device, the electronic device comprising: a processor, the processor being configured to execute a computer program or instructions in a memory to implement the method as described in any of the first aspects above.

[0029] Thirdly, embodiments of this application also propose a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method as described in any of the first aspects above.

[0030] Fourthly, embodiments of this application also propose a computer program product containing instructions that, when run by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of the first aspects above.

[0031] Fifthly, embodiments of this application also provide a computer-readable storage medium including computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of the first aspects above.

[0032] In a sixth aspect, embodiments of this application also provide a chip system, including: a communication interface for inputting and / or outputting data; and a processor for executing a computer-executable program, causing a device equipped with the chip system to perform the method as described in any one of the first aspects above. Attached Figure Description

[0033] Figure 1a is a schematic diagram showing that the drafting stage and the verification stage are executed sequentially in time in related technologies;

[0034] Figure 1b is a schematic diagram of the process in which the drafting stage and the verification stage are executed sequentially in time in the relevant technology;

[0035] Figure 2 is a schematic diagram of the structure of the electronic device (terminal device) provided in the embodiment of this application;

[0036] Figure 3 is a schematic diagram of the structure of the electronic device (server) provided in an embodiment of this application;

[0037] Figure 4 is a schematic diagram of the heterogeneous computing architecture of the speculative reasoning method provided in the embodiments of this application;

[0038] Figure 5 is another example diagram of the heterogeneous computing architecture of the speculative inference method provided in the embodiments of this application;

[0039] Figure 6 is an example diagram of the system architecture of the speculative reasoning method provided in the embodiments of this application;

[0040] Figure 7 is a flowchart illustrating the parallel execution of the drafting and verification stages of the speculative reasoning method provided in this application embodiment;

[0041] Figure 8 is a schematic diagram of the token update sequence in the speculative reasoning method provided in the embodiments of this application;

[0042] Figure 9 is a schematic diagram of another example of updating the token sequence in the speculative reasoning method provided in the embodiments of this application;

[0043] Figures 10a to 10c are interactive schematic diagrams showing the parallel execution of draft tasks and verification tasks in different rounds in some embodiments of the speculative reasoning method provided in this application.

[0044] Figure 11 is a flowchart illustrating the speculative reasoning method provided in an embodiment of this application;

[0045] Figure 12 is a flowchart illustrating some embodiments of the speculative reasoning method provided in this application. Detailed Implementation

[0046] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of, and not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0047] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more.

[0048] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0049] In the embodiments of this application, the words "exemplary" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options.

[0050] The reasoning process of a Large Language Model (LLM) includes a prefill stage and a decoding stage. In the decoding stage, each token needs to be decoded serially. Due to memory access speed limitations, generating each token requires transferring a large number of parameters from the storage unit to the computing unit. Therefore, memory access bandwidth becomes a bottleneck for improving inference efficiency.

[0051] Speculative inference is one method to address the inference efficiency bottleneck caused by memory access bandwidth. Speculative inference typically uses two models: a large original model, such as the LLM model, and a lightweight draft model with a much smaller parameter size. In decoding using the original LLM model, some tokens are relatively easy to decode, while others are relatively difficult. In speculative inference, simple token drafting tasks can be efficiently handled by the draft model, while difficult token drafting tasks can be handled by the LLM model. Compared to the original LLM model, the draft model not only requires less computation but, more importantly, has a significantly reduced number of parameters, thus reducing memory access requirements.

[0052] A token is a fundamental unit in natural language processing, typically referring to the breakdown of text into smaller units, such as words, subwords, or characters. Token segmentation divides the original text into manageable units so that the model can effectively understand and process it. Tokens can be words or punctuation marks, which are then converted into vectors for use in the model's computation and processing.

[0053] Speculative reasoning can be divided into a draft stage and a verification stage. In the draft stage, the draft model performs autoregressive serial sampling. The time required for one inference operation during the decoding stage of the original large model (which is also the time required for the original large model to generate one token) allows the draft model to continuously generate multiple draft tokens to be verified. In the verification stage, the original large model verifies these multiple draft tokens to ensure the accuracy of the output token sequence. The verification process involves parallel decoding of multiple draft tokens, allowing multiple draft tokens to be verified simultaneously.

[0054] In related technologies, the verification and drafting phases of speculative reasoning schemes are generally executed sequentially. As shown in Figure 1a, a complete speculative reasoning process can include several phases such as configuration preparation, drafting, verification, and output. Among them, the drafting, verification, and output are executed cyclically until the reasoning task is completed. Specifically, as shown in Figure 1b, after determining the original large model and the corresponding draft model, based on the input prompt, the original LLM large model performs a Prefill phase, generating the first token in the token sequence (i.e., the first token shown in Figure 1b). The first token is input into the draft model, continuously generating multiple draft tokens to be verified. For example, in Figure 1b, the solid box represents the token sequence waiting to be verified in this round. Next, the original LLM large model verifies the multiple draft tokens in the token sequence. The verified draft tokens are concatenated with the first token to obtain a new token sequence. This token sequence is output, and the new token sequence is input into the draft model to execute the next round of drafting and verification processes... In this way, the drafting, verification, and output processes are executed in a loop until a termination condition is encountered, speculative inference stops, and the latest token sequence obtained is output.

[0055] As can be seen from Figures 1a and 1b, speculative reasoning in related technologies is based on the same type of neural processing unit (NPU). The draft and verification stages are executed sequentially. The verification stage needs to wait for the draft model to generate multiple draft tokens to be verified before it can be verified. The next round of draft generation needs to wait for the verification stage to complete the verification of the multiple draft tokens generated in this round before it can be executed. The draft and verification stages need to wait for each other, which affects the efficiency of speculative reasoning.

[0056] Based on the above introduction and specific analysis, speculative reasoning in related technologies may have the following problems:

[0057] Delay issue: In the case of sequential drafting and verification, verification can only be performed after the drafting stage is completed, and the next round of drafting can only be performed after the verification stage is completed. The two stages wait for each other, which limits the inference rate of the system and increases response latency.

[0058] Resource waste issue: Multiple draft tokens generated in the draft stage may fail verification in some or all of them. This may result in no benefit from the execution of tasks in the draft stage. In the serial mechanism, other stages are suspended while the draft model is executing draft tasks in the draft stage. The draft stage occupies NPU resources alone during this period, resulting in low resource utilization and resource waste.

[0059] It is evident that speculative reasoning in related technologies suffers from problems such as low reasoning efficiency and low resource utilization.

[0060] In view of this, this application proposes a speculative reasoning method and an electronic device, and proposes a speculative reasoning mechanism based on a heterogeneous computing architecture running in parallel. The heterogeneous computing architecture includes a first processing unit and a second processing unit. Under the parallel speculative reasoning mechanism, the first processing unit and the second processing unit can process tasks in parallel. For example, the first processing unit is used to execute tasks in the draft stage, and the second processing unit can be used to execute tasks in the verification stage. The draft task in round t+1 can be executed in parallel with the verification task in round t. In this way, during the iterative execution of the draft stage and the verification stage, there is a situation where the draft stage and the verification stage are executed in parallel. The draft stage and the verification stage do not need to wait for each other, thereby improving the efficiency of speculative reasoning.

[0061] The speculative reasoning method proposed in this application can be applied to various application scenarios that utilize large models such as LLM to perform tasks, thereby improving the reasoning efficiency or response speed of large models. For example, it can be applied to application scenarios such as natural language processing, content creation, writing assistance, dialogue systems and virtual assistants, knowledge base construction and question-answering systems, speech recognition and speech synthesis, as well as data analysis and intelligent decision-making, personalized recommendation systems, education and online learning, medical and health management, social media and public opinion analysis, legal and compliance management, and many other practical application scenarios.

[0062] One typical application scenario involves improving the inference efficiency and reducing response latency of large edge models through parallel speculative inference. Large edge models refer to large-scale deep learning models running locally on the device. In other application scenarios, it can also be used to optimize speculative inference for large cloud models, thereby improving the efficiency of speculative inference. Large cloud models refer to large models deployed in the cloud.

[0063] In application scenarios involving speculative inference on large edge models, the electronic devices proposed in this application can be terminal devices such as smartphones. For example, Figure 2 is a schematic diagram of the structure of an electronic device provided in some embodiments of this application; this electronic device can be a smartphone that supports running LLM or other large models.

[0064] Specifically, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0065] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0066] Processor 110 is a processor with a heterogeneous computing architecture and may include a first processing unit and a second processing unit, which are processing units of different types. Alternatively, in addition to the first and second processing units, processor 110 may also include one or more of the following processors: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. The different processing units may be independent devices or integrated into one or more processors.

[0067] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0068] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0069] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0070] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0071] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0072] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 via the power management module 141.

[0073] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0074] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0075] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0076] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0077] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0078] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0079] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-CDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS). In other words, electronic device 100 has positioning and wireless communication functions.

[0080] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0081] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0082] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0083] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0084] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0085] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0086] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0087] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0088] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0089] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0090] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0091] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0092] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0093] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0094] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0095] The electronic device 100 proposed in this application embodiment can also be other terminal devices or Internet of Things (IoT) devices. For example, other terminal devices can be tablet PCs, laptops, desktop computers, wearable devices, augmented reality (AR) / virtual reality (VR) devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc. This application embodiment does not impose any limitations on the specific type of electronic device.

[0096] In application scenarios involving speculative inference on large cloud-based models, the electronic device proposed in this application embodiment can be a server, which can be a physical machine or a virtual machine. For example, Figure 3 is a schematic diagram of the server structure in one embodiment of this application. As shown in Figure 3, the server 200 may include: one or more processors 210, a communication interface 220, a memory 230, and a communication bus 240 connecting different components (including the memory 230, the communication interface 220, and the processor 210).

[0097] Communication bus 240 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, or a local bus using any of the various bus architectures. For example, communication bus 240 may include, but is not limited to, an industry standard architecture (ISA) bus, a micro channel architecture (MCA) bus, an enhanced ISA bus, a video electronics standards association (VESA) local bus, and a peripheral component interconnection (PCI) bus.

[0098] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0099] Memory 230 may include a computer system readable medium in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Memory 230 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the speculative reasoning method provided in the embodiments of this application.

[0100] A program / utility having a set (at least one) of program modules can be stored in memory 230. Such program modules include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this application.

[0101] The processor 210 executes various functional applications and data processing by running programs stored in the memory 230, such as implementing the speculative reasoning method provided in the embodiments of this application.

[0102] It should be understood that the processor 210 in the server 200 shown in Figure 3 is a processor built on a heterogeneous computing architecture, including at least a first processing unit and a second processing unit. Alternatively, in addition to the first processing unit and the second processing unit, the processor 210 may also include a central processing unit (CPU), and further include other types of processors, such as a graphics processing unit (GPU).

[0103] Specifically, the first processing unit can be a processing unit based on in-memory computing technology, that is, the first processing unit can be one of Computing In Memory (CIM) unit, Processing In Memory (PIM) unit, or Processing Near Memory (PNM) unit. Alternatively, the first processing unit can be a CPU or a GPU.

[0104] The second processing unit can be either an NPU or a GPU. The first and second processing units are different types of processing units. For example, when the first processing unit is a GPU, the second processing unit is an NPU; when the first processing unit is an NPU, the second processing unit can be a GPU.

[0105] For example, in some embodiments, a typical example of a heterogeneous computing architecture is shown in Figure 4, in which the first processing unit is a CIM and the second processing unit is an NPU.

[0106] Exemplary examples, in some other embodiments, the heterogeneous computing architecture may also employ one of the following architectures: Compute Architecture for Neural Networks (CANN) or Compute Unified Device Architecture (CUDA). For instance, as shown in Figure 6, the heterogeneous computing architecture may include a host side and a device side. The host side includes at least one central processing unit (CPU), and the device side includes a graphics processing unit (GPU), a neural network processing unit (NPU), or a tensor processing unit (TPU) (only GPUs are shown in the figure). The first processing unit may be the CPU on the host side. The second processing unit may be the GPU or NPU on the device side.

[0107] For example, the speculative reasoning method provided in this application embodiment can be implemented based on the system architecture shown in FIG6.

[0108] As shown in Figure 6, the system architecture may include a server 301 and one or more terminal devices 302 that communicate remotely with the server 301. The LLM large model can run on the terminal device 302 or on the server 301. Figure 6 illustrates the example of terminal device 302 supporting the deployment of the terminal-side large model. The processor in terminal device 302 is a processor using a heterogeneous computing architecture.

[0109] A 301 redirect can be applied to a standalone server or a server cluster, such as a Highly Available (HA) cluster, a Load Balance (LB) cluster, or a High Performance Computing (HPC) cluster.

[0110] Figure 6 illustrates the deployment of the LLM large model on the terminal device side. In practical applications, the LLM large model can also be deployed on the cloud side. Taking the scenario shown in Figure 6 as an example, the processor of the terminal device 302 adopts a heterogeneous computing architecture, such as the heterogeneous computing architecture shown in Figure 4 or Figure 5. To implement the speculative inference method proposed in this application embodiment, the cloud side can provide installation packages for the LLM large model and the corresponding draft model, and distribute them to the terminal device 302. The terminal device 302 installs and runs the LLM large model and the draft model to implement the speculative inference method proposed in this application embodiment, which allows the verification stage and the draft stage to be executed in parallel.

[0111] In other scenarios, the large LLM model can be deployed on server 301. The processor in server 301 is built on a heterogeneous computing architecture, and a draft model corresponding to the large LLM model is also deployed on server 301. Terminal device 302 can receive user input questions and then upload the user questions to server 301. Server 301 runs the large LLM model and the draft model to realize the speculative reasoning method proposed in this application embodiment, which allows the verification stage and the draft stage to be executed in parallel.

[0112] Specifically, taking the original large model as an LLM large model as an example, based on the deployment of the LLM large model and draft model in the terminal device as shown in Figure 6, taking the intelligent question answering scenario as an example, the terminal device 302 can receive the user's input question, generate the corresponding prompt according to the user's question, and input it into the original LLM large model. The LLM large model performs the Prefill stage, predicts the first token in the answer according to the prompt, and then inputs the first token into the draft model. The draft model continuously predicts multiple draft tokens to be verified. The first token and multiple tokens to be verified form a token sequence to be verified, which is input into the LLM large model. The LLM large model performs a parallel decoding task similar to the Prefill stage to obtain a probability sequence. The elements in the probability sequence represent the predicted probability of the next token corresponding to each of the multiple draft tokens to be verified. According to the probability sequence, each draft token is verified to see if it meets the preset requirements. If it does, the verification is passed, and the verification of the next draft token continues. If not, then stop verifying subsequent draft tokens, and use the token sequence corresponding to the probability sequence obtained from the current LLM large model as the verified correct token sequence, overwriting the original token sequence to be verified, and use it as the input for the next round of draft tasks. The verified token sequence can be converted into a text sequence and displayed on the interactive interface of terminal device 302. If each token in multiple draft tokens to be verified passes verification, then the first token generated in the next round of draft tasks can be verified based on the last token in the probability sequence. Based on the verification result, the verified token sequence is determined and used as the input for the next draft task or verification task.

[0113] In the scenario where the LLM large model is deployed on server 301, the terminal device can upload the user's input question to server 301. Server 301 generates a prompt based on the user's question and inputs the prompt into the original LLM large model. The LLM large model performs a Prefill phase, predicting the first token in the answer based on the prompt. Next, the first token is input into a draft model, which continuously predicts multiple draft tokens to be verified. The LLM large model then verifies the draft tokens. Based on the verification results, a sequence of verified tokens is obtained. This token sequence can be converted into a text sequence and sent to terminal device 302 for display on the interactive interface of terminal device 302. Furthermore, the sequence of verified tokens is input back into the draft model to execute the next round of draft tasks, and so on, until the user's question is answered or a command to stop prediction is received.

[0114] It should be noted that in speculative reasoning, the draft model and the main model correspond to each other. The draft model in the embodiments of this application can be an independent drafting model or a self-drafting model.

[0115] The independent draft model can be obtained in the following ways: Method 1: Using the same training data, train both a large model with a large parameter scale and a lightweight model with a smaller parameter scale corresponding to the large model. The resulting large model is the original large model, and the resulting lightweight model is the independent draft model. Method 2: Perform knowledge distillation on the original large model to obtain the independent draft model.

[0116] The self-draft model is a lightweight draft model obtained by skipping specified intermediate layers of the original large model or skipping specified calculations.

[0117] The following are some specific examples.

[0118] In some embodiments, the heterogeneous computing architecture may include an in-memory computing unit and a neural network processing unit. For example, the in-memory computing unit may be an in-memory computing unit (CIM). Taking the first processing unit as CIM and the second processing unit as NPU as an example, a feasible exemplary scheme based on the functional module distribution and interaction between modules of this heterogeneous computing architecture is shown in Figure 7. During speculative reasoning, CIM is used to perform the task of quickly generating draft tokens, and NPU is used to perform the task of efficiently verifying the draft tokens generated by CIM.

[0119] For example, the CIM deploys a control module and a generation module. The control module controls the generation module in the CIM to start or end draft tasks according to user instructions. The generation module can run a draft model, inputting the first token or the current token sequence generated or verified in the previous draft task into the draft model, executing various calculation tasks in the draft model, and generating multiple draft tokens to be verified. The generated multiple draft tokens are concatenated with the first token or the token sequence verified in the previous round to obtain the token sequence to be verified, which is then transmitted to the NPU. The generation module can be a functional circuit module obtained by combining hardware and software, which can be configured with a memory for caching draft tokens and token sequences, such as for caching the first token, the verified token sequence, or the token sequence to be verified.

[0120] The memory can specifically be one or more of a buffer, a register, or random access memory (RAM).

[0121] The NPU deploys a configuration module and a verification module. The configuration module is used to configure the configuration information required during speculative inference (or speculative decoding). For example, the configuration information includes at least the verification position, and may also include one or more of the following: skip vector, confidence factor, draft length, etc.

[0122] Verification position, used to indicate to the NPU which token in the token sequence to start verification from in this round of verification.

[0123] In embodiments where the draft model is itself, a skip vector needs to be configured. This skip vector indicates which specified intermediate layers the draft model needs to skip relative to the original large model. In embodiments where the draft model is an independent draft model, no skip vector needs to be configured.

[0124] The confidence factor is a threshold used to control whether validation passes, thus controlling the stringency of the validation process. In some embodiments, different confidence factors can be set for different types of tasks. For example, for difficult or complex tasks, the confidence factor is generally lowered to generate more diverse answers. For simpler, general tasks, the confidence factor can be increased to ensure prediction accuracy, but the predicted answers may be relatively homogeneous.

[0125] The draft length controls the number of draft tokens generated in each round of the draft task. It can be a fixed value, for example, setting the draft length N=5, meaning 5 draft tokens are generated per round. In other embodiments, the draft length can be adjusted based on the verification results; a high verification pass rate allows for an increase in the draft length, while a low pass rate allows for a decrease.

[0126] The verification module verifies the tokens generated by CIM and updates the relevant configuration information and token sequence based on the verification results. The updated token sequence is the verified token sequence. The NPU transmits the verified token sequence to CIM, which then inputs it into the draft model to generate multiple draft tokens for the next round of verification.

[0127] Before generating a draft token in CIM, the NPU loads the latest configuration information into CIM to read information such as draft length and skip vector before CIM executes the draft task.

[0128] The following example, using the system architecture shown in Figure 6 and the functional module interaction diagram shown in Figure 7 as examples, with the original large model being LLM and the application scenario being intelligent question answering, will be used to illustrate the concept.

[0129] If a user enables the speculative reasoning function through the interactive interface, or if a user specifically specifies in a task request that speculative reasoning needs to be enabled for acceleration, the control module will generate a control command based on the user's instruction, and control the generation module to enable the speculative reasoning function.

[0130] After the terminal device 302 detects the user's input question, it generates a corresponding prompt based on the user's question and inputs the prompt into the original LLM model. The original LLM model performs a prefill stage, which decodes the prompt in parallel to generate a token sequence. Generally, the prefill stage only generates the first token.

[0131] For example, if a user asks a question in the interactive interface of a terminal device, such as "Please recite Li Bai's 'Quiet Night Thoughts'", the terminal device generates a corresponding prompt based on this question. After segmenting the prompt into words, each token is converted into a word vector and input into the original LLM model. As shown in Figure 8, the NPU runs the original LLM model. In the Prefill stage, based on the prompt, the first token generated is "bedside". The answer displayed in the interactive interface of the terminal device 302 is "bedside".

[0132] In this embodiment, the value of the draft length can vary. The NPU transmits "in front of the bed" to the CIM, and the NPU needs to read the value of the draft length in the configuration module and load the value of the draft length into the CIM.

[0133] It should be noted that in other embodiments, the draft length can be a fixed value, that is, the value of the draft length is a constant, and a fixed number of draft tokens are generated in each round of draft tasks. Then, the NPU does not need to read the value of the draft length before each round of draft tasks and can read the value of the draft length and load it into the CIM before executing the first round of draft tasks. In subsequent rounds of draft tasks, the NPU does not need to read the value of the draft length again, nor does it need to perform the loading operation. Alternatively, in other embodiments, the value of the draft length can be written into the code of the generation module, and there is no need to read or load it in advance.

[0134] The generation module in the CIM executes the first round of draft tasks: according to the draft length, input the word vector corresponding to "in front of the bed" into the draft model, run the draft model, and execute multiple iterative decoding processes of the draft model to generate a continuous plurality of draft tokens to be verified. For example, the value of the draft length N is 6, and 6 draft tokens to be verified need to be generated in each draft task. After inputting "in front of the bed" into the draft model, the 6 draft tokens to be verified are "bright moonlight, shining on". Concatenate the 6 draft tokens with "in front of the bed", and the token sequence to be verified obtained is "in front of the bed bright moonlight, shining on", and transmit this token sequence to the NPU for verification.

[0135] The draft tasks executed by the draft model can be understood as multiple iterations in the decode stage of the draft model. The parameters in the draft model are relatively fewer, and within the latency of one decode of the original large model, the draft model can achieve multiple decodes.

[0136] Next, the CIM continues to execute the second round of draft tasks, input the word vector corresponding to "in front of the bed bright moonlight, shining on" into the draft model to continue generating draft tokens.

[0137] In parallel with the second round of draft tasks, the NPU executes the first round of verification tasks:

[0138] The NPU runs the original large LLM model to verify the multiple draft tokens generated by the CIM. The verification process is similar to the Prefill stage of the large model. Different from the Prefill stage, a verification operation is added.

[0139] Specifically, the NPU inputs the word vectors corresponding to "Before my bed a pool of light, Shining" into the large model and performs parallel decoding similar to the prefill stage to obtain the probability distribution of the next token corresponding to each token. According to the probability distribution predicted by the original large model, it compares with the probability value corresponding to the draft token. Based on the comparison result and combined with the confidence factor, the verification result is obtained.

[0140] Verification method 1: The probability value predicted by the original large model can be compared with the confidence factor. If the probability value predicted by the original large model is greater than or equal to the confidence factor, the verification passes; if the probability value predicted by the original large model is less than the confidence factor, the verification fails. For example, when the original large model performs parallel decoding and predicts the probability distribution of the next token corresponding to "Before my bed", the probability value of the next token being "bright" is 0.65, and the current confidence factor value is 0.5. Since 0.65 > 0.5, the verification passes. Another example, when the original large model performs parallel decoding and predicts the probability distribution of the next token corresponding to "Before my bed a pool of light,", the probability value of the next token being "shine" is 0.05, and the current confidence factor value is 0.5. Since 0.05 < 0.5, the verification fails.

[0141] Verification method 2: The probability sequence corresponding to the token sequence to be verified predicted by the draft model in this round can be obtained. Calculate the difference between the probability values of the elements corresponding to the same token in the probability sequence predicted by the original large model and the probability sequence predicted by the draft model, and compare the confidence factor with this difference. If the difference is greater than the confidence factor, the verification fails; if the difference is less than the confidence factor, the verification passes. For example, when the original large model performs parallel decoding and predicts the probability distribution of the next token corresponding to "Before my bed", the probability value of the next token being "bright" is 0.65; the probability of the next token being "bright" predicted by the draft model is 0.55, and the difference is 0.65 - 0.55 = 0.1. If the confidence factor value is 0.2, since 0.1 < 0.2, the verification passes. Another example, when the original large model performs parallel decoding and predicts the probability distribution of the next token corresponding to "Before my bed a pool of light,", the probability value of the next token being "shine" is 0.05; the probability of the next token being "shine" predicted by the draft model is 0.65, and the difference is 0.65 - 0.05 = 0.6. If the confidence factor value is 0.2, since 0.6 > 0.2, the verification fails.

[0142] Among them, the probability sequence predicted by the original large model or the draft model can specifically be the logits vector or the probability sequence obtained after normalizing the logits vector.

[0143] The value of the confidence factor can be set to different values according to different request categories. For example, in the case of an intelligent question-and-answer scenario, when the questions and answers are relatively single and general, if the above verification method one is adopted, the value of the confidence factor can be increased. This can improve the accuracy of the answer, but the answers obtained are often relatively single, lacking diversity and unable to handle more complex questions. If verification method two is adopted, the value of the confidence factor needs to be decreased.

[0144] In an intelligent question-and-answer scenario where more complex or diverse answers are required, if verification method one is adopted, the value of the confidence factor can be decreased to make the answers more diverse. If verification method two is adopted, the value of the confidence factor needs to be increased.

[0145] In some embodiments, it is also possible to choose to update the value of the confidence factor once per round, or update the value of the confidence factor once every multiple rounds.

[0146] It should be noted that during verification, verification needs to be carried out sequentially starting from the position indicated by the verification position. For example, after "明" passes the verification, "月" is verified. If "月" passes the verification, "光" is continued to be verified, and so on until a token that fails the verification appears, or all the multiple draft tokens generated in this round are verified.

[0147] After verifying multiple draft tokens, according to the verification results, an updated token sequence is obtained, or a verified token sequence is obtained. How to determine the token sequence after this round of verification based on the verification results will be specifically described in the following content.

[0148] The NPU can update the value of the verification position according to the number of tokens in the obtained token sequence. Before the first round of verification, the position encoding of the first token "床" in "床前" is 1, and "前" is 2, so the value of the verification position is 3; if the position encoding of "床" is 0 and the position encoding of "前" is 1, then the verification position is 2. That is, verification needs to start from the next token of "床前". In each subsequent round of verification, the NPU first reads the value of the verification position from the configuration module. For example, if the current value of the verification position is 3, verification starts from the token with the position encoding of 3 in the token sequence to be verified. After the first round of verification, the obtained token sequence is "床前明月光,疑是地". If the position encoding of "地" is 8, then the value of the verification position is updated to 9. That is, if the position encoding of the last token in the updated token sequence is M, then the value of the verification position is updated to M + 1.

[0149] It should be noted that among multiple parameters such as the above-mentioned draft length, verification position, confidence factor, etc., the verification position needs to be updated in each round. Other parameters, such as the draft length and confidence factor, can be updated or not updated.

[0150] Specifically, after obtaining the verification result, according to the verification result, determine the token sequence after this round of verification. The specific rules are as follows:

[0151] Case 1: When multiple draft tokens in the to-be-verified token sequence are partially verified successfully, when the NPU verifies each draft token one by one among the multiple draft tokens and the first draft token that fails the verification appears, immediately end the verification, reject the subsequent draft tokens, and use the token sequence predicted by the original large model to overwrite the draft token sequence of this round as the token sequence after this round of verification. Moreover, transmit the updated sequence (i.e., the token sequence after this round of verification) to the CIM. After receiving the token sequence transmitted by the NPU, the CIM immediately stops the current draft task and instead uses the token sequence transmitted by the NPU as the input to re-execute the draft task of this round.

[0152] For example, as shown in FIGS. 8 and 10a, in the first-round draft task of the CIM, the multiple draft tokens generated are "Bright moonlight, shines on", and the to-be-verified draft token sequence is "In front of the bed, bright moonlight, shines on". In the first-round (t = 1) verification task, the token sequence corresponding to the probability sequence predicted by the original large model is "In front of the bed, bright moonlight, seems to be the ground". According to the probability sequence predicted by the LLM large model, verify one by one. When verifying the draft token "shines on", the verification fails, then stop verifying the subsequent tokens, and use the sequence "In front of the bed, bright moonlight, seems to be the ground" generated by the original LLM large model to overwrite "In front of the bed, bright moonlight, shines on" output by the CIM as the token sequence after this round of verification and transmit it to the CIM. The CIM stops the currently executing second-round draft task and re-enters the word vectors of "In front of the bed, bright moonlight, seems to be the ground" into the draft model to regenerate draft tokens. For example, in the (t + 1)-th round draft task, the generated to-be-verified token sequence is "In front of the bed, bright moonlight, seems to be frost on the ground. Looking up".

[0153] Alternatively, as shown above the dotted line in Figure 9, the token sequence to be verified generated by the CIM's first-round draft task is "Before my bed a pool of light, shining on the ground". When the NPU performs verification and fails to pass the verification when verifying the draft token "shining", it stops verifying the subsequent tokens and clears the subsequent draft tokens in the token sequence of the draft to be verified. Instead, it uses the token sequence generated by the NPU running the LLM large model to overwrite the draft token sequence.

[0154] Case 2: During the t-th round of verification, all the draft tokens in the token sequence to be verified output by the t-th round of draft task pass the verification, and the NPU receives the token sequence to be verified (including multiple draft tokens corresponding to the t + 1-th round) transmitted by the CIM after executing the (t + 1)-th draft task, and the last token predicted and generated in the verification stage is the same as the first token among the multiple draft tokens generated by the CIM in the (t + 1)-th round. Then, all the draft tokens of the CIM in the (t + 1)-th round are included in the tokens to be verified, and the (t + 1)-th verification is continued. Meanwhile, the CIM is executing the (t + 2)-th draft task in parallel with the NPU.

[0155] For example, as shown in Figure 10b, the draft token sequence to be verified generated by the CIM in the first-round draft task is "Before my bed a pool of light, I wonder". The NPU inputs it into the LLM large model for the t-th round of verification (at this time t = 1). And the CIM inputs the draft token sequence to be verified into the draft model opinion for the (t + 1)-th draft task (t = 1). In the verification, the token sequence predicted by the LLM large model is "Before my bed a pool of light, I wonder if it's the ground". All the draft tokens generated by the first-round draft task pass the verification. The draft token sequence to be verified generated by the next round (the (t + 1)-th round, t = 1) of the draft task is "Before my bed a pool of light, I wonder if it's frost on the ground. Looking up". Among them, "frost on the ground, Looking up" are the draft tokens generated by the (t + 1)-th round of the draft task, and the first draft token is "ground"; the last token in the token sequence predicted by the original large model is "ground", and the two are the same. Then, the draft token sequence to be verified obtained by the CIM in the new round, that is, the (t + 2)-th round, "Before my bed a pool of light, I wonder if it's frost on the ground. Looking up" is included in the current token sequence to be verified, and the (t + 1)-th round of verification task is continued.

[0156] Alternatively, as shown below the dashed line in Figure 9, the token sequence to be verified generated in the first round of CIM is "Before my bed a pool of light, I wonder", and in the first round of the NPU verification task, the predicted token sequence is "Before my bed a pool of light, I wonder if". In the second-round draft task of CIM, multiple draft tokens to be verified are generated as "It's frost on the ground", and the token sequence to be verified including the multiple draft tokens generated in the second round is "Before my bed a pool of light, I wonder if it's frost on the ground". The last token in "Before my bed a pool of light, I wonder if" predicted by the original large model is "if", which is the same as the first draft token "It's" among the multiple draft tokens generated in the second-round draft task of CIM. Then, the "Before my bed a pool of light, I wonder if it's frost on the ground" obtained in the second round of CIM is used as the draft token sequence to be verified in the next round.

[0157] Case 3: During the verification process of the t-th round, all the multiple draft tokens in the token sequence to be verified output by the draft task of the t-th round are verified successfully, and the NPU receives the token sequence to be verified (including the multiple draft tokens corresponding to the (t + 1)-th round) transmitted by CIM after performing the (t + 1)-th draft task. Moreover, the last token predicted and generated during the verification phase is different from the first token among the multiple draft tokens generated in the (t + 1)-th round of CIM. Then, the verification token sequence generated by the NPU is used to overwrite the draft token sequence. That is, the NPU overwrites the draft token sequence of this round with the token sequence generated by the original large model, and transmits the updated or overwritten sequence (i.e., the verified sequence) to CIM. CIM stops the current (t + 2)-th round task and re-enters the updated token sequence to generate the draft tokens of the (t + 2)-th round.

[0158] For example, as shown in Figure 10c, assume that in the (t + 1)-th round (where t = 1) draft task of CIM, the token sequence to be verified generated is "Before my bed a pool of light, I wonder if they...", where "they..." are the draft tokens generated in the (t + 1)-th round draft task, and the first draft token is "he". In the t-th round verification task, the token sequence predicted by the LLM large model is "Before my bed a pool of light, I wonder if it's", and the last token is "it's", which is different from "he". Then, the token sequence "Before my bed a pool of light, I wonder if it's" predicted by the original large model is used to overwrite "Before my bed a pool of light, I wonder if they...", and is input into the draft model to perform the (t + 2)-th round draft task.

[0159] After updating the token sequence, update the value of the verification position to the position encoding of the next token after the last token in the updated token sequence. For example, if the updated token sequence is "Before my bed a pool of light, I wonder if it's frost on the ground.", and the verification position of "bright" is 2 under the numbering rule, then the position encoding of the next token after "ground" is 8, and the value of the verification position is updated to 8. Or rather, when the position encoding of the last token in the token sequence after this round of verification obtained according to the verification result is M, the value of the verification position is updated to M + 1.

[0160] According to the speculative reasoning strategies in the above three scenarios, loop or iterate through the draft stage and the verification stage like this until "Before my bed a pool of light, I wonder if it's frost on the ground. I lift my head to see the moon, I bend my head and think of home." is output.

[0161] As can be seen from FIGS. 10a to 10c, the t-th round of verification task and the (t + 1)-th round of draft task can be executed in parallel in time, that is, while verifying and while drafting, which can reduce the situation of waiting for each other between the draft stage and the verification stage. In practical applications, it may occur that all the draft tokens generated in the previous rounds pass. At this time, the (t + 2)-th round of draft task may not be aborted, that is, the execution of the (t + 2)-th round of draft task does not need to wait for the execution of the (t + 1)-th round of verification task to complete, thereby improving the efficiency of speculative reasoning, making full use of the computing resources of the heterogeneous computing architecture, and improving the response speed.

[0162] According to the above exemplary description, the speculative reasoning method proposed in the embodiments of the present application can be applied to an electronic device. The processor architecture in the electronic device adopts a heterogeneous computing architecture, and the heterogeneous computing architecture may include a first processing unit (such as CIM) and a second processing unit (such as NPU) for parallel processing of tasks. The first processing unit is used to execute the task of generating draft tokens; the second processing unit is used to execute the task of verifying the draft tokens. As shown in FIG. 11, this method may specifically include the following processes: [[ID=ed]]

[0163] S10: In the t-th round of draft task, the first processing unit inputs the word vector corresponding to the current token sequence into the draft model to generate multiple first draft tokens to be verified.

[0164] S11: Transmit the first token sequence including multiple first draft tokens to the second processing unit.

[0165] When t ≥ 1, the current token sequence is the first token generated by the original large model during the prefill stage based on the prompt. The first processing unit inputs the word vector corresponding to the first token into the draft model to generate multiple first draft tokens to be verified. When t ≥ 2, the current token sequence is the token sequence verified by the NPU.

[0166] S12: In the (t+1)th round of the draft task, the first processing unit inputs the word vectors corresponding to the first token sequence into the draft model to generate multiple second draft tokens to be verified.

[0167] S13: In the t-th round of verification task, the second processing unit inputs the word vector corresponding to the first token sequence into the original large model corresponding to the draft model, and verifies multiple first draft tokens through the original large model to obtain the verification result.

[0168] S14: Determine the third token sequence based on the verification results.

[0169] The third token sequence is a token sequence verified by the NPU and is used as input for the next round of draft or verification tasks.

[0170] The t-th round of verification tasks and the t+1-th round of draft tasks are executed in parallel, that is, S12 and S13 can be executed in parallel.

[0171] Specifically, in some embodiments, the process of using NPU and CIM to implement pipelined parallel processing of the drafting and verification stages may include the following steps:

[0172] S20: The NPU reads the configuration information and loads some of the information into the CIM.

[0173] The terminal device pre-installs the original LLM large model and draft model, and writes initial values ​​for various configuration information. The configuration information can be modified by the user. The configuration information includes one or more of the following: verification location, draft length, skip vector, and confidence factor. After reading the configuration information, the NPU loads some information from the configuration information (i.e., information needed to execute the draft task, such as draft length and skip vector) into the CIM. Upon receiving a user's command to start the speculative inference function or in response to a question entered by the user on the interactive interface, the control module in the CIM generates a start command, triggering the generation module to generate a draft token.

[0174] S21: CIM performs the first draft task.

[0175] Specifically, after receiving the start command, CIM skips the corresponding intermediate layer in the large model according to the intermediate layer indicated by the skip vector in the configuration information. Starting from the position where the last draft ended, it generates a corresponding number of draft tokens based on the number indicated by the draft length. The draft tokens are concatenated with the first token generated in the large model prefill stage to obtain the token sequence to be verified.

[0176] S22: CIM performs the second draft task, while NPU performs the first verification task in parallel.

[0177] In other words, the current verification task and the next draft task run in parallel: CIM sends the generated draft tokens to be verified to the NPU for verification, while simultaneously starting the next draft phase. During the NPU verification process, CIM can determine whether multiple tokens matching the draft length indication have been generated based on the draft length, and CIM can decide to continue execution or terminate the current draft phase early based on information such as the received token sequence or verification results (e.g., verification failure).

[0178] S23: The NPU updates the configuration information and token sequence based on the verification results.

[0179] S24: Determine whether the current question-and-answer task has been completed. If yes, end the task; otherwise, transmit the updated token sequence to CIM and continue with the next round of draft tasks.

[0180] Thus, repeat steps S21 to S23 until the verification sequence meets the requirements. At this point, output the relevant sequence and the process ends.

[0181] Specifically, in some embodiments, the verification of multiple first draft tokens using the original large model to obtain verification results can be achieved by running the original large model, decoding the multiple first draft tokens in parallel to obtain a probability sequence corresponding to the multiple first draft tokens; and then verifying each draft token in the multiple first draft tokens sequentially according to the probability sequence to obtain a verification result. The probability sequence can be the original score logits predicted by the original large model, or it can be the probability distribution obtained by transforming the logits using the softmax function.

[0182] In some embodiments, according to the verification result, a third token sequence is determined. For example, when the verification result indicates that multiple first draft token parts pass the verification, the token sequence corresponding to the probability sequence is used as the third token sequence; the third token sequence is transmitted to the first processing unit. For example, as shown in FIG. 8, "Before my bed a pool of light, I wonder if it's" is the token sequence corresponding to the probability sequence, that is, the token sequence predicted by the original large model.

[0183] After determining the third token sequence, in the 3rd round of draft task, the first processing unit inputs the word vectors corresponding to the third token sequence into the draft model to generate multiple third draft tokens to be verified, and transmits the fourth token sequence including the multiple third draft tokens to the second processing unit.

[0184] In some embodiments, before verifying multiple first draft tokens through the original large model, the second processing unit reads the value of the verification position. The verification position is used to indicate the position of the first draft token among the currently to-be-verified multiple draft tokens in the first token sequence. The position can specifically be the position encoding of each token in the token sequence. Verifying N draft tokens can be starting from the position indicated by the verification position through the original large model. After determining the third token sequence, the second processing unit updates the value of the verification position according to the third token sequence. A feasible update method is to update the value of the verification position to M + 1 according to the number of tokens M in the third token sequence.

[0185] For example, in the 3rd round of draft task, CIM inputs the word vectors corresponding to the third token sequence "Before my bed a pool of light, I wonder if it's" into the draft model, and the multiple third draft tokens generated to be verified are "On the ground frost. Looking up", and the fourth token sequence is "Before my bed a pool of light, I wonder if it's frost on the ground. Looking up", which is transmitted to the NPU. The NPU starts verification from the token with the position encoding of 8 in the fourth token sequence according to the current value in the verification position in the configuration information. For example, the number of tokens in the third token sequence is 7, so the position encoding of the token "On" in the token sequence is 8, and the value of the verification position updated in the previous round is 8.

[0186] In some embodiments, after generating multiple second draft tokens to be verified in the second draft task, a second token sequence including the multiple second draft tokens can be transmitted to the second processing unit. Based on the verification results, a third token sequence is determined. This can be achieved by verifying the first second draft token in the second token sequence based on the last token in the probability sequence if the verification result indicates that all first draft tokens have passed verification; if the first second draft token has passed verification, the second token sequence is used as the third token sequence. After determining the third token sequence, in the second verification task, the word vector corresponding to the third token sequence is input into the original large model, and the original large model is used to verify the multiple second draft tokens to obtain the verification result.

[0187] For example, as shown in Figure 10b, the second draft task generates multiple second draft tokens, "Frost on the ground. I raise my head and look." The first second draft token is "ground," resulting in the second token sequence "Bright moonlight before my bed, I suspect it is frost on the ground. I raise my head and look." In the first validation task, the token sequence predicted by the original large model is "Bright moonlight before my bed, I suspect it is ground." "Ground" is the last token in the probability sequence, which is consistent with the first second draft token. Therefore, the second token sequence "Bright moonlight before my bed, I suspect it is frost on the ground. I raise my head and look" is used as the input for the second validation.

[0188] In the second draft task, after generating multiple second draft tokens to be verified, a second token sequence including these tokens is transmitted to the second processing unit. Based on the verification results, a third token sequence is determined. This can be achieved by verifying the first second draft token in the second token sequence based on the last token in the probability sequence if all first draft tokens have passed verification; or by using the token sequence corresponding to the probability sequence as the third token sequence if the first second draft token fails verification. This third token sequence is then transmitted to the first processing unit. After determining the third token sequence, in the third draft task, the first processing unit inputs the word vectors corresponding to the third token sequence into the draft model to generate multiple third draft tokens to be verified, and transmits a fourth token sequence including these tokens to the second processing unit.

[0189] For example, as shown in Figure 10c, the second draft task generates multiple second draft tokens of “they…”, and the first second draft token is “he”, which is different from the last token “ground” predicted by the original big model. Therefore, the latest token sequence predicted by the original big model, “the moonlight before my bed, I suspect it is ground”, is taken as the current verified token sequence (i.e. the third token sequence), and its corresponding word vector is taken as the input of the third draft task.

[0190] In some embodiments, before the first processing unit inputs the word vector corresponding to the first token into the draft model, the first processing unit reads the draft length value; the draft length indicates the number of draft tokens to be verified generated by the draft model in this round of drafting task. After obtaining the verification result, the second processing unit adjusts the draft length value according to the verification result.

[0191] For example, the draft length can be adjusted based on the verification results. The adjustment principle could be that a higher verification pass rate results in a larger draft length, and a lower pass rate results in a smaller draft length. If the verification results indicate that M out of N draft tokens have passed verification, and 0 ≤ M < N, the draft length is adjusted to X, where X ≤ N. If the verification results indicate that all N draft tokens have passed verification, the draft length is adjusted to Y, where Y ≥ N.

[0192] In some embodiments, before validating multiple first draft tokens using the original large model, the second processing unit reads the value of the confidence factor. Based on the probability sequence, each draft token among the multiple first draft tokens is validated sequentially. This can be done by calculating the difference between the probability values ​​of tokens corresponding to the same position in the first probability sequence predicted by the original large model and the second probability sequence predicted by the draft model, based on the difference and the confidence factor value. After obtaining the validation result, the confidence factor value is adjusted accordingly. The confidence factor is a threshold used to control whether the validation of a draft token passes.

[0193] For example, if the probability value of the token with position code 8 in the first probability sequence is 0.65, and the probability value of the token with position code 8 in the second probability sequence is 0.25, assuming the confidence factor is 0.3, then 0.65 - 0.25 = 0.4, and 0.4 > 0.3, so the verification fails. If the probability value of the token with position code 8 in the first probability sequence is 0.45, and the probability value of the token with position code 8 in the second probability sequence is 0.35, assuming the confidence factor is 0.3, then 0.45 - 0.35 = 0.1, and 0.1 < 0.3, so the verification passes.

[0194] In some embodiments, the draft model is a self-draft model obtained by skipping specified intermediate layers based on the original large model. Before executing each round of the draft task, the first processing unit can read the value of the skip vector; the skip vector is used to indicate the intermediate layers skipped by the draft model relative to the original large model. When executing each round of the draft task, the corresponding intermediate layers in the original large model can be skipped according to the skip vector, and the current round of the draft task can be executed.

[0195] In summary, the embodiments of this application employ a heterogeneous computing architecture, such as a heterogeneous architecture composed of an NPU and a CIM. The CIM is used for fast drafting tasks, while the NPU is used for efficient verification tasks. The high bandwidth and low power consumption characteristics of the CIM help alleviate bandwidth and power consumption issues on the edge, improve energy efficiency, and contribute to long battery life.

[0196] Furthermore, NPU and CIM are used for pipelined parallel processing of the draft and verification stages. That is, CIM and NPU work in parallel, and NPU verifies more tokens at once, which improves resource utilization, alleviates bandwidth issues, shortens inference time, and enhances user experience.

[0197] This application also provides an electronic device, the electronic device comprising: a processor, the processor being configured to execute a computer program or instructions in a memory to implement the method as described in any of the above embodiments.

[0198] It should be noted that a processor can be a chip with computing capabilities, and is not limited to a central processing unit (CPU). For example, a processor can be a chip that includes one or more transistors, resistors, capacitors, and other circuit elements to perform a certain function; or it can be an integrated circuit in various packages that can implement the above methods.

[0199] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the above-described speculative reasoning method.

[0200] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the above-described speculative reasoning method, or instruct the computing device to perform a speculative reasoning method.

[0201] This application also provides a chip system, including: a communication interface for inputting and / or outputting data; and a processor for executing a computer-executable program, causing a device equipped with the chip system to perform the speculative reasoning method as described in any of the above embodiments.

[0202] This application also provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method as described in any of the above embodiments.

[0203] This application also provides a computer program product containing instructions that, when executed by a computing device cluster, cause the computing device cluster to perform the method described in any of the above embodiments.

[0204] This application also provides a computer-readable storage medium including computer program instructions, which, when executed by a cluster of computing devices, perform the method described in any of the above embodiments.

[0205] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speculative reasoning method, characterized in that, The method is applied to an electronic device, the processor of which includes a first processing unit and a second processing unit for parallel processing tasks; the method includes: In the first draft task, the first processing unit inputs the word vector corresponding to the first token into the draft model to generate multiple first draft tokens to be verified; and transmits the first token sequence including the multiple first draft tokens to the second processing unit; the first token is generated by the original large model in the prefill stage based on the prompt; In the second draft task, the first processing unit inputs the word vectors corresponding to the first token sequence into the draft model to generate multiple second draft tokens to be verified. In the first round of verification task, the second processing unit inputs the word vectors corresponding to the first token sequence into the original large model corresponding to the draft model, and verifies the multiple first draft tokens through the original large model to obtain the verification result; and, based on the verification result, determines the third token sequence; the third token sequence is used as the input for the next round of draft task or verification task; the first round of verification task and the second round of draft task are executed in parallel.

2. The method as described in claim 1, characterized in that, The multiple first draft tokens are verified using the original large model to obtain verification results, including: Run the original large model to decode the multiple first draft tokens in parallel and obtain the probability sequences corresponding to the multiple first draft tokens; Based on the probability sequence, each draft token in the plurality of first draft tokens is verified sequentially to obtain the verification result.

3. The method as described in claim 2, characterized in that, Based on the verification results, the third token sequence is determined, including: If the verification result indicates that the plurality of first draft tokens have passed verification, the token sequence corresponding to the probability sequence is taken as the third token sequence; the third token sequence is then transmitted to the first processing unit. After determining the third token sequence, the method further includes: In the third draft task, the first processing unit inputs the word vector corresponding to the third token sequence into the draft model to generate multiple third draft tokens to be verified, and transmits the fourth token sequence including the multiple third draft tokens to the second processing unit.

4. The method as described in claim 2, characterized in that, In the second round of the draft task, after generating multiple second draft tokens to be verified, the method further includes: The second token sequence, including the plurality of second draft tokens, is transmitted to the second processing unit; Based on the verification results, the third token sequence is determined, including: If the verification result indicates that all of the plurality of first draft tokens have been verified, the first second draft token in the second token sequence is verified based on the last token in the probability sequence. If the first draft token passes verification, the second token sequence will be used as the third token sequence; After determining the third token sequence, the method further includes: In the second round of verification, the word vectors corresponding to the third token sequence are input into the original large model, and the original large model is used to verify the multiple second draft tokens to obtain the verification results.

5. The method as described in claim 2, characterized in that, In the second round of the draft task, after generating multiple second draft tokens to be verified, the method further includes: The second token sequence, including the plurality of second draft tokens, is transmitted to the second processing unit; Based on the verification results, the third token sequence is determined, including: If the verification result indicates that all of the plurality of first draft tokens have been verified, the first second draft token in the second token sequence is verified based on the last token in the probability sequence. If the first draft token fails verification, the token sequence corresponding to the probability sequence is used as the third token sequence; the third token sequence is then transmitted to the first processing unit. After determining the third token sequence, the method further includes: In the third draft task, the first processing unit inputs the word vector corresponding to the third token sequence into the draft model to generate multiple third draft tokens to be verified, and transmits the fourth token sequence including the multiple third draft tokens to the second processing unit.

6. The method according to any one of claims 1-5, characterized in that, Before validating the multiple first draft tokens using the original large model, the method further includes: The second processing unit reads the value of the verification position; the verification position is used to indicate the position of the first draft token among the multiple draft tokens to be verified in the first token sequence; The multiple first draft tokens are verified using the original large model, including: The verification process begins from the position indicated by the verification location, using the original large model. After determining the third token sequence, the method further includes: The second processing unit updates the value of the verification position according to the third token sequence.

7. The method as described in claim 6, characterized in that, Update the value of the verification position based on the third token sequence, including: Based on the fact that the number of tokens in the third token sequence is M, the value of the verification position is updated to M+1.

8. The method according to any one of claims 1-7, characterized in that, Before the first processing unit inputs the word vector corresponding to the first token into the draft model, the method further includes: The first processing unit reads the draft length value; the draft length is used to indicate the number of draft tokens to be verified generated by the draft model in this round of draft task; After obtaining the verification result, the method further includes: The second processing unit adjusts the draft length value based on the verification result.

9. The method as described in claim 8, characterized in that, Based on the verification results, adjust the draft length value, including: If the verification result indicates that M out of N draft tokens have passed verification, and 0 ≤ M < N, then the length of the draft is adjusted to X, where X ≤ N. If the verification result indicates that all N draft tokens have passed verification, the length of the draft is adjusted to Y, where Y ≥ N.

10. The method as described in claim 2, characterized in that, Before validating the multiple first draft tokens using the original large model, the method further includes: The second processing unit reads the value of the confidence factor; the confidence factor is a threshold used to control whether a draft token passes verification. Based on the probability sequence, each draft token in the plurality of first draft tokens is verified sequentially, including: Based on the first probability sequence predicted by the original large model, calculate the difference between the probability values ​​of tokens corresponding to the same position in the first probability sequence and the second probability sequence predicted by the draft model, and obtain the verification result based on the difference and the value of the confidence factor. After obtaining the verification result, the method further includes: Adjust the value of the confidence factor based on the verification results.

11. The method according to any one of claims 1-10, characterized in that, The draft model is a self-draft model obtained by skipping the specified intermediate layers based on the original large model; Before executing each round of drafting tasks, the method further includes: The first processing unit reads the value of the skip vector; the skip vector is used to indicate the intermediate layers that the draft model skips relative to the original large model; When performing each round of drafting tasks, the method includes: Based on the skip vector, skip the corresponding intermediate layers in the original large model and execute the current draft task.

12. The method according to any one of claims 1-11, characterized in that, The first processing unit is an in-memory computing (CIM) unit; the second processing unit is a neural network processing (NPU) unit.

13. An electronic device, characterized in that, The electronic device includes: A processor for executing a computer program or instructions in memory to implement the method as described in any one of claims 1-11.

14. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-11.

15. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1-11.

16. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-11.

Images