Accelerating transformer inference through speculative precomputation

Abstract

Transformer inference may be accelerated by a prescient paraclete module based on speculations made at intermediate layers. A main thread may perform a first inference process of a transformer model. The prescient paraclete module may extract one or more speculated tokens generated at an intermediate layer during the first inference process and instruct one or more speculative threads to each perform a second inference process using a speculated token. The prescient paraclete module may detect whether any of the speculated token(s) matches a predicted token generated at the final layer of the model. When there is a match, the prescient paraclete module may retire the main thread. The speculative thread corresponding to the correctly speculated token may become the new main thread. When there is no match, the prescient paraclete module may allow the main thread to continue and discard the one or more speculative threads.

Description

ACCELERATING TRANSFORMER INFERENCE THROUGH SPECULATIVE PRECOMPUTATIONCross-Reference to Related Application

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 713,928, filed October 30, 2024, and entitled "ACCELERATION OF TRANSFORMER COMPUTATION ON HETEROGENEOUS ACCELERATORS," which is incorporated by reference in its entirety for all purposes.Technical Field

[0002] This disclosure relates generally to artificial intelligence (Al), and more specifically, accelerating transformer inference (including transformer inference on heterogenous hardware) through speculative precomputation.Background

[0003] Neural networks (also referred to as "deep neural networks" or "DNNs") are used extensively for a variety of Al applications ranging from natural language processing to computer vision, speech recognition, and image processing due to their ability to achieve high accuracy. However, the high accuracy comes at the expense of significant computation cost. DNNs have extremely high computing demands as there can be a large number of operations as well as a large amount of data to read and write. Therefore, techniques to improve efficiency of DNNs are needed.Brief Description of the Drawings

[0004] Embodiments can be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0005] FIG. 1 illustrates an example transformer architecture, in accordance with various embodiments.

[0006] FIG. 2 is a block diagram of an Al system with heterogenous hardware, in accordance with various embodiments.

[0007] FIG. 3 illustrates a prescient layer-wise transformer timeline, in accordance with various embodiments.

[0008] FIG. 4 illustrates acceleration of transformer inference in a frugal paraclete mode, in accordance with various embodiments.

[0009] FIG. 5 illustrates acceleration of transformer inference in an eager paraclete mode, in accordance with various embodiments.

[0010] FIG. 6 illustrates layer-level tokens generated in the first inference process of a transformer model, in accordance with various embodiments.

[0011] FIG. 7 illustrates layer-level tokens generated in the second inference process of a transformer model, in accordance with various embodiments.

[0012] FIG. 8 illustrates an example transformer model, in accordance with various embodiments.

[0013] FIG. 9A and 9B illustrate inferences of a transformer model, in accordance with various embodiments.

[0014] FIG. 10 is a block diagram of a neural processing unit (NPU), in accordance with various embodiments.

[0015] FIG. 11 is a flowchart of a method of accelerating transformer inference, in accordance with various embodiments.

[0016] FIG. 12 is a block diagram of an example computing device, in accordance with various embodiments.Detailed DescriptionOverview

[0017] The last decade has witnessed a rapid rise in Al based data processing, particularly based on DNNs. DNNs are widely used in various domains (e.g., language processing, computer vision, speech recognition, autonomous driving, image processing, video processing, etc.) mainly due to their ability to achieve beyond human-level accuracy. A DNN typically includes a sequence of layers. A DNN layer may include one or more deep learning operations (also referred to as "neural network operations"), such as matrix multiplication, convolution, layer normalization, batch normalization, Softmax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on.

[0018] Transformer computation is one of the predominant areas for accelerated computing. Transformers are a universal foundation algorithm to solve many critical Al problems. A fundamental challenge in transformer computation is that for a given input prompt, the final token is predicted after all the attention head computation across all the layers in the transformer model architecture. Then the predicted token is appended to the prompt and then the new prompt is sent to the transformer to do all the computation to predict the next token. So on and so forth. All these computations are typically performed regardless of the values of the prompt, and regardless of whether the final token could have been predicted effectively much sooner at early layers.

[0019] Many currently available solutions use speculative decoding to tackle the above issue. The idea is typically to use training or fine-tuning to augment the canonical transformer computation to predict multiple tokens (instead of a single token) at the final layer. In an analogy to classical computer architecture or uArch design, the speculative decoding is analogous to trace prediction based on profile guided compilation. For a given transformer model originally designed to support one token prediction, a new round of training or fine-tuning is required to achieve speculative decoding. The extra training and fine-tuning can be significant overhead and impair the efficiency of transformer inference.

[0020] Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by providing Prescient Layer-wise Transformer (PLT), a novel architectural technique and framework that leverages Layer-Level Parallelism (LLP) to accelerate inference by exploiting the predictive capabilities of intermediate transformer layers. The prescient transform technique in this disclosure can identify opportunities early and take potentially profitable speculative actions to accelerate computation to overcome sequential bottlenecks. The prescient transformer technique can honor the program order of layer-wise computation. The overall computation may be identical or substantially identical to running the default transformer algorithm without speculative computation. This aspect can make the prescient transformer technique resemble classic microarchitectural innovations in modern processors that exploit Instruction-Level Parallelism (ILP). The prescient transformer technique can accelerate the foundational transformer algorithm, encompassing all its known variants.

[0021] In various embodiments of the present disclosure, an Al system may include one or more processing units, which can perform transformer inference, and a prescient paracletemodule, which can accelerate transformer inference based on speculative computation. The one or more processing units may facilitate threads that can perform transformer inference in parallel. For the description of various embodiments of this disclosure, the notion of thread is to represent the computational resource for a transformer inference task. A transformer inference task may be an Al task for predicting one or more tokens based on an input prompt. An input prompt may also be referred to as an input sequence. An input sequence incudes one or more tokens, these tokes are referred to as input tokens. Typically, the transformer model may predict a single token through a single inference process, and the token is generated at the final layer of the transformer model. This token is also referred to as the final token or predicted token. Layers in the transformer model are arranged in an order, and the final layer is the last layer. A layer precedent to the final layer is an intermediate layer. The transformer inference task may include a sequence of inference processes to predict a sequence of tokens. The first inference process may be performed based on the input prompt, and each subsequent inference process may be performed based on a combination of the input prompt and the token(s) predicted in all the previous inference process(es). Such as transformer inference task is referred to as a predictive inference task. The prescient paraclete module may instruct a main thread on the one or more processing unit to perform the predictive inference task.

[0022] To accelerate a transformer inference task, the prescient paraclete module may dynamically perform layer-wise logits analysis at earlier layers during the predictive inference task (e.g., during an inference process within the predictive inference task) to predict the top- / < potential candidate tokens that may match the final token, where K is 1 or a greater integer. For instance, the prescient paraclete module may extract K potential candidate tokens generated at an intermediate layer during the current inference process by the main thread, where K is 1 or a greater integer. These tokens are referred to as speculated tokens. For each speculated token, the prescient paraclete module may initiate a speculative thread to perform a speculative inference task using the speculated token. The prescient paraclete module may instruct the main thread to spawn the speculative precomputation thread. The speculative inference task may start with the next inference process using a sequence that includes the speculated token, the input token(s), and all the predicted token(s) from previous inference process(es), if any. The K speculative inference tasks by the K speculative threads and the predictive inference task by the main thread maybe in parallel. The main thread may continue the current inference process in the predictive inference task till the current inference process is complete and the final token is generated at the last layer. The prescient paraclete module may then detect if any of the K speculated tokens match the final token. In cases where there is no match, the prescient paraclete module may allow the main thread to continue the predictive inference task and discard all the speculative threads. In cases where a match is found, the prescient paraclete module may retire the main thread and fast forward the predicted inference task to the speculative thread initiated with the correct speculation. The speculative thread may effectively become the new main thread. The prescient paraclete module may discard the speculative thread(s) for the speculated tokens not matching the final token.

[0023] By fast-forwarding the predicted inference task based on the speculations made at intermediate layers, the prescient paraclete module can accelerate the transformer inference and save compute resources. Speculative precomputation can beget further speculative precomputation, which can lead to a tree of speculative precomputation and significantly accelerate transformer inference. Final token can be predicted at high confidence at much earlier layer much sooner than the final layer and therefore, speculative precomputations can be performed with high probability of success. The prescient paraclete module may also facilitate reuse of data (e.g., KV cache, etc.) generated by retired threads, either main thread or speculative thread. For instance, the prescient paraclete module may instruct all the threads to save data into a shared memory and may instruct the new main thread to read reusable data from the shared memory for performing the rest of the inference task.

[0024] The processing units in the Al system may be heterogeneous processing units, i.e., processing units of different types. For instance, the Al system may include one or more central processing units (CPUs), graphics processing units (GPUs), NPUs (e.g., Al accelerators), other types of processing units, or some combination thereof. The prescient paraclete module may find threads on the heterogenous processing units that are available and have the capability to be either main thread or speculative thread. In an example, the prescient paraclete module may initiate a main thread on a CPU to avoid data traffic as less data transfer would be needed when the CPU performs predictive inference tasks. In another example, the prescient paraclete module may initiate a speculative thread on a GPU or NPU, which are designed with parallel processing capabilities and can accelerateneural network inference by leveraging their parallel processing capabilities, as the speculative thread may have a high chance to become main thread and to finish the inference task.

[0025] Unlike speculative decoding techniques that require model retraining or fine-tuning, the approach in this disclosure can support existing transformer models and their variants without any model parameter modifications. Retraining or fine-tuning would not be required. This method can parallel classic microarchitectural ILP optimizations in modern microprocessor designs, accelerating latency performance using control and data speculation without necessitating code recompilation. The prescient transform technique in this disclosure can do much better without requiring model retraining or fine-tuning yet achieving better performance and compute efficiency than any of the existing speculative decoding approaches.

[0026] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it can be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or / and that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0027] Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0028] Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter.However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0029] For the purposes of the present disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of the present disclosure, thephrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0030] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives "first," "second," and "third," etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0031] In the following detailed description, various aspects of the illustrative implementations are described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0032] The terms "substantially," "close," "approximately," "near," and "about," generally refer to being within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between the elements, generally refer to being within + / - 5-20% of a target value as described herein or as known in the art.

[0033] In addition, the terms "comprise," "comprising," "include," "including," "have," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, device, or DNN accelerators. Also, the term "or" refers to an inclusive "or" and not to an exclusive "or."

[0034] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.

[0035] FIG. 1 illustrates an example transformer architecture, in accordance with various embodiments. The transformer architecture is a canonical transformer architecture that starts with an embedding layer 110, followed by a series of residual blocks 120 (individually referred to as "residual block 120"), and a unembedding layer 130. For the purpose of illustration and simplicity, FIG. 1 shows one residual block. The embedding layer 110 may also be referred to as a token embedding layer. The unembedding layer 130 may also be referred to as a token unembedding layer. Each residual block 120 may be a transformer layer where tokens are generated. The last residual block of the transformer model may be referred to as the last layer or final layer. The top token generated at the last residual block may be considered the output token of the transformer model, i.e., the token predicted by the transformer model through a single inference process. The transformer model may predict multiple tokens through multiple inference processes.

[0036] In the example shown in FIG. 1, the transformer model receives an input sequence 101. The input sequence 101 includes one or more input tokens. An input token may include a word, symbol, number, or some combination thereof. The embedding layer 110 may translate an input token into a vector representation through token embedding. In some embodiments, the embedding layer 110 may convert each input token in the input sequence 101 into an embedding vector, which may be a numerical representation of the input token. The embedding vector may be denoted as x0= WEt, where t denotes the index of the input token in the vocabulary of the model, and WEdenotes a weight matrix ("embedding matrix"). The embedding vector xQis input into the residual blocks.

[0037] Each residual block 120 includes an attention layer 140, a Multi-Layer Perceptron (MLP) layer 150, an adder 160, and another adder 170. Data going through the residual block 120 is referred to as a residual stream, which is denoted as x. The attention layer 140 includes a plurality of attention heads, each of which is denoted as ht, where i = 0, 1, ... and denotes the attention head index. The attention heads may operate in parallel. The output of the attention heads ("attention outputs") may be added to the residual stream in the adder 160: xi+1= xtwhere xtdenotes the residual steam before the attention heads,denote the attention output, and xi+1denotes the residual output, i.e., the residual steam after the attention layer 140 and adder 160. In some embodiments, the attention heads can enable the model to selectively focus on different parts of the input sequence when processing information. The attention heads may compute attention scoresbetween different elements of the input sequence (e.g., words in a sentence) and use these scores to weigh the input representations. The attention outputs may be contextualized representations.

[0038] xi+1is further processed in the MLP layer 150, e.g., through feedforwarding functions in the MLP layer 150. The output of the MLP layer 150 may be further added to the residual stream in the adder 170: xi+2= xi+1+ m(xi+1), where m denotes the MLP layer 150. In some embodiments, the attention layer 140 and MLP layer 150 in each residual block 120 may read their input from the residual stream, e.g., by performing a linear projection. The attention layer 140 and MLP layer 150 may then write their result back to the residual stream by adding a linear projection in the adder 160 or adder 170. The final residual stream xi+2may be the output of the residual block 120.

[0039] After the completion of each residual block computation (also referred to as transformer layer computation), the layer's residual stream can be mapped to logits via the same unembedding function used for token selection. As shown in FIG. 1, xi+2is further processed in the unembedding layer 130. xi+2may be the hidden state of the transformer model and may be mapped through the unembedding function in the unembedding layer 130 to produce a score distribution over the vocabulary of the model. The unembedding layer 130 produces logics 102 by performing unembedding: T(t) = I / l / y^ ^ . The logics 102 may refer to the model's raw, unnormalized output values before applying an activation function, such as Softmax, Sigmoid, and so on. The logics 102 correspond to tokens generated at the residual block 120. The top token(s) may be selected from the tokens by various token selection criteria. In a greedy algorithm for example, the token with the highest logits value is selected as the top token. Logits value may indicate a probability that the token is a correct prediction. Logits value may also be referred to as token score or confidence score. In other examples, top- / < tokens having token scores higher than the other tokens or higher than a threshold score are selected. Token selection may be performed at the last residual block (i.e., the last / fi na I layer) to get predicted token. Additionally or alternatively, token selection may be performed earlier at an intermediate residual block (i.e., an early / intermediate layer) for speculative precomputation.

[0040] FIG. 2 is a block diagram of an Al system 200 with heterogenous hardware, in accordance with various embodiments. The Al system 200 includes a prescient paraclete module 210, a CPU 220A, a GPU 220B, and an NPU 220C. In other embodiments, alternativeconfigurations, different or additional components may be included in the Al system 200. For instance, the Al system 200 may include multiple CPUs, GPUs, or NPUs. Also, the Al system 200 may include other types of processing units. Further, functionality attributed to a component of the Al system 200 may be accomplished by a different component included in the Al system 200 or a different system. For instance, functionality attributed to the prescient paraclete module 210 may be accomplished by one or more threads on the CPU 220A, GPU 220B, or NPU 220C. The CPU 220A, GPU 220B, and NPU 220C may be collectively referred to as heterogenous processing units 220, individually referred to as "heterogenous processing unit 220." The heterogenous processing units 220 may be implemented in separate chips. In an example, each heterogenous processing unit 220 may be implemented as a separate chip. Even though a single CPU, GPU, or NPU is used in some embodiments or examples described below, multiple CPUs, GPUs, or NPUs may be used in these embodiments or examples.

[0041] The prescient paraclete module 210 manages acceleration of transformer inference processes by the heterogenous processing units 220 in the Al system 200. Transformer inference processes may be done for performing Al tasks. Al tasks may be tasks for various Al applications, such as language processing, speech recognition, image or video analysis, autonomous driving, project management, and so on. An Al task involves deployment of a DNN, e.g., a transformer model. The deployment of a transformer model may be transformer inference, which is a process of applying a pretrained transformer model on an input to obtain tokens predicted by the transformer model. Transformer deployment may involve a sequence of transformer inference processes for predicting a sequence of tokens. The prescient paraclete module 210 may optimize the efficiency of Al tasks by accelerating transformer inference. The prescient paraclete module 210 can leverage the heterogenous processing units 220 and LLP to exploit the predictive capabilities of intermediate transformer layers.

[0042] As shown in FIG. 2, the prescient paraclete module 210 includes an interface module 230, a predictive inference module 240, a speculative inference module 250, a match detector 260, a precomputation tuning module 270, and a hardware manager 280. The prescient paraclete module 210 may include fewer, more, or different components. Further, functionality attributed to a component of the prescient paraclete module 210 may be accomplished by a different component included in the prescient paraclete module 210 or adifferent module. In some embodiments, the prescient paraclete module 210 may run on a CPU, e.g., the CPU 220A, or a microcontroller. The prescient paraclete module 210 may play an important role in coordinating distribution of speculative precomputation of future tokens and handling of aborting of mis-speculation and fast forward / reuse of correctly speculated computation and associated sequence of future tokens. CPU or microcontroller may be used in the case of multi-node cluster (such as single-system-image (SSI), MPI / Modin, etc.) and scale-out distributed computing environment like Zerum Machine Learning® (ZML).

[0043] The interface module 230 facilitates communications of the prescient paraclete module 210 with other modules or systems. In some embodiments, the interface module 230 establishes communications between the prescient paraclete module 210 with an external database or system to receive data that can be used to perform Al tasks. For example, the interface module 230 may receive a user input from a client device associated with the user. The user input may be a prompt or may be used by the prescient paraclete module 210 to generate a prompt. The prompt may include one or more input tokens and may be an input to a transformer model. As another example, the interface module 230 may receive images from a camera for image or video processing. As yet another example, the interface module 230 may receive sensor data from a robot for robot navigation. In some embodiments, the interface module 230 may transmit data generated by the prescient paraclete module 210 to the heterogenous processing units 220 for transformer inference process. The interface module 230 may also facilitate transmission of data used for controlling operations of the heterogenous processing units 220 to the heterogenous processing units 220. The data may include executable instructions generated by the hardware manager 280, the predictive inference module 240, or the speculative inference module 250.

[0044] The predictive inference module 240 manages predictive inference processes of transformer models. Predictive inference may also be referred to as main inference. A predictive inference process includes execution of all the layers of the transformer model based on input token(s) or predicted token(s). A predictive inference process may produce a predicted token. The predicted token may be generated at a final layer of the transformer model, e.g., from execution of all the layers in the transformer model. In some embodiments, there may be a sequence of predictive inference processes of a transformermodel to produce a sequence of predicted tokens. In an example, the first predictive inference process may start with inputting one or more input tokens into the transformer model. The first predicted token may be generated at the final layer of the transformer model after all the layers are executed. For the second predictive inference process, the first predicted token may be appended to the one or more input tokens to form a sequence. The sequence is input into the transformer model, and the second predicted token is generated at the final layer. For the third predictive inference process, the second predicted token may be appended to the sequence to form another sequence, which is input into the transformer model, and the third predicted token is generated at the final layer. This may continue till all the predicted inference processes are done.

[0045] The predictive inference module 240 may initiate predictive inference tasks. A predictive inference task may be a task of performing one or more predictive inference processes of a transformer model to generate one or more predicted tokens. In some embodiments, the predictive inference module 240 may initiate a predictive inference task after the prescient paraclete module 210 receives a request to perform an Al task using the transformer model. The predictive inference module 240 may send information of the predictive inference task to the hardware manager 280 to allocate a thread to carry out the predictive inference task. Certain aspects of allocating threads are described below in conjunction with the hardware manager 280.

[0046] The predictive inference module 240 may also control the progress of a predictive inference task. For instance, the 240 may terminate a predictive inference task while the predictive inference task is being performed. In some embodiments, the predictive inference module 240 may terminate a predictive inference task so that the Al task can be fast forwarded to a corresponding speculative inference task. In some embodiments, after the predictive inference module 240 terminates the predictive inference task, the main thread (i.e., the thread performing the predictive inference task) would not start the next predictive inference of the transformer model after the predictive token is generated.

[0047] In some embodiments, the predictive inference module 240 may perform just-in- time optimization of codes to be executed by the heterogenous processing units 220. The codes may include, for example, x86 code for the CPU 220A or kernel code for the GPU 220B or NPU 220C. The just-in-time optimization may enable the target hardware destinated to run speculative precomputation thread differs in architecture (or potentially uArch) fromthe heterogenous processing unit 220 on which the main transformer computation is performed. In some embodiments, the predictive inference module 240 may perform the just-in-time optimization using compilation and runtime techniques. The predictive inference module 240 may perform the just-in-time optimization during runtime, such as during an inference process. The predictive inference module 240 may perform the just-in- time optimization by re-compiling the transformer model. In some embodiments, the target hardware on the receiving end of performing speculative precomputation for accelerating transformer inference may vary in architecture, uArch, capability, system configuration, resource sharing, geo-location, or colocation. Dynamic optimization may entail just-in-time adaptation of arithmetic intensity and precision. For example, the speculative precomputation thread can perform potentially aggressive quantization optimization and pruning on attention heads exhibiting low attention scores as computation progress through layers, thus making speculative precomputation even faster and ahead.

[0048] In some embodiments, the predictive inference module 240 may compress a transformer model by reducing the precision of data in the transformer model or introducing sparsity into the data in the transformer model. The compression of the transformer model may be performed for a predictive inference task so that the time needed for generating the final token can be shorter and the match detection can be conducted earlier. In some embodiments, the predictive inference module 240 may compress a transformer model based on configuration of the processing unit that the thread is on. For instance, the predictive inference module 240 may change data precision to a data precision that the processing unit supports. In some embodiments, the predictive inference module 240 may reduce the data precision of input data of one or more layers of the transformer model. The predictive inference module 240 may instruct the thread to execute the one or more layers with the input data having the reduced data precision.

[0049] The speculative inference module 250 manages speculative inference processes of transformer models. Speculative inference process is transformer inference process based on speculated token(s). A speculated token is generated at an intermediate layer of the transformer model, e.g., from execution of a subset of the layers in the transformer model. One or more layers in the transformer model (e.g., the final layer) are not executed yet when the speculated token is generated. A speculated token may be generated during apredictive inference process. Speculated tokens may also be referred to as token candidates.

[0050] The speculative inference module 250 may initiate one or more speculative inference tasks for a predictive inference task initiated by the predictive inference module 240. In an example, during the predicted inference process, the speculative inference module 250 may extract one or more tokens generated at an intermediate layer of the transformer model. The intermediate layer is a layer that is precedent to the final layer in the transformer model. The one or more tokens are speculated tokens as they are generated from execution by a part of the transformer model, as opposed to the entire model. The speculative inference module 250 may generate a new prompt by appending a speculated token to the sequence used for the inference process in which the speculated token is generated and initiate a speculative inference task by inputting the new prompt into the transformer model. When there are multiple speculated tokens, the speculative inference module 250 initiates multiple speculative inference tasks, each of which uses a respective one of the speculated tokens.

[0051] For instance,, the speculative inference module 250 extracts K token candidate(s) from the intermediate layer, where K > 1. K may be a precomputation parameter determined by the precomputation tuning module 270. A number of tokens may be generated at the intermediate layer. The confidence of the transformer model for these tokens are usually different. The speculative inference module 250 may select K token candidate(s) based on the confidence score of each token. The confidence score of a token indicates the degree of confidence of the transformer model for the token. The speculative inference module 250 may compare the confidence score of each toke with a threshold score and select token(s) having confidence score(s) greater than the threshold. Additionally or alternatively, the speculative inference module 250 may rank the tokens based on their confidence scores and select K token candidate(s) with the highest confidence score(s). The speculative inference module 250 may use each of the K speculated tokens to kick off K parallel speculative precomputation by appending each candidate token to the current prompt. The current prompt may be the input sequence for the predicted inference process. The K parallel speculative precomputation threads may run in parallel with the main thread which has not reached its final layer.

[0052] A speculative inference task includes at least one speculative inference process, i.e., the speculative inference process based on the speculated token generating during the predictive inference process. This speculative inference process may be the first inference process of the speculative inference task. The next inference process (or subsequent inference process(es)) may be another speculative inference process or a predictive inference process. The speculative inference module 250 may facilitate multiple modes for speculative inference tasks. In an example, the speculative inference module 250 may initiate speculative inference tasks in a frugal paraclete mode. In the frugal paraclete mode, the first inference process of the speculative inference task is a speculative inference tasks, and each subsequent inference process is a predictive inference process. Alternatively, the speculative inference module 250 may initiate speculative inference tasks in an eager paraclete mode. In the eager paraclete mode, a speculative inference task has multiple speculative inference processes. These speculative inference processes may be followed by one or more predictive inference processes. Certain aspects regarding different modes for speculative inference tasks are described below in conjunction with FIGS. 4 and 5.

[0053] In embodiments where there are multiple speculated tokens and multiple speculative inference tasks, the speculative inference tasks may be performed in parallel while the predictive inference process continues towards the final layer. For some embodiments of the PLT approach, through early token candidate prediction, a distinct inference process path is established for each speculated token based on the current prompt appended with the predicted token. When this occurs early enough, the speculative precomputation of the next token can run in parallel with the main inference process path that continues through the remaining layers. In some embodiments, the speculative inference module 250 may run-aside, run-behind monitor and manage parallel computation or hardware performing of kind computation that runs in the shadow of a given main transformer computation thread. It may be of fly-weight and timely track transformer computation at attention head granularity or layer granularity. It may have full observability to the residual stream (aka hidden state) evolution and can judiciously snapshot of the full residual stream or a subset and use that snapshot to perform fast determination of top token candidates with high probability of being the final token, at early layer(s). In some embodiments, the speculative inference module 250 may schedule and distribute speculative precomputations initiated from the candidate tokens.

[0054] In some embodiments, for each of the speculative precomputation path, the speculative inference module 250 may extract speculated tokens from the speculative inference processes and initiate new speculative inference tasks. This can rapidly establish a tree of speculative computation groups rooted at the original token candidates identified from the predictive inference process. This nesting expansion of speculative computation may be limited by the amount of compute resources available in the Al system 200 at the scale of interest. For instance, there may be a single node with one GPU, with multiple GPUs, or a cluster of multiple nodes with a single or multiple GPUs. There may even be a geo-dispersed virtual cluster of participating nodes across the internet, e.g. using ZML or Modin framework. All speculative precomputations in this ensemble may be parallel amongst each other. Speculative precomputation may be on the same node sharing resource (e.g., virtual memory) with the main thread (i.e., the thread performing the predictive inference process). The speculative threads (i.e., the threads performing the speculative inference processes) may exhibit extreme constructive interference due to spatial and temporal locality of shared model parameters and KV data structures. The KV data done by the precomputation initiated on correctly predicted token can be reused by the main thread when it catches up the pace to the leading speculative precomputation on the "correct path" with matching tokens leading to this round.

[0055] The match detector 260 detects matches between speculated tokens and predicted tokens. For instance, the match detector 260 may determine whether any speculated token(s) extracted by the speculative inference module 250 for a predictive inference process matches the predicted token produced from the predictive inference process. The match detector 260 may make the determination after the predictive inference process is complete, e.g., once the predictive inference process is complete. In some embodiments, the determination by the match detector 260 may be triggered by the completion of the predictive inference process. The match detector 260 may receive a notification from the predictive inference module 240 or the thread performing the predictive inference process that the predictive inference process is complete. In response to the notification, the match detector 260 may detect whether there is any match. The speculative inference task(s) initiated by the speculative inference module 250 based on the speculated token(s) may be underway when the match detector 260 makes the match detection.

[0056] When the match detector 260 detects a match, the match detector 260 may retire the main thread or instruct the predictive inference module 240 to retire the main thread. The predictive inference task initiated by the predictive inference module 240 may be terminated. The match detector 260 may present the main thread from performing another inference process for the predictive inference task. The speculative inference task triggered by the speculative token matching the predicted token may continue. This way, the transformer inference can fast forward to the speculative thread initiated with the correct prediction. The speculative thread may become the new main thread, thus benefiting from run-ahead execution. When the match detector 260 detects no match, the match detector 260 may terminate the speculative inference task(s) or instruct the speculative inference module 250 to terminate the speculative inference task(s). For instance, the match detector 260 may present the spread thread(s) from performing any further computation for the speculative inference task(s). Timely halt of the speculative inference task(s) can avoid waste of resources. In some embodiments, the main thread may terminate the speculative thread(s) and continue with the next token prediction in the predictive inference task. The match detector 260 may manage fast forward of the main thread to catch up speculative precomputation initiated by the correctly speculated token.

[0057] The match detector 260 may also coordinate pause or reuse when any speculative precomputation thread was initiated by a mis-predicted token. For instance, after determining that the token candidate is mis-predicted (i.e., the token candidate does not match the predicted token), the match detector 260 may pause the speculative inference task triggered by the token candidate, whether another token candidate matches the predicted token or not. The match detector 260 may facilitate reuse of data generated in the speculative inference task, such as KV cache generated while performing the speculative inference task. The match detector 260 may provide the reusable data to the main thread or the speculative thread that is not halted.

[0058] In some embodiments, functionality attributed to the match detector 260, such as the functionality described above, may be accomplished by the main thread. For instance, when the main thread on its own pace reaches the final layer and makes the final token prediction, it can check the final token against all token candidates that were used in to kick off respective parallel speculative precomputation. On incorrect prediction (i.e., when no match is found), the main thread may retire all the speculative threads. For instance, themain thread may signal all the speculative threads to halt by writing to a shared memory flag amongst speculative threads sharing memory. Alternatively, the main thread may broadcast a flag (e.g. via a direct memory access (DMA) engine) to set up the flag on other nodes which do not share virtual memory with the main thread. These other nodes may include collocated nodes or geo-dispersed nodes (e.g. ZML). The speculative inference module 250 may monitor the flag and enact halt of the transformer computation when the initiating token is signaled as mis-predict. On a correct prediction (i.e., when a match is found), the main thread may retire itself. The thread of speculative precomputation may run further ahead and may be effectively anointed as the main thread as it would not see its mis-speculation flag set. For all the mis-speculated tokens, the respective speculated computed token sequences may not need to be totally discarded. There is a possibility that some common sequences produced by some of the speculative precomputation initiated on the mis-speculated tokens may still be deemed as good and usable by the main thread, thus allowing support for the main thread to reuse and fast forward. This level of reuse of token sequences generated by speculative precomputation initiated on "wrong" token candidate may be domain specific. The CPU 220A can be enlisted to do run-behind synthesis to decide whether and when such reuse is honored.

[0059] The precomputation tuning module 270 tunes hyperparameters for accelerating transformer inference. Tuning a hyperparameter may include adjusting the value of the hyperparameter. In some embodiments, the hyperparameters for transformer inference may include sliding window size W, vocabulary size D, and number of token candidates K. W indicates the size of the window of input tokens. W may equal the input prompt length in some embodiments. D may indicate the size of a vocabulary subset to perform logits / probability-based determination of token candidates. D may be a model parameter for full vocabulary. K is the number of high-confidence candidates to initiate speculative precomputations. K may be an integer that is 1 or greater. The precomputation tuning module 270 may tune one or more hyperparameters to tune a dynamic range of options to balance the degree of speculation (or eagerness) to match the amount of compute resources available to support parallel speculative run-ahead precomputation.

[0060] The hardware manager 280 manages and dispatches the heterogenous processing units 220 for accelerating transformer inferences through speculative precomputations. In some embodiments, the hardware manager 280 may find threads on the heterogenousprocessing units 220 and allocate these threads to predictive inference tasks initiated by the predictive inference module 240 or speculative inference tasks initiated by the speculative inference module 250. The hardware manager 280 may assign each inference to a different thread. A thread performing a predictive inference task may be switched to performing a speculative inference task, or vice versa. In some embodiments, a thread may be one or more compute units in a heterogenous processing unit 220. For instance, a thread may include one or more computing cores, computing units, streams, microprocessing threads, or microprocessors in a heterogenous processing unit 220. Multiple threads may share a memory, which can facilitate data sharing among the threads. For instance, a thread may save data (e.g., KV cache, tokens, etc.) into the shared memory, and another thread may read the data from the shared memory and use the data in its task.

[0061] In some embodiments, the hardware manager 280 may identify available threads on the heterogenous processing units 220. An available thread may be a thread that is not performing any inference at the time. The hardware manager 280 may map available threads to inference tasks specified by the predictive inference module 240 or speculative inference module 250. For instance, the hardware manager 280 may select an available thread for a predictive inference task. The hardware manager 280 may select one or more other threads for speculative inference task(s) associated with the predictive inference.

[0062] In some embodiments, to find a thread for a transformer inference, the hardware manager 280 may check the compatibility of the transformer inference with available threads. For instance, the hardware manager 280 may determine whether a thread has sufficient components to support the requirements of the computations in the transformer model. Examples of the requirements may be data format, computation type, latency requirement, and so on. The hardware manager 280 may evaluate configurations of the heterogenous processing units 220 and characteristics of the transformer inference for the mapping. For example, the hardware manager 280 may consider the threads on the GPU 220B and NPU 220C desirable for crucial inferences as transformer models may be designed to leverage the parallel processing capabilities of the GPU 220B or NPU 220C. The speculative inference(s) may be considered more crucial than the predictive inference in some cases. As another example, the hardware manager 280 may consider the threads on the CPU 220A more desirable for transformer inference requirement heavy data traffic as the CPU 220A may require less data movement than the GPU 220B or NPU 220C. In anexample, the hardware manager 280 may select a thread on the CPU 220A or NPU 220C for a predictive inference. In another example, the hardware manager 280 may select a thread on the GPU 220B or NPU 220C for a speculative inference.

[0063] The hardware manager 280 may and instruct the threads to perform the inference tasks. The hardware manager 280 may facilitate transfer of data needed to execute a transformer model, such as input sequence, to the GPU 220B or NPU 220C. For instance, the hardware manager 280 may command the GPU 220B or NPU 220C to read the data from a memory or command another processing unit (e.g., a previously used processing unit) to transfer the data to the GPU 220B or NPU 220C. The data may include configuration descriptors generated from compiling the transformer model. Components of the GPU 220B or NPU 220C may be configured by the configuration descriptors for the transformer model execution.

[0064] In some embodiments, the heterogenous processing units 220 may share memory physical addresses. The heterogenous processing units 220 may have varying amount of compute intensity. For instance, the CPU 220A may have 5TOPS, the GPU 220B may have 100 TOPS, while the NPU 220C may have 50TOPS. The heterogenous processing units 220 may have different advantages for different needs. For instance, the CPU 220A may be a great hardware for decision making threads / modules that coordinate to send the right subset of transformer computation to the right hardware. The CPU 220A or GPU 220B, when idle, may beget its own speculative precomputation threads. For the speculative precomputation thread dispatched to the NPU 220C, it may not be able to spawn its speculative precomputation due to limited amount of resources. The NPU 220C may run speculative precomputation. When the NPU 220C detects high-probability final token candidate at the early layer, it can signal to the CPU 220A which identifies the right target to run speculative precomputation using the high-confidence token prediction. In a system with powerful CPU(s), GPU(s), and NPU(s), a broad spectrum of judicious use of the underlying hardware can be achieved to fulfill role of running or managing speculative precomputations that together in harmony can drastically accelerate the overall transformer computation.

[0065] In some embodiments, for each available thread, the hardware manager 280 may estimate various types of costs of the thread performing the inference task. The costs may include a latency cost indicating an estimation of the latency caused by performing theinference task, an energy cost indicating an estimation of energy consumed by the thread for performing the inference task, a performance cost indicating an estimation of a performance of the thread for performing the inference task, and so on. The hardware manager 280 may use cost models that can output estimates for latency, energy consumption, and performance per watt. The hardware manager 280 may select a thread for an inference task based on the estimated cost. For instance, the hardware manager 280 may select the thread with the lowest cost for the inference task.

[0066] For each new token prediction, the transformer computation may depend on the model parameters and the prompt and be otherwise stateless. This means any speculated token results in a new prompt ended with this token can be speculatively executed on any processing unit available in the system including the same processing unit that performs the main transformer thread computation, and the payload of communication is literally as minimal as a token or as large as a prompt which can be prefetched and prepared ahead after encoding layer away ahead of any non-speculative or speculative computation. Given the high confidence of early start of high probability and stateless speculative computation on any available processing unit resources in a given system of scope of single node, multinode or even over LAN / WAN / interconnect scale, such speculative can scale, due to the minimal amount of payload (many order magnitude smaller than activation communication by scale-out clustering / pod environment), thus should work as well like the latest internetscale inference by heterogenous and geo-scale-out framework like ZML.

[0067] FIG. 3 illustrates a prescient layer-wise transformer timeline, in accordance with various embodiments. The pipeline has three players: a main thread 310, speculative thread(s) 320, and prescient paraclete module 330. An example of the prescient paraclete module 330 may be the prescient paraclete module 210 in FIG. 2. The main thread 310 and speculative thread(s) 320 may be run on the heterogenous processing units 220 in FIG. 2. For the purpose of illustration, FIG. 3 shows seven timestamps in the timeline: t0through t6and critical events occurred at these times. There may be other critical events at other times not shown in FIG. 3.

[0068] At t0, the main thread 310 starts predictive inference process N. N denotes the index of the inference process in an inference task. The inference task may be a predictive inference task. In an example, when N = 0, predictive inference process N is the first inference process in the predictive inference task. When N = 1, predictive inferenceprocess N is the second inference process in the predictive inference task. The main thread performs predictive inference process N by executing all the layers in the transformer model.

[0069] At t1;the prescient paraclete module 330 extracts K token candidate(s) from the transformer model and initiates K speculative inference tasks. The K token candidate(s) may be generated at an intermediate layer of the transformer model before inference process N is complete.

[0070] At t2, the speculative thread(s) 320 starts speculative inference process(es) N + 1. Each speculative inference process N + 1 may be the first inference process in a speculative inference task using the sequence. Each speculative inference task may be triggered by a respective token candidate and performed by a respective speculative thread. In embodiments where N > 1, each speculative inference task may start with a sequence including one or more tokens that have been predicted by the transformer model through the previous inference process(es) of the predictive inference task. The sequence may be input into the transformer model to start the speculative inference process(es) N + 1.

[0071] At t3, the main thread 310 completes predictive inference process N and produces predicted token N. The token may be generated at the last layer of the transformer model.

[0072] At t4, the prescient paraclete module 330 detects whether there is a match between any of the K token candidate(s) and predicted token N. In some embodiments, there is a match when a token candidate encodes the same word(s) as predicted token N.

[0073] In embodiments where the prescient paraclete module 330 detects no match, the main thread 310 starts predictive inference process N + 1 at t5. Also, the prescient paraclete module 330 halts speculative inference by the speculative thread(s) 320 at t5. Even though not shown in FIG. 3, the prescient paraclete module 330 may also facilitate the main thread 310 to reuse data generated by the thread(s) 320 from performing the speculative inference process(es) N + 1. For instance, the prescient paraclete module 330 may instruct the thread(s) 320 to store the data in a shared memory accessible to the main thread 310. The prescient paraclete module 330 may also instruct the main thread 310 to read the data from the memory and use the data for inference process N + 1. The events for these embodiments are represented by dotted boxes in FIG. 3.

[0074] In embodiments where the prescient paraclete module 330 detects a match, the main thread 310 retires at t5. The speculative thread for the correctly speculated token (i.e.,the token candidate that matches predicted token N) may continue speculative inference process N + 1 triggered by the correctly speculated token. The speculative thread may become the new main thread. The other speculative thread(s), if any, may retire too as their speculative inference tasks are based on incorrectly speculated token. The event for these embodiments is represented by a dash box in FIG. 3.

[0075] At t6, either the speculative thread finishes speculative inference process N + 1 using the correctly speculated token or the main thread 310 finishes predictive inference process N + 1 using predicted token N. New token N + 1 is produced from speculative inference process N + 1 or predictive inference process N + 1. Even though the same timestamp is used for the completion of speculative inference process N + 1 and predictive inference process N + 1 in FIG. 3, speculative inference process N + 1 may be complete earlier than predictive inference process N + 1 as speculative inference process N + 1 starts earlier. Also, in some embodiments, speculative inference process N + 1 may include execution of some, not all, layers of the transformer model, while predictive inference process N + 1 may include execution of all layers of the transformer model.

[0076] FIG. 4 illustrates acceleration of transformer inference in a frugal paraclete mode, in accordance with various embodiments. For the purpose of illustration and simplicity, the transformer model shown in FIG. 4 has four layers. The transformer model may have four heads. FIG. 4 shows a predictive inference task 410, which is an example of the predictive inference tasks described above. For the purpose of illustration and simplicity, the predictive inference task 410 includes four inference processes through which four tokens are predicted: TO through T3. The first inference process for producing TO is represented by blank boxes. The second inference process for producing T1 is represented by boxes with a dot pattern. The third inference process for producing T2 is represented by boxes with a diamond grid pattern. The fourth inference process for producing T1 is represented by boxes with a pattern of diagonal strips. Each inference process in the predictive inference task 410 includes execution of all the four layers. The first inference process is done using one or more input tokens and produces TO. The second inference process is done using the input token(s) and TO and produces Tl. The third inference process is done using the input token(s), TO, and Tl and produces T2. The last inference process is done using the input token(s), TO, Tl, and T2 and produces T3.

[0077] FIG. 4 also shows a speculative inference task 420. The speculative inference task 420 is performed in the frugal paraclete mode. The speculative inference task 420 is initiated based on the first inference process of the predictive inference task 410. A speculated token (T0_SP) generated at the second layer of the transformer model is identified and triggers the speculative inference task 520. A match between T0_SP and TO may be detected after TO is produced in the predictive inference task 410. In some embodiments, in response to detecting the match, the predictive inference task 410 may be terminated, and the speculative inference task 420 may continue. The speculative thread performing the speculative inference task 420 may become the new main thread. The speculative inference task 420 includes three inference processes: the first one is done using the input token(s) and T0_SP to produce Tl, the second one is done using the input token(s), T0_SP, and Tl to produce T2, and the last one is done using the input token(s), T0_SP, Tl, and T2 to produce T3. As shown in FIG. 4, each of the three inference processes include execution of all the four layers of the transformer model.

[0078] The initiation of the speculative precomputation at the end of the second layer can fast forward the transformer inference and save resources. As shown in FIG. 4, the speculative inference task 420 can finish earlier than the predictive inference task 410. The time saved is shown as time 401 in FIG. 4, which may be the time required to execute two layers.

[0079] FIG. 5 illustrates acceleration of transformer inference in an eager paraclete mode, in accordance with various embodiments. For the purpose of illustration and simplicity, the transformer model shown in FIG. 5 has four layers. The transformer model may have four heads. FIG. 5 shows a predictive inference task 510, which is the same as the predictive inference task 410 described above in conjunction with FIG. 4.

[0080] FIG. 5 also shows a speculative inference task 520. The speculative inference task 520 is performed in the eager paraclete mode. The speculative inference task 520 is initiated based on the first inference process of the predictive inference task 510. A speculated token (T0_SP) generated at the second layer of the transformer model is identified and triggers the speculative inference task 520. A match between T0_SP and TO may be detected after TO is produced in the predictive inference task 510. In some embodiments, in response to detecting the match, the predictive inference task 510 may beterminated, and the speculative inference task 520 may continue. The speculative thread performing the speculative inference task 520 may become the new main thread.

[0081] The speculative inference task 520 includes three inference processes. The first inference process is done using the input token(s) and T0_SP. In the eager paraclete mode, not all the four layers are executed during the first inference process. Rather, the first two layers are executed but the other two layers are bypassed. Another speculated token, T1_SP, is extracted from the second layer. T1_SP may be compared with T1 to detect whether they match. After T1_SP is produced, the second inference process is done using the input token(s), T0_SP, and T1_SP to produce T2. Then the third inference process is done using the input token(s), T0_SP, T1_SP, and T2 to produce T3. In the embodiments of FIG. 5, all the four layers are executed in the second inference process and third inference process. In other embodiments, the second inference process or the third inference process may include the execution of the first two layers but not the execution of the last two layers.

[0082] The speculative precomputation shown in FIG. 5 begets its own speculative precomputation at an early layer (i.e., the speculative precomputation in the first inference process of the speculative inference task 520). When the early prediction of TO and T1 are accurate, the speculative precomputation in the eager paraclete mode can save more resources than the speculative precomputation in the frugal paraclete mode shown in FIG. 4. In FIG. 5, the speculative inference task 520 finishes earlier than the predictive inference task 510 and the time saved is time 502, which may be the time required to execute four layers. The overall computation is accelerated further. In some embodiments, the eager paraclete mode may require more compute resources. For instance, another speculative thread may be needed to perform the new speculative precomputation. The eager paraclete mode may be applied when there are sufficient compute resources available. Whether there are sufficient compute resources available or not may be determined by the hardware manager 280 in FIG. 2.

[0083] FIG. 6 illustrates layer-level tokens generated in the first inference process of a transformer model, in accordance with various embodiments. For the purpose of illustration, the transformer model has 48 layers, i.e., Layer 0-47 as shown in FIG. 6. In some embodiments, the transformer model has 48 heads corresponding to the 48 layers. Layer 47 is the last layer or final layer. Layers 0-46 are intermediate layers. A number of tokens maybe generated at each layer. The tokens generated at the same layer have different confidence scores. A higher confidence score indicates that the layer or the transformer model has a higher confidence for the token. The tokens may be ranked based on the confidence scores. For the purpose of illustration, FIG. 6 shows the top-3 tokens for each layer. In an example, Token 1 may have the highest confidence score, Token 2 may have the second highest confidence score, and Token 3 may have the third highest confidence score.

[0084] In an example, the first inference process starts with receiving an input sequence: "once upon a time in a land far, far away" that includes a number of input tokens. Token 1 generated at Layer 47 is the predicted token as it has the highest confidence score and is generated at the last layer of the transformer model. In this example, the predicted token is a comma. FIG. 6 shows that the comma has been predicted at earlier layers, such as Layer 18, Layer 19, Layer 20, and Layer 46. That means in this transformer computation instance, a probability of 1 can be achieved much sooner than the last layer. An early exit at Layer 18 would not influence the accuracy of the transformer model.

[0085] For situations like this one, speculative precomputation can significantly accelerate the transformer inference. In some embodiments, a main thread may perform the first inference process that includes execution of all the 48 layers. Three speculative threads may be initiated, e.g., when the execution of Layer 18 by the main thread is complete, based on the three tokens generated at Layer 18. The first speculative thread may start the second inference process using Token 1 generated at Layer 18 (i.e., "where"), the second speculative thread may start the second inference process using Token 2 generated at Layer 18 (i.e., "from"), and the third speculative thread may start the second inference process using Token 3 generated at Layer 18 (i.e., ","). After the main thread finishes the execution of Layer 47, the three tokens generated at Layer 18 are compared with Token 1 generated at Layer 47 to determine whether there is a match. In the example shown in FIG. 6, Token 3 generated at Layer 18 matches Token 1 generated at Layer 47. In response to the detection of the match, the main thread retires, and the third speculative thread replaces the main thread to become the new main thread. This way, the main thread can simply skip Layers 19-47 and start to catch up on the speculative precomputation threads.

[0086] In response to determining that Token 1 and Token 2 generated at Layer 18 does not match Token 1 generated at Layer 47, the first speculative thread and the second speculative thread retire and their speculative inference tasks are terminated. In someembodiments, data generated by the original main thread, the first speculative thread, or the second speculative thread may be reused by the third speculative thread for further inference process(es). The data may include KV cache data or other types of data.

[0087] In many cases, top token predictions can stabilize in earlier layers for a significant portion of generated tokens. This phenomenon suggests that intermediate layers often capture sufficient contextual information to make accurate predictions. By analyzing the layer-wise logits (e.g., the raw outputs before the activation function, such as Softmax) at each layer, candidate tokens likely matching the final output can be identified since the logits represent the model's confidence scores for each token in the vocabulary. In the example of FIG. 4, the top 3 token candidates of each layer can be identified based on logits at each layer. All the token candidates that match the final token (i.e., Token 1 generated at Layer 47) are highlighted with a dotted pattern in FIG. 6.

[0088] FIG. 7 illustrates layer-level tokens generated in the second inference process of the transformer model in FIG. 6, in accordance with various embodiments. The second inference process may be performed after the first inference process. The sequence input into the transformer model for the second inference may be "once upon a time in a land far, far away," that includes the original input sequence appended by the comma predicted in the first inference process. The second inference process may be performed by the third speculative thread described above in conjunction with FIG. 6, which may be the main thread for the second inference process.

[0089] The main thread may perform the second inference process that includes execution of all the 48 layers. Token 1 generated at Layer 47 (i.e., "there") is the final token of the second inference process. As shown in FIG. 7, "there" can be predicted as early as Layer 23. In some embodiments, three speculative threads may be initiated when the execution of Layer 23 by the main thread is complete based on the three tokens generated at Layer 23. The speculative precomputation here is begotten from the speculative precomputation described above in conjunction with FIG. 6. For the speculative precomputation here, the first speculative thread may start the third inference process using Token 1 generated at Layer 23 (i.e., "where"), the second speculative thread may start the second inference process using Token 2 generated at Layer 23 (i.e., "inhabited"), and the third speculative thread may start the second inference process using Token 3 generated at Layer 23 (i.e., "there"). After the main thread finishes the execution of Layer 47, the three tokensgenerated at Layer 23 are compared with the final token to determine whether there is a match. In the example shown in FIG. 7, Token 3 generated at Layer 23 matches the final token. In response to the detection of the match, the main thread retires, and the third speculative thread replaces the main thread to become the new main thread. This way, the main thread can simply skip Layers 24-47 and start to catch up on the speculative precomputation threads.

[0090] In response to determining that Token 1 and Token 2 generated at Layer 23 does not match the final token, the first speculative thread and the second speculative thread retire and their speculative inference tasks are terminated. In some embodiments, data generated by the original main thread, the first speculative thread, or the second speculative thread may be reused by the third speculative thread for further inference process(es). The data may include KV cache data or other types of data. The speculative precomputation here may beget further speculative precomputation to further accelerate inference of the transformer model.

[0091] FIGS. 6 and 7 demonstrate a common phenomenon in inference processes using pretrained models. The final token determined at the final layer can often be predicted early when early layers' logits are examined. One or more parameters may be used to quantify the predictability and stability of token predictions across layers. Examples of the parameters includes E(t), which denotes the earliest candidate layer, the earlier layer at which a token t appears in the top- / < predictions; S(t), which denotes the stable layer, i.e., the layer after which the token t remains consistently in the top- K predictions across subsequent layers; Z7(t), which denotes the unique prediction set, i.e., the set of unique tokens that appear in the top- K predictions for token t across all layers. The evolution of predictable tokens may not be necessarily monotonic or converging. For example, in FIG. 6, the final token appears among the top-3 tokens around the middle layers and then fades away until Layer 44, which is close to the final layer. In contrast, in FIG. 7, a high-confidence stable prediction is made much sooner, and the distribution represented by the logits settles earlier. In such cases, computation after S(t) becomes less impactful on the final prediction.

[0092] The intuition behind E(t) is that the sooner token candidates can be identified at E(t), the greater the advantage in performing run-ahead precomputations, even though candidates at an earliest candidate layer may not exhibit stability like those at a stable layer.In some cases (e.g., the example shown in FIG. 7), both metrics may not differ significantly, indicating that it can be easier to establish high-confidence early predictions that remain stable.

[0093] Another dimension of trade-off is the value of K. For the purpose of illustration, FIGS. 6 and 7 shows top-3 token candidates of each layer, i.e., K = 3. In other embodiments, K may have other values. In some embodiments, as K increases, both E(t) and S(t) may lower further, thus exposing further headroom for advantageous run-ahead opportunities. Such potential can be balanced by the increase of U (t) the set of unique token candidates that do not match the final token. The innate uncertainty for early prediction can increase as more candidates come under consideration, even though there may be higher likelihood that a winning candidate is amongst the candidates.

[0094] These parameters can help characterize both opportunities for successful early precomputation and the potential waste of resources dedicated to precomputations initiated by mis-predicted token candidates. Such phenomena are analogous to trade-offs considered in microarchitectural innovations to balance the benefits of profitable speculation under resource constraints. In this context, the notion of criticality may be foundational in prioritizing resources towards computations that contribute to the critical path bottleneck.

[0095] FIG. 8 illustrates an example transformer model 800, in accordance with various embodiments. The transformer model 800 may transform input sequences into output sequences. In some embodiments, the transformer model 800 is a DNN that can learn context and meaning by tracking relationships in sequential data, such as sequential words in a sentence, sequential audio signals, sequential images, and so on. In an example, the transformer model 800 may be at least part of an LLM. The transformer model 800 may be an example of the transformers described above. As shown in FIG. 8, the transformer model 800 includes an encoder block 810, a decoder block 820, and a head block 830. In other embodiment, different or additional components may be included in the transformer model 800. Further, functionality attributed to a component of the transformer model 800 may be accomplished by a different component included in the transformer model 800 or a different model or module.

[0096] The encoder block 810 receives input sequences and generates matrix representations of the input sequences. In the embodiments of FIG. 8, the encoder block810 receives an input 801 and generates an encoder output 802. The input 801 may be an input prompt. In some embodiments, the input 801 may include one or more input tokens, such as words, phrases, sentences, images, audio signals, other types of input tokens, or some combination thereof. In an example, the input 801 may include a prompt received from a user of the transformer model 800. The prompt may include a question or request made by the user. A word in the prompt may be an input token. The encoder output 802 may include one or more vectors that are contextualized representations of the input 801. Each vector in the encoder output 802 may represent a token in the input 801 with contextual understanding.

[0097] The encoder block 810 includes an embedding layer 813, a positional encoding layer 815, and a plurality of layers 840 (individually referred to as "layer 840"). In other embodiments, the encoder block 810 may have different, fewer, or more components. Also, the arrangement of the components in the encoder block 810 may be different from the arrangement shown in FIG. 8. For the purpose of illustration, the encoder block 810 has N layers in FIG. 8, where N is an integer. Each layer 840 may include one or more neural network operations. The layers 840 may transform a sequence of embeddings into a representation that encapsulates the learned information from the input 801. Different layers 840 may have different internal parameters, e.g., different weights, bias, or other types of internal parameters. In some embodiments, the layers 840 have identical components. The components in a layer 840 may be layers and may also be referred to as sub-layers of the layer 840. As shown in FIG. 8, a layer 840 includes four sub-layers: a multihead attention (MHA) layer 841, an add & norm layer 842, a feed forward layer 843, and another add & norm layer 844.

[0098] The decoder block 820 iteratively generates outputs 803 using encoded representations generated by the encoder block 810. The decoder block 820 includes an embedding layer 823, a positional encoding layer 825, and a plurality of layers 850 (individually referred to as "layer 850"). For the purpose of illustration, the decoder block 820 has N layers in FIG. 8, where N is an integer. In the embodiments of FIG. 2, the number of layers 850 in the decoder block 820 is the same as the number of layers 840 in the encoder block 810. In other embodiments, the number of layers 850 in the decoder block 820 may be different from the number of layers 840 in the encoder block 810. Each layer 850 may include one or more neural network operations. Different layers 850 may havedifferent internal parameters. In some embodiments, the layers 850 may have identical components. The components in a layer 850 may be layers and may also be referred to as sub-layers of the layer 850. As shown in FIG. 8, a layer 850 includes six sub-layers: an MH A layer 851, an add & norm layer 852, another MHA layer 853, another add & norm layer 854, a feed forward layer 855, and another add & norm layer 856.

[0099] In some embodiments, a sequence of inference stages is performed in the decoder block 820 using encoder outputs, e.g., the encoder output 802. A matrix may be predicted through each inference stage. The outputs 803 may include a plurality of matrices. Each matrix may be further processed in the head block 830 to predict a token. The plurality of matrices may be used to predict a sequence of tokens. For the first inference stage, the decoder block 820 may receive one or more start tokens as input tokens and compute a first matrix from the input tokens and the output of the encoder block 810. The first matrix may be used by the head block 830 to predict a first token. The predicted token may be used as a new input token, in addition to the start token(s), in the second inference stage. Similarly, a second token may be predicted through the second inference stage and may be used in the third inference stage. This iteration may continue till all the inference stages are complete.

[0100] The head block 830 receives the output of the decoder block 820 and processes it in a linear layer 833 and a Softmax layer 835. A linear operation may be performed on the output of the decoder block 820 in the linear layer 833. The linear operation may include a multiplication of the output of the decoder block 820 with a weight matrix. The output of the linear layer 833 may be a vector. In some embodiments, the head block 830 may function as a classifier. The number of data elements in the vector computed in the linear layer 833 may depend on the number of classes involved. In an example where there are M classes, where M is an integer, the vector computed in the linear layer 833 may have M data elements representing the prediction for the M classes, respectively.

[0101] The output of the linear layer 833 may be input into the Softmax layer 835. A Softmax function may be applied on the output of the linear layer 833 to compute probability scores. A probability score may have a value in the range from 0 to 8. In some embodiments, a probability value is computed for each data element in the vector computed in the linear layer 833. The highest one of the probability scores may be the key. The corresponding index of the key may point to the token that the transformer model 800 predicts as the next in the sequence. The final output of the transformer model 800 may bethe sequence of predicted tokens. In some embodiments, the head block 830 may be a language modeling head.

[0102] An embedding layer (e.g., the embedding layer 813 or the embedding layer 823) converts an input of the embedding layer (e.g., the input 801 or the outputs 803) into one or more embeddings. An embedding may be a vector, which is also referred to as an embedding vector or a vector embedding. The vector embedding may include a sequence of data elements. In some embodiments, the embedding layer 813 may generate a plurality of embeddings, each of which may be converted from a different input token in the input 801. The embeddings may capture the semantic meaning of the tokens in the input 801. The embeddings may be numerical representations that capture the relationships or meanings of words, phrases, or other data types. In an example where the input 801 is a prompt including a sequence of words, the embedding layer 813 may generate an embedding from each word in the input 801. The embedding layer 823 in the decoder block 820 may generate a plurality of embeddings from tokens received by the decoder block 820 in a similar manner as the embedding layer 813.

[0103] A positional encoding layer (e.g., the positional encoding layer 815 or the positional encoding layer 825) performs positional encoding on embeddings generated in the corresponding embedding layer. In some embodiments, the positional encoding layer may apply one or more positional encoding vectors (e.g., a positional encoding vector 804 or positional encoding vector 805) on vector embeddings from the corresponding embedding layer to generate new vector embeddings that represent the embeddings with positional context. The positional encoding vector may encode information about the position of the embedding in a sequence of embeddings. In some embodiments, the positional encoding layer performs an addition operation on a positional encoding vector and a vector embedding. The addition operation may be elementwise addition. The positional encoding layer may output an embedding matrix that includes the vector embeddings computed in the positional encoding layer.

[0104] An MHA layer (e.g., the MHA layer 841, the MHA layer 851, or the MHA layer 853) may implement a multi-head attention mechanism, which may be a multi-head selfattention mechanism or a multi-head cross-attention mechanism. In some embodiments, the MHA layer 841 or the MHA layer 851 may implement a self-attention mechanism. For self-attention, the queries, keys, and values may come from the same place. For instance,for the MHA layer 841, the queries, keys, and values may all come from the positional encoding layer 815. For the MHA layer 851, the queries, keys, and values may all come from the positional encoding layer 825. The self-attention mechanism may enable the transformer model 800 to relate each token with other tokens. The MHA layer may compute attention scores from embeddings generated in the corresponding positional encoding layer. In some embodiments, the MHA layer may receive one or more queries, one or more keys, and one or more values. In some embodiments, the MHA layer has a number of heads that receive different linearly projected versions of the queries, keys, and values and produce outputs in parallel that are then used to generate the final result.

[0105] In some embodiments, the queries, keys, and values input into the MHA layer 841 may be computed from vector embeddings generated by the positional encoding layer 815. The queries, keys, and values input into the MHA layer 851 may be computed from vector embeddings generated by the positional encoding layer 825. A query, key, or value may be a vector the represents a token in a sequence. In some embodiments, a query matrix Q G IRWx / lmay be computed by multiply an embedding matrix X G IRWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WqG IRdx / l, where d is the dimension of a vector embedding, N is the number of vector embeddings in the embedding matrix, and h is the number of attention heads. Each row in the query matrix may be a query. A key matrix K G IRWx / lmay be computed by multiple an embedding matrix X G IRWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WkG IRdx / l. Each row in the key matrix may be a key. A value matrix V G IRWx / lmay be computed by multiple an embedding matrix X G IRWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WvE IRdx / l. Each row in the value matrix may be a value.

[0106] In some embodiments, the MHA layer 851 may implement masked multi-head selfattention. The MHA layer 851 may prevent positions from attending to subsequent positions. For instance, each token in the sequence may not be influenced by future tokens. This masking can ensure that the predictions of a particular position can depend on known outputs at positions before it and not depend on unknown outputs at positions after it.

[0107] In some embodiments, the MHA layer 853 may implement a cross-attention mechanism, such as encoder-decoder cross-attention. The MHA layer 853 may use outputsfrom the previous layer (i.e., the add & norm layer 852) as queries and use outputs from the encoder block 810 as keys and values. The cross-attention can align the encoder's input with the decoder's, empowering the decoder block 820 to identify and emphasize the most relevant parts of the encoder's input.

[0108] In some embodiments, an MHA layer includes linear layers, a MatMul layer, a scale layer, a Softmax layer, another MatMul layer, a concatenation layer, and another linear layer. These layers may be arranged in a sequence. The MHA layer may receive three input matrices: a query matrix, a key matrix, and a value matrix, which are inputs of three linear layers, respectively. The linear layers may include matrix multiplication (MatMul) operations. For instance, a first linear layer may perform a multiplication of the query matrix with a weight matrix to compute a first parameter matrix. The first parameter matrix may be denoted as QW , where Q is the query matrix and W G Jdmodei*dqjs t^e wejghtmatrix. A second linear layer may perform a multiplication of the key matrix with a weight matrix to compute a second parameter matrix. The second parameter matrix may be denoted as KW , where K is the key matrix and W G ^-modei^k jsthe weight matrix. A third linear layer may perform a multiplication of the value matrix with a weight matrix to compute a third parameter matrix. The third parameter matrix may be denoted as VW-', where V is the value matrix and WtvG j^dmodei><dk jsthe weight matrix, i may indicate the index of the head. dqis the dimension of a query vector. dkis the dimension of a key vector. dvis the dimension of a value vector. In some embodiments, dq= dk= dv= dmodei / h. In some embodiments, the linear layers may be in a linear block of the MHA layer. In some embodiments, the MHA layer may include multiple linear blocks. For instance, the MHA layer includes h linear blocks. The linear blocks may have the same layers as each other.Each linear block may compute three parameter matrices from the query matrix, key matrix, and value matrix, respectively.

[0109] The MatMul layer, scale layer, mask layer, Softmax layer, and MatMul layer may be in an attention block of the MHA layer. The attention block may implement a scaled dotproduct attention mechanism. In some embodiments, the MHA layer includes a plurality of attention blocks that includes the attention block. For the purpose of illustration, the MHA layer includes h attention blocks. The attention blocks may have the same layers as each other. A linear block and an attention block may constitute a head of the MHA layer. Whenthe MHA layer has h linear blocks and h attention blocks, the MHA layer has h heads. A head may be denoted as headt= Attention QW , KW , VW-').

[0110] A matrix multiplication operation may be performed on parameter matrices in the MatMul layer, which computes a score matrix. In some embodiments, the score matrix may establish the degree of emphasis each token should place on other tokens. The score matrix may include a plurality of scores. Each token may be assigned a score in relation to other tokens within the same time step. A higher score may indicate a higher focus or emphasis. The score matrix may be scaled in the scale layer. In some embodiments, the score matrix is scaled down in the scale layer by dividing the scores in the score matrix by the square root of the dimension of the query vector and the key vector, which may be denoted asThe output of the scale layer may be a scaled matrix, which includes adjusted scores. The mask layer may be optional in some embodiments. The mask layer may add an attention mask (which may be an input to the attention block) to the output of the scale layer to mask out some elements in the output of the scale layer. The positions of the masked-out elements may be defined by the attention mask. A Softmax function may be applied on the scaled matrix in the Softmax layer to compute an attention weight matrix. The attention weight matrix includes attention weights. The attention weights may be probability values ranging from 0 to 1. The Softmax function may emphasize high scores while diminishing low scores, which can enhance the model's ability to determine which tokens should get more attention.

[0111] In the MatMul layer, a matrix multiplication operation is performed on the attention weight matrix computed in the Softmax layer and the parameter matrix computed from value matrix in the corresponding linear layer. The result of the matrix multiplication operation is a single-head output matrix, which is an output of the attention block.

[0112] When the MHA layer has h attention blocks, there may be h single-head output matrices. The single-head output matrices are concatenated in the concatenation layer to form a concatenated matrix. A linear operation (also referred to as "linear transformation") is performed on the concatenated matrix using a weight matrix in the linear layer. In some embodiments, the MHA may be denoted as MultiHead Q, K, 7) = Concat (head- , head2, ... , headh~)W° , where Concat denotes concatenation, and W° G j^ / idpxdmode;js t e wejght matrix in the corresponding linear layer.

[0113] An add & norm layer in the transformer model 800, such as the add & norm layer 842, 844, 852, 854, and 856, has an addition operation followed by a layer normalization operation. The addition operation may be an addition of the output of the preceding layer and the input of the preceding layer. The preceding layer is a layer that is arranged right before the add & norm layer. For example, the preceding layer of the add & norm layer 842 is the MHA layer 841. As another example, the preceding layer of the add & norm layer 854 is the MHA layer 853.

[0114] Then the layer normalization operation is applied on the result of the addition operation, which may be denoted as LayerNorm(x + sub lay er (x)), where LayerNorm denotes layer normalization, x is the input of the preceding layer, and sub lay er (x denotes the output of the preceding layer. In some embodiments, the layer normalization operation may include a sequence of computations. In an example, the layer normalization operation may include a mean computation, which may be denoted as y.xy= - X z=i ^xyz, where Axyzdenotes a data element in the input tensor, x may be the positional index of the data element in one of the spatial dimensions, y may be the positional index of the data element in the other one of the spatial dimensions, z may be the positional index of the data element in the channel dimension, and ixydenotes the output of the mean computation, which may be a 2D matrix. The mean computation may be channel-wise reduction operation. The layer normalization operation may convert ixyto a 3D tensor [ixyz, e.g., by replicating every data element over z output points.

[0115] The layer normalization operation may also include an elementwise subtraction, which may be denoted as Dxyz= Axyz— .xyz. The layer normalization operation may further include a variance computation denoted as <j2%y= z=iD2xyzand a division computation denoted as Mxy= . ^=. Mxymay be a 2D tensor. The layer x<AzXy+exZ) normalization operation may also convert Mxyto a 3D tensor Mxyz, e.g., by replicating every data element over z output points. Further, the layer normalization operation may have an element multiplication denoted as A1xyz= X1. = = Dxyzx Mxyz. The layer normalization operation may further compute ljx(ff2xy+e)A"Xyz = A'xyz+ and LNxyz= A"xyzX yz. LNxyzmay be the output of the layernormalization operation.

[0116] A feed forward layer (e.g., the feed forward layer 843 and the feed forward layer 855) may be a position-wise fully-connected feed forward network. In an example, the feed forward layer may include two linear layers with an activation function in between. An example of the activation function is Rectified Linear Unit (ReLU).

[0117] FIG. 9A and 9B illustrate inferences of a transformer model 900, in accordance with various embodiments. FIG. 9A illustrates the first inference process of the transformer model 900, in accordance with various embodiments. The transformer model 900 includes an encoder 910, a decoder 920, and a head 930. An example of the transformer model 900 may be the transformer model 800 in FIG. 8. In the embodiments of FIG. 9A, the encoder 910 receives an input tensor 901. The input tensor 901 may be a feature map extracted from one or more images, text documents, audio files, videos, other types of data, or some combination thereof. The encoder 910 generates an output tensor 902 from the input tensor 901. The shape of the output tensor 902 may be denoted as[batch size,SLencoder, dmodei\, where SLencodermay be the dimension along the X axis (i.e., the width of the output tensor 902), and dmodeimay be the dimension along the Y axis (i.e., the height of the output tensor 902). The encoder 910 may include a plurality of layers arranged in a sequence, such as the layers inside the encoder block 810 in FIG. 8. The output tensor 902 is provided to the decoder 920.

[0118] The decoder 920 receives the output tensor 902 and an input sequence 903. The input sequence 903 may be a sequence of tokens. A token may be a numerical representation of an input signal, such as word, image, audio signal, video signal, etc. The dimension of the input sequence 903, which may be denoted as SLinput, may be the total number of tokens in the input sequence 903. For the purpose of illustration and simplicity, SLinputis 4. In other embodiments, the input sequence 903 may have a different shape. For instance, the input sequence 903 may be a 2D tensor. The dimension of the 2D tensor along the X axis may be SLinput, while the dimension of the 2D tensor along the Y axis may be a batch size indicating the number of batches in the input sequence 903.

[0119] The decoder 920 computes an output tensor 904, a self-attention key tensor 905, a self-attention value tensor 906, a cross-attention key tensor 907, and a cross-attentionvalue tensor 908. In some embodiments, the shape of the output tensor 904 may be denoted as [batch size,SLinput, dmodei]. The shape of the self-attention key tensor 905 or the shape of the self-attention value tensor 906 may be denoted as N X[batch size, h, SLinput, dhead], where N is the number of identical layers in the decoder (e.g., the number of layers 850 in the decoder block 820), h is the total number of heads in a MHA layer, and dheadis the dimension of a query vector, key vector, or value vector. In some embodiments, dmodei= h x dhead. The shape of the cross-attention key tensor 907 or the shape of the cross-attention value tensor 908 may be denoted as N x [batch size, h, S Lencoder, d / ieadJ.

[0120] The output tensor 904 may be provided to the head 930 and the head 930 outputs a predicted token 909. The shape of the token 909 may be denoted as [batch size, 1], For the purpose of illustration and simplicity, batch size is 1 in FIG. 9A. In other embodiments, batch size may be a larger number. The predicted token 909 may be stored in a buffer. In some embodiments, the predicted token 909 may be used to update the input sequence 903. For instance, the predicted token 909 may be added to the right of the input sequence 903. The updated input sequence may be used as the input sequence in the second inference phase. In the second inference phase, the decoder 920 may receive the updated input sequence and the output tensor 902 for predicting another token. The output tensor902 may remain the same during inference of the decoder 920. Certain aspects of subsequent inference processes are described below in conjunction with FIG. 9B.

[0121] In some embodiments, the self-attention key tensor 905 and the self-attention value tensor 906 may be provided to a self-attention layer in the decoder 920, an example of such a self-attention layer is the MHA layer 151. The self-attention key tensor 905 may be stored in a self-attention key cache. The self-attention key cache may have the same shape as the self-attention key tensor 905. The self-attention value tensor 906 may be stored in a selfattention value cache. The self-attention value cache may have the same shape as the selfattention value tensor 906.

[0122] In some embodiments, the decoder 920 computes the self-attention key tensor 905 and the self-attention value tensor 906 from the input sequence 903. The input sequence903 may be dynamic during inference of the decoder 920. For instance, a new token may be added to the input sequence 903 after each inference phase, as described above. As the input sequence 903 changes, the self-attention key tensor 905 and the self-attention valuetensor 906 would also change. For instance, the dimension of the self-attention key tensor 905 or the self-attention value tensor 906 along the X axis may increase as SLinputincreases. The self-attention key cache and the self-attention value cache may change during all the inference phases of the decoder 920 to accommodate the changes in the selfattention key tensor 905 and the self-attention value tensor 906.

[0123] In some embodiments, the cross-attention key tensor 907 and the cross-attention value tensor 906 may be provided to a cross-attention layer in the decoder 920, an example of such a cross-attention layer is the MHA layer 153. The cross-attention key tensor 907 may be stored in a cross-attention key cache. The cross-attention key cache may have the same shape as the cross-attention key tensor 907. The cross-attention value tensor 908 may be stored in a cross-attention value cache. The cross-attention value cache may have the same shape as the cross-attention value tensor 908. In some embodiments, the decoder 920 computes the cross-attention key tensor 907 and the cross-attention value tensor 906 from the output tensor 902 generated in the encoder 910. As the output tensor 902 does not change during inference of the decoder 920, the cross-attention key tensor 907 and the cross-attention value tensor 906 may remain the same during all the inference phases of the decoder 920. The cross-attention key cache and the cross-attention value cache may remain the same during all the inference phases of the decoder 920.

[0124] FIG. 9B illustrates subsequent inference processes of the transformer model 900, in accordance with various embodiments. In the second inference phase, the decoder 920 may reuse the self-attention key tensor 905, self-attention value tensor 906, cross-attention key tensor 907, and cross-attention value tensor 908. The decoder 920 also receives the predicted token 909. The decoder 920 may compute self-attention key vectors from the predicted token 909 and concatenate the self-attention key vectors with the self-attention key tensor 905 to generate a new self-attention key tensor 915. For instance, a selfattention key vector for each head may be added to the right of a self-attention key matrix in the self-attention key tensor 905, and the self-attention key vector and the self-attention key matrix may correspond to the same head. The elements highlighted with a dot pattern in the self-attention key tensor 915 are the self-attention key vectors generated from the predicted token 909.

[0125] Similarly, the decoder 920 may compute self-attention value vectors from the predicted token 909 and concatenate the self-attention value vectors with the self-attentionvalue tensor 906 to generate a new self-attention value tensor 916. For instance, a selfattention value vector for each head may be added to the right of a self-attention value matrix in the self-attention value tensor 906, and the self-attention value vector and the self-attention value matrix may correspond to the same head. The elements highlighted with a dot pattern in the self-attention value tensor 916 are the self-attention value vectors generated from the predicted token 909.

[0126] The decoder 920 also generates an output tensor 914. The decoder 920 may generate the output tensor 914 using the new self-attention key tensor 915 and new selfattention value tensor 916. The output tensor 914 is used by the head 930 to generate another predicted token 919. The predicted token 919 is the output of the transformer model 900 in the second inference phase.

[0127] One or more other subsequent inference processes may be conducted. In each subsequent inference phase, the decoder 920 receives a token predicted in the previous inference phase, a self-attention key tensor generated in the previous inference phase, a self-attention value tensor generated in the previous inference phase, the cross-attention key tensor 907, and the cross-attention value tensor 908. The decoder 920 may, in the subsequent inference phase, generate a larger self-attention key tensor and a larger selfattention value tensor, in addition to an output tensor which can be used by the head 930 to predict a new token.

[0128] In embodiments where the total number of inference phases is N, the input sequence 903 is updated to an input sequence 913 after N — 1 inference phases. In the last inference phase (i.e., the Nth inference phase), the decoder 920 may receive the predicted token generated in the (N — l)th inference phase, the self-attention key tensor generated in the (N — l)th inference phase, the self-attention value tensor generated in the (N — l)th inference phase, the cross-attention key tensor 907, and the cross-attention value tensor 908. The decoder 920 may generate a self-attention key tensor 925 and a selfattention value tensor 926 using the predicted token generated in the (N — l)th inference phase, the self-attention key tensor generated in the (N — l)th inference phase, and the self-attention value tensor generated in the (N — l)th inference phase. The dimensions of the self-attention key tensor 925 or self-attention value tensor 926 along the X axis is SLinput+ N. The decoder 920 also generates an output tensor 924, which is used by the head 930 to generate the last predicted token 929. The N tokens predicted by thetransformer model in the N inference phases may constitute an output tensor 939, which may be the final output of the transformer model.

[0129] FIG. 10 is a block diagram of a NPU 1000, in accordance with various embodiments. The NPU 1000 can execute DNNs, including transformer models described above. For instance, the NPU 1000 can execute layers in a DNN by carrying out neural network operations in the layers. The layers may be arranged in a sequence, and the NPU 1000 may execute the layers in the sequence. The execution of the DNN may be for training the DNN or for using the DNN to perform Al tasks. As shown in FIG. 10, the NPU 1000 includes a memory 1010, a DMA engine 1020, and compute blocks 1030 (individually referred to as "compute block 1030"). In other embodiments, alternative configurations, different or additional components may be included in the NPU 1000. For example, the NPU 1000 may include more than one memory 1010 or DMA engine 1020. As another example, the NPU 1000 may include a single compute block 1030. Further, functionality attributed to a component of the NPU 1000 may be accomplished by a different component included in the NPU 1000 or by a different system. A component of the NPU 1000 may be implemented in hardware, software, firmware, or some combination thereof.

[0130] The memory 1010 stores data associated with neural network operations performed by the NPU 1000. In some embodiments, the memory 1010 may store data to be used by the compute blocks 1030 for executing neural network operations. The memory 1010 may store inputs to DNNs and outputs of DNNs. The memory 1010 may also store activations (such as input activations and output activations of neural network operations) and weights (such as weights determined by training DNNs) in DNNs. In some embodiments, the memory 1010 may store activations and weights with floating-point precisions, such as FP4, SF4, NF4, FP16, BP16, FP32 and so on. The memory 1010 may also quantized activations or weights. The memory 1010 includes one or more dynamic random-access memories (DRAMs).

[0131] The DMA engine 1020 facilitates data transfer between the memory 1010 and the compute blocks 1030. For example, the DMA engine 1020 can read data from the memory 1010 and write data into a local memory of a compute block 1030. As another example, the DMA engine 1020 can read data from a local memory of a compute block 1030 and write data into the memory 1010. For instance, the DMA engine 1020 may read input activations and weights of convolution from the memory 1010 and load the input activations andweights to one or more compute blocks 1030. The DMA engine 1020 may also write output activations of convolutions computed by one or more compute blocks 1030 to the memory 1010. The DMA engine 1020 provides a DMA feature that allows the compute block 1030 to initiate data transfer between the memory 1010 and the local memories of the compute blocks 1030 and to perform other operations while the data transfer is being conducted. In some embodiments, the DMA engine 1020 may read tensors from the memory 1010, modify the tensors in a way that is optimized for the compute block 1030 before it writes the tensors into the local memories of the compute blocks 1030.

[0132] The compute blocks 1030 perform neural network operations in DNNs. For instance, a compute block 1030 may execute a DNN layer by running one or more deep learning operations in the DNN layer. A compute block 1030 may execute a layer, or a portion of a layer, at a time. In some embodiments, the operations of the DNN layers may be run by multiple compute blocks 1030 in parallel. For instance, multiple compute blocks 1030 may each perform a portion of a workload for a neural network operation. Data may be shared between the compute blocks 1030. A compute block 1030 may also be referred to as a compute tile. The compute blocks 1030 may be capable of running various types of neural network operations, such as convolution, matrix multiplication, Softmax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on. Neural network operations performed by the compute blocks 1030 include tensor operations, i.e., operations whose inputs are tensors or operations whose outputs are tensors. In an example, the compute block 1030 receives an input tensor and one or more convolutional kernels and performs a convolution with the input tensor and convolutional kernels. The result of the convolution may be an output tensor, which can be further computed, e.g., by the compute block 1030 or another compute block 1030.

[0133] In the embodiments of FIG. 10, each compute block 1030 includes a local memory 1040, a digital signal processor (DSP) 1050, and a data processing unit (DPU) 1055. The DPU 1055 includes an input delivery unit (IDU) 1060, a processing engine 1070, a post-processing engine 1080, and an output delivery unit (ODU) 1090. Some or all the components of the compute block 1030 can be implemented on the same chip. In other embodiments, alternative configurations, different or additional components may be included in the compute block 1030. Further, functionality attributed to a component of the compute block 1030 may be accomplished by a different component included in the compute block 1030, adifferent compute block 1030, another component of the NPU 1000, or a different system. A component of the compute block 1030 may be implemented in hardware, software, firmware, or some combination thereof.

[0134] The local memory 1040 is local to the corresponding compute block 1030. The local memory 1040 is accessible to both the DSP 1050 and DPU 1055. In the embodiments of FIG. 10, the local memory 1040 is inside the compute block 1030. In other embodiments, the local memory 1040 may be outside the compute block 1030. Data in the local memory 1040 may be transferred to or from the memory 1010, e.g., through the DMA engine 1020. In some embodiments, data in the local memory 1040 may be transferred to or from the local memory of another compute block 1030. The local memory 1040 may store data received, used, or generated by the IDU 1060, the processing engine 1070, the post-processing engine 1080, or the ODU 1090. Examples of the data may include input activations, weights, output activations, configuration parameters, and so on.

[0135] In some embodiments, the local memory 1040 includes one or more static randomaccess memories (SRAMs). The local memory 1040 may be byte-addressable, and each memory address identifies a single byte (eight bits) of storage. In some embodiments, the local memory 1040 may include memory banks. The number of data banks in the local memory 1040 may be 16, 64, 128, 1056, 512, 1024, 2048, or other numbers. A memory bank may include a plurality of storage units. In an example, a data bank may include 8, 16, 64, or a different number of storage units. A memory bank or a storage unit in a memory bank may have a memory address. In an example, a storage unit may store a single byte, and data larger than a single byte may be stored in storage units with consecutive memory addresses, i.e., adjacent storage units. For instance, a storage unit can store an integer number in the INT8 format, versus two storage units may be needed to store a number in the FP16 or BF16 format, which has 16 bits. In some embodiments, 16 bits can be transferred from the local memory 1040 in a single read cycle. In other embodiments, 16 bits can be transferred from the local memory 1040 in multiple read cycles, such as two cycles.

[0136] The DSP 1050 performs computations in DNN layers, including computations in group quantization-based neural network operations. In some embodiments, the DSP 1050 can perform generic computations such as addition, subtraction, multiplication, division, logical, bitwise operations, and other nonlinear computations (in terms of table look-up orpolynomial approximation). The DSP 1050 may be a very long instruction word (VLIW) processor. In some embodiments, the DSP 1050 may have an architecture optimized for the operational needs of digital signal processing. In some embodiments, the DSP 1050 may perform some computations in a neural network operation, while other computations in the neural network operation may be performed by the DPU 1055. The DSP 1050 may support non-traditional operations or non-MatMul or non-convolution-based operations within DNNs.

[0137] In some embodiments, the DSP 1050 may operate in accordance with a clock signal. For instance, the timing when the DSP 1050 can execute instructions may be synchronized with the clock signal. In some embodiments, the DSP 1050 may be pipelined along with the DMA engine 1020 or the DPU 1055, thereby enabling parallel computations to improve overall performance. The DSP 1050 may be implemented on a microprocessor chip, which may be separate from a chip implementing the DPU 1055. In some embodiments, the DSP 1050 may be a Streaming Hybrid Architecture Vector Engine (SHAVE) processor. Even though FIG. 10 shows a single DSP, the compute block 1030 may include multiple DSPs. The DSPs may be arranged in an array.

[0138] The IDU 1060 loads data from the local memory 1040 to the processing engine 1070 or to the post-processing engine 1080. The IDU 1060 may read tensors from the local memory 1040. The tensors may include activation tensors, weights tensor, and so on. The IDU 1060 may perform group-wise loading of activations or weights. In some embodiments, the IDU 1060 may read data from the local memory 1040 and write the data into storage units in the processing engine 1070. For instance, the IDU 1060 may load activations into activation register files in the processing engine 1070 and load weights into weight register files in the processing engine 1070. The IDU 1060 may have an activation reader for loading activations and a weight reader for loading weights. In some embodiments, the IDU 1060 may read configuration parameters from the local memory 1040 and load the configuration parameters into configuration registers or other configurable components (e.g., LUTs) of the processing engine 1070 or post-processing engine 1080.

[0139] The processing engine 1070 performs operations in DNNs. The processing engine 1070 may include one or more processing cells. In some embodiments, the processing cells may be arranged in one or more rows and one or more columns in the processing engine 1070. Each processing cell may include processing elements (PEs) that may be arranged inan array that includes rows and columns. All the PEs in the processing engine 1070 may constitute a bigger array that includes more rows and columns. An example PE may be or may include one or more multiply-accumulate (MAC) units that can perform MAC operations. In some embodiments (e.g., embodiments where the compute block 1030 executes a convolutional layer), a computation in an MAC unit may be an MAC operation on an activation operand and a weight operand. The activation operand may be an activation tensor that may include one or more activations in the input tensor of the convolution. Different activations may be in different input channels. The weight operand may be a weight tensor that may include one or more weights in the filter of the convolution. The values of the weights are determined through training the DNN or compressing the neural network operation after training. The weights in the weight operand may be in different input channels. In some embodiments, the activation operand or weight operand is a vector along the input channel dimension.

[0140] In some embodiments, an MAC unit includes one or more multipliers for performing multiplications. An MAC unit may also include one or more accumulators ("adders") for performing accumulations. An MAC unit may also include one or more shifters to facilitate mixed-precision computations. A column of MAC units is referred to as an MAC column. An MAC column may be associated with one or more MAC lanes. A MAC lane is a path for loading data e.g., by the IDU 1060, into an MAC column. A MAC lane may be also referred to as a data transmission lane or data loading lane. An MAC column may have multiple MAC lanes. The loading bandwidth of the MAC column is an aggregation of the loading bandwidths of all the MAC lanes associated with the MAC column. With a certain number of MAC lanes, data can be fed into the same number of independent MAC units simultaneously. In some embodiments where an MAC column has four MAC lanes for feeding activations or weights into the MAC column and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes can have a total loading bandwidth of 64 bytes.

[0141] In some embodiments, a processing cell may have a sparsity logic unit for accelerating computations in DNNs based on data sparsity. For instance, the sparsity logic unit may obtain or generate a sparsity bitmap and use the sparsity bitmap to identify nonzero values in the activation register files or weight registers files and send nonzero values to the PEs for performing computation, while zero values in the activation register files or weight registers files are skipped.

[0142] The post-processing engine 1080 processes outputs of the processing engine 1070. The post-processing engine 1080 may include one or more post-processing elements (PPEs). In some embodiments, the PPEs in the post-processing engine 1080 may be arranged in an array that has rows and columns. In some embodiments, the post-processing engine 1080 computes activation functions. The post-processing engine 1080 may receive outputs of the processing engine 1070 as inputs to the activation functions. In addition or alternative to activation functions, the post-processing engine 1080 may perform other types of post processing on outputs of the processing engine 1070. For instance, the post-processing engine 1080 may apply a bias on an output of the processing engine 1070. In some embodiments, the post-processing engine 1080 may be bypassed for certain neural network operations.

[0143] The ODU 1090 drains data from the processing engine 1070 or from the postprocessing engine 1080, e.g., from register files in the processing engine 1070 or from the post-processing engine 1080. The drain module may write the data to the local memory 1040. The drained data may be tensors, such as output tensors of neural network operations. In some embodiments, the ODU 1090 may drain data on a cell level. For each processing cell, the ODU 1090 may drain outputs of PEs in the processing cell based on a row index or column index of each PE. For instance, the ODU 1090 may use a sequence of cycles to drain data from a processing cell. The ODU 1090 may drain the output of some of the PEs in each cycle. The sequence of the cycles may be configured based on a configuration parameter indicating the operation mode of the IDU 1060.

[0144] In some embodiments, the ODU 1090 includes sparsity encoding logic that can convert outputs of the processing engine 1070 from a dense format to a sparse format. For instance, the ODU 1090 may be implemented with one or more sparsity encoders. A sparsity encoder converts dense data to compressed data based on sparsity in the dense data. For instance, the sparsity encoder may remove zeros from data computed by the processing engine 1070. The sparsity encoder may also generate sparsity maps that represent sparsity in the dense data.

[0145] In some embodiments, the data drained from the processing engine 1070 may be output data elements of a DNN layer. The sparsity encoder may generate a compressed version of the output tensor. The sparsity encoder may identify every zero activation in the output tensor and remove these activations from the output tensor to generate acompressed activation tensor (aka "sparse activation tensor"). The sparsity encoder may also generate one or more sparsity maps for the output tensor. A sparsity map may indicate sparsity in at least part of the output tensor. The sparsity map may include sparsity elements (e.g., bits), each of which corresponds to a different activation in the vector and indicates whether the corresponding activation is zeroed or not.

[0146] The ODU 1090 may write the compressed activation tensor and the one or more sparsity maps into the local memory 1040. The sparse activation tensor and the one or more sparsity maps may be further loaded to the memory 1010, e.g., through the DMA engine 1020. Additionally or alternatively, the sparse activation tensor and the one or more sparsity maps may be loaded by the IDU 1060 to the processing engine 1070 for further computation, e.g., for performing a deep learning operation in the next layer.

[0147] FIG. 11 is a flowchart of a method 1100 of accelerating transformer inference, in accordance with various embodiments. The method 1100 may be performed by the prescient paraclete module 210 in FIG. 2. Although the method 1100 is described with reference to the flowchart illustrated in FIG. 11, many other methods for accelerating transformer inference may alternatively be used. For example, the order of execution of the steps in FIG. 11 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

[0148] The prescient paraclete module 210 instructs 1110 a first thread to perform a first inference process of a transformer model. The transformer model includes a sequence of layer. In some embodiments, examples of the transformer model include the transformer model 100 in FIG. 1, the transformer model 800 in FIG. 8, and the transformer model 900 in FIGS. 9A and 9B. In some embodiments, the prescient paraclete module 210 reduces a data precision of input data of one or more layers of the transformer model. The prescient paraclete module 210 instructs the first thread to execute the one or more layers with the input data having the reduced data precision.

[0149] The prescient paraclete module 210 extracts 1120 one or more speculated tokens from the transformer model during the first inference process. The one or more speculated tokens are generated from a layer that is precedent to the final layer in the transformer model.

[0150] The prescient paraclete module 210 initiates 1130 one or more second inference processes of the transformer model using the one or more speculated tokens. The one ormore second inference processes are respectively performed by one or more additional threads. In some embodiments, the prescient paraclete module 210 selects the one or more speculated tokens from a plurality of tokens generated from the layer in the first inference process of the transformer model. In some embodiments, the prescient paraclete module 210 selects the one or more speculated tokens based on one or more confidence scores of the transformer model for the one or more speculated tokens and a threshold confidence value. For instance, the prescient paraclete module 210 determines that the one or more confidence scores is greater than the threshold value. In some embodiments, the one or more second inference processes are in parallel with a part of the first inference process.

[0151] The prescient paraclete module 210 detects 1140 whether any of the one or more speculated tokens matches a predicted token generated at a final layer of the sequence of layers during the first inference process. The predicted token is generated after the one or more second inference processes are initiated. In some embodiments, the prescient paraclete module 210 determines whether any speculated token represents the same word(s) as the predicted token.

[0152] In response to determining that a speculated token matches the predicted token, the prescient paraclete module 210 instructs 1150 an additional thread of the one or more additional threads to complete a second inference process of the transformer model using the speculated token. The additional thread corresponds to the speculated token. In some embodiments, in response to determining that the speculated token matches the predicted token, the prescient paraclete module 210 prevents the first thread from performing another inference process of the transformer model.

[0153] In some embodiments, the second inference process of the transformer model comprises executing the final layer. In other embodiments, the second inference process of the transformer model comprises executing the layer and bypassing one or more layers subsequent to the layer. The one or more layers include the final layer.

[0154] In some embodiments, in response to determining that none of the one or more speculated tokens matches the predicted token, the prescient paraclete module 210 instructs the first thread to perform another inference process of the transformer model using the predicted token. In some embodiments, response to determining that none of the one or more speculated tokens matches the predicted token, the prescient paracletemodule 210 instructs the one or more additional threads to terminate the one or more second inference processes of the transformer model.

[0155] FIG. 12 is a block diagram of an example computing device 1200, in accordance with various embodiments. In some embodiments, the computing device 1200 can be used as at least part of the Al system 200. A number of components are illustrated in FIG. 12 as included in the computing device 1200, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 1200 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 1200 may not include one or more of the components illustrated in FIG. 12, but the computing device 1200 may include interface circuitry for coupling to the one or more components. For example, the computing device 1200 may not include a display device 1206, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 1206 may be coupled. In another set of examples, the computing device 1200 may not include an audio input device 1218 or an audio output device 1208 but may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which an audio input device 1218 or audio output device 1208 may be coupled.

[0156] The computing device 1200 may include a processing device 1202 (e.g., one or more processing devices). The processing device 1202 processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. The computing device 1200 may include a memory 1204, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid state memory, and / or a hard drive. In some embodiments, the memory 1204 may include memory that shares a die with the processing device 1202. In some embodiments, the memory 1204 includes one or more non-transitory computer- readable media storing instructions executable to perform operations for accelerating transformer inference (e.g., the method 1100 described in conjunction with FIG. 11) or some operations performed by one or more components of the Al system 200 (e.g., theprescient paraclete module 210). The instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 1202.

[0157] In some embodiments, the computing device 1200 may include a communication chip 1212 (e.g., one or more communication chips). For example, the communication chip 1212 may be configured for managing wireless communications for the transfer of data to and from the computing device 1200. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

[0158] The communication chip 1212 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 1212 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E- HSPA), or LTE network. The communication chip 1212 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1212 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 1212 may operate in accordance with other wireless protocols in other embodiments. The computing device 1200 may include an antenna 1222 to facilitatewireless communications and / or to receive other wireless communications (such as AM or FM radio transmissions).

[0159] In some embodiments, the communication chip 1212 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, the communication chip 1212 may include multiple communication chips. For instance, a first communication chip 1212 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 1212 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication chip 1212 may be dedicated to wireless communications, and a second communication chip 1212 may be dedicated to wired communications.

[0160] The computing device 1200 may include battery / power circuitry 1214. The battery / power circuitry 1214 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1200 to an energy source separate from the computing device 1200 (e.g., AC line power).

[0161] The computing device 1200 may include a display device 1206 (or corresponding interface circuitry, as discussed above). The display device 1206 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.

[0162] The computing device 1200 may include an audio output device 1208 (or corresponding interface circuitry, as discussed above). The audio output device 1208 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0163] The computing device 1200 may include an audio input device 1218 (or corresponding interface circuitry, as discussed above). The audio input device 1218 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0164] The computing device 1200 may include a GPS device 1216 (or corresponding interface circuitry, as discussed above). The GPS device 1216 may be in communication witha satellite-based system and may receive a location of the computing device 1200, as known in the art.

[0165] The computing device 1200 may include another output device 1210 (or corresponding interface circuitry, as discussed above). Examples of the other output device 1210 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0166] The computing device 1200 may include another input device 1220 (or corresponding interface circuitry, as discussed above). Examples of the other input device 1220 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0167] The computing device 1200 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultramobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 1200 may be any other electronic device that processes data.

[0168] The following paragraphs provide various examples of the embodiments disclosed herein.

[0169] Example 1 provides a method including instructing a first thread to perform a first inference process of a transformer model, the transformer model including a sequence of layer; extracting one or more speculated tokens from the transformer model during the first inference process, the one or more speculated tokens generated at a layer that is precedent to the final layer in the transformer model; initiating one or more second inference processes of the transformer model using the one or more speculated tokens, the one or more second inference processes respectively performed by one or more additional threads; detecting whether any of the one or more speculated tokens matches a predicted token generated at a final layer of the sequence of layers during the first inference process,the predicted token generated after the one or more second inference processes are initiated; and in response to determining that a speculated token matches the predicted token, instructing an additional thread of the one or more additional threads to complete a second inference process of the transformer model using the speculated token, the additional thread corresponding to the speculated token.

[0170] Example 2 provides the method of example 1, further including in response to determining that the speculated token matches the predicted token, preventing the first thread from performing another inference process of the transformer model.

[0171] Example 3 provides the method of example 1 or 2, in which the second inference process of the transformer model includes executing the final layer.

[0172] Example 4 provides the method of example 1 or 2, in which the second inference process of the transformer model includes executing the layer and bypassing one or more layers subsequent to the layer, the one or more layers including the final layer.

[0173] Example 5 provides the method of any one of examples 1-4, further including in response to determining that none of the one or more speculated tokens matches the predicted token, instructing the first thread to perform another inference process of the transformer model using the predicted token.

[0174] Example 6 provides the method of any one of examples 1-5, further including in response to determining that none of the one or more speculated tokens matches the predicted token, instructing the one or more additional threads to terminate the one or more second inference processes of the transformer model.

[0175] Example 7 provides the method of any one of examples 1-6, further including selecting the one or more speculated tokens from a plurality of tokens generated from the layer in the first inference process of the transformer model.

[0176] Example 8 provides the method of example 7, in which selecting the one or more speculated tokens includes selecting the one or more speculated tokens based on one or more confidence scores of the transformer model for the one or more speculated tokens and a threshold value.

[0177] Example 9 provides the method of any one of examples 1-8, in which the one or more second inference processes are in parallel with a part of the first inference process.

[0178] Example 10 provides the method of any one of examples 1-9, in which instructing the first thread to perform the first inference process of the transformer model includesreducing a data precision of input data of one or more layers of the transformer model; and instructing the first thread to execute the one or more layers with the input data having the reduced data precision.

[0179] Example 11 provides the method of any one of examples 1-10, in which the first thread is on a different type of processing unit from the one or more additional threads.

[0180] Example 12 provides the method of example 11, in which the one or more additional threads are on a graphics processing unit or a neural processing unit.

[0181] Example 13 provides the method of example 11 or 12, in which the first thread is on a central processing unit or a neural processing unit.

[0182] Example 14 provides one or more non-transitory computer-readable media storing instructions executable to perform operations, the operations including instructing a first thread to perform a first inference process of a transformer model, the transformer model including a sequence of layer; extracting one or more speculated tokens from the transformer model during the first inference process, the one or more speculated tokens generated at a layer that is precedent to the final layer in the transformer model; initiating one or more second inference processes of the transformer model using the one or more speculated tokens, the one or more second inference processes respectively performed by one or more additional threads; detecting whether any of the one or more speculated tokens matches a predicted token generated at a final layer of the sequence of layers during the first inference process, the predicted token generated after the one or more second inference processes are initiated; and in response to determining that a speculated token matches the predicted token, instructing an additional thread of the one or more additional threads to complete a second inference process of the transformer model using the speculated token, the additional thread corresponding to the speculated token.

[0183] Example 15 provides the one or more non-transitory computer-readable media of example 14, in which the operations further include in response to determining that the speculated token matches the predicted token, preventing the first thread from performing another inference process of the transformer model.

[0184] Example 16 provides the one or more non-transitory computer-readable media of example 14 or 15, in which the second inference process of the transformer model includes executing the final layer.

[0185] Example 17 provides the one or more non-transitory computer-readable media of example 14 or 15, in which the second inference process of the transformer model includes executing the layer and bypassing one or more layers subsequent to the layer, the one or more layers including the final layer.

[0186] Example 18 provides the one or more non-transitory computer-readable media of any one of examples 14-17, in which the operations further include in response to determining that none of the one or more speculated tokens matches the predicted token, instructing the first thread to perform another inference process of the transformer model using the predicted token.

[0187] Example 19 provides the one or more non-transitory computer-readable media of any one of examples 14-18, in which the operations further include in response to determining that none of the one or more speculated tokens matches the predicted token, instructing the one or more additional threads to terminate the one or more second inference processes of the transformer model.

[0188] Example 20 provides the one or more non-transitory computer-readable media of any one of examples 14-19, in which the operations further include selecting the one or more speculated tokens from a plurality of tokens generated from the layer in the first inference process of the transformer model, in which selecting the one or more speculated tokens includes selecting the one or more speculated tokens based on one or more confidence scores of the transformer model for the one or more speculated tokens and a threshold value.

[0189] Example 21 provides the one or more non-transitory computer-readable media of any one of examples 14-20, in which the one or more second inference processes are in parallel with a part of the first inference process.

[0190] Example 22 provides the one or more non-transitory computer-readable media of any one of examples 14-21, in which instructing the thread to perform the first inference process of the transformer model includes reducing a data precision of input data of one or more layers of the transformer model; and instructing the thread to execute the one or more layers with the input data having the reduced data precision.

[0191] Example 23 provides the one or more non-transitory computer-readable media of any one of examples 14-22, in which the first thread is on a different type of processing unit from the one or more additional threads.

[0192] Example 24 provides the one or more non-transitory computer-readable media of example 23, in which the one or more additional threads are on a graphics processing unit or a neural processing unit, and the first thread is on a central processing unit or a neural processing unit.

[0193] Example 25 provides an apparatus including a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform the method of any one of examples 1-13.

[0194] The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art can recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

Claims1. A method comprising: instructing a first thread to perform a first inference process of a transformer model, the transformer model comprising a sequence of layer; extracting one or more speculated tokens from the transformer model during the first inference process, the one or more speculated tokens generated at a layer that is precedent to the final layer in the transformer model; initiating one or more second inference processes of the transformer model using the one or more speculated tokens, the one or more second inference processes respectively performed by one or more additional threads; detecting whether any of the one or more speculated tokens matches a predicted token generated at a final layer of the sequence of layers during the first inference process, the predicted token generated after the one or more second inference processes are initiated; and in response to determining that a speculated token matches the predicted token, instructing an additional thread of the one or more additional threads to complete a second inference process of the transformer model using the speculated token, the additional thread corresponding to the speculated token.

2. The method of claim 1, further comprising: in response to determining that the speculated token matches the predicted token, preventing the first thread from performing another inference process of the transformer model.

3. The method of claim 1 or 2, wherein the second inference process of the transformer model comprises executing the final layer.

4. The method of claim 1 or 2, wherein the second inference process of the transformer model comprises executing the layer and bypassing one or more layers subsequent to the layer, the one or more layers comprising the final layer.

5. The method of any one of claims 1-4, further comprising: in response to determining that none of the one or more speculated tokens matches the predicted token, instructing the first thread to perform another inference process of the transformer model using the predicted token.

6. The method of any one of claims 1-5, further comprising: in response to determining that none of the one or more speculated tokens matches the predicted token, instructing the one or more additional threads to terminate the one or more second inference processes of the transformer model.

7. The method of any one of claims 1-6, further comprising: selecting the one or more speculated tokens from a plurality of tokens generated from the layer in the first inference process of the transformer model.

8. The method of claim 7, wherein selecting the one or more speculated tokens comprises: selecting the one or more speculated tokens based on one or more confidence scores of the transformer model for the one or more speculated tokens and a threshold value.

9. The method of any one of claims 1-8, wherein the one or more second inference processes are in parallel with a part of the first inference process.

10. The method of any one of claims 1-9, wherein instructing the first thread to perform the first inference process of the transformer model comprises: reducing a data precision of input data of one or more layers of the transformer model; and instructing the first thread to execute the one or more layers with the input data having the reduced data precision.

11. The method of any one of claims 1-10, wherein the first thread is on a different type of processing unit from the one or more additional threads.

12. The method of claim 11, wherein the one or more additional threads are on a graphics processing unit or a neural processing unit.

13. The method of claim 11 or 12, wherein the first thread is on a central processing unit or a neural processing unit.

14. One or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising: instructing a first thread to perform a first inference process of a transformer model, the transformer model comprising a sequence of layer; extracting one or more speculated tokens from the transformer model during the first inference process, the one or more speculated tokens generated at a layer that is precedent to the final layer in the transformer model; initiating one or more second inference processes of the transformer model using the one or more speculated tokens, the one or more second inference processes respectively performed by one or more additional threads; detecting whether any of the one or more speculated tokens matches a predicted token generated at a final layer of the sequence of layers during the first inference process, the predicted token generated after the one or more second inference processes are initiated; and in response to determining that a speculated token matches the predicted token, instructing an additional thread of the one or more additional threads to complete a second inference process of the transformer model using the speculated token, the additional thread corresponding to the speculated token.

15. The one or more non-transitory computer-readable media of claim 14, wherein the operations further comprise: in response to determining that the speculated token matches the predicted token, preventing the first thread from performing another inference process of the transformer model.

16. The one or more non-transitory computer-readable media of claim 14 or 15, wherein the second inference process of the transformer model comprises executing the final layer.

17. The one or more non-transitory computer-readable media of claim 14 or 15, wherein the second inference process of the transformer model comprises executing the layer and bypassing one or more layers subsequent to the layer, the one or more layers comprising the final layer.

18. The one or more non-transitory computer-readable media of any one of claims 14-17, wherein the operations further comprise: in response to determining that none of the one or more speculated tokens matches the predicted token, instructing the first thread to perform another inference process of the transformer model using the predicted token.

19. The one or more non-transitory computer-readable media of any one of claims 14-18, wherein the operations further comprise: in response to determining that none of the one or more speculated tokens matches the predicted token, instructing the one or more additional threads to terminate the one or more second inference processes of the transformer model.

20. The one or more non-transitory computer-readable media of any one of claims 14-19, wherein the operations further comprise: selecting the one or more speculated tokens from a plurality of tokens generated from the layer in the first inference process of the transformer model, wherein selecting the one or more speculated tokens comprises selecting the one or more speculated tokens based on one or more confidence scores of the transformer model for the one or more speculated tokens and a threshold value.

21. The one or more non-transitory computer-readable media of any one of claims 14-20, wherein the one or more second inference processes are in parallel with a part of the first inference process.

22. The one or more non-transitory computer-readable media of any one of claims 14-21, wherein instructing the thread to perform the first inference process of the transformer model comprises: reducing a data precision of input data of one or more layers of the transformer model; and instructing the thread to execute the one or more layers with the input data having the reduced data precision.

23. The one or more non-transitory computer-readable media of any one of claims 14-22, wherein the first thread is on a different type of processing unit from the one or more additional threads.

24. The one or more non-transitory computer-readable media of claim 23, wherein the one or more additional threads are on a graphics processing unit or a neural processing unit, and the first thread is on a central processing unit or a neural processing unit.

25. An apparatus comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform the method of any one of claims 1-13.

Images