Key-value selection for early exit of a machine learning network

The early exit configuration in transformer-based networks addresses inefficiencies in key-value processing by selectively skipping layers based on hidden state vectors, enhancing efficiency and reducing computational overhead.

WO2026240841A1PCT designated stage Publication Date: 2026-11-19QUALCOMM INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/US2026/017559
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-16
Filing Date
2026-03-03
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

The processing of large amounts of video data for high-quality video consumption places a burden on communication networks and devices, and existing neural network architectures, such as transformer-based models, incur significant overhead in key-value processing due to the need to compute attention for all previous tokens, leading to inefficient hardware resource utilization.

Method used

Implementing a difficulty-aware early exit configuration in transformer-based machine learning networks, where an early exit prediction is made based on hidden state vectors to skip processing by certain layers, allowing for selective key-value selection and reducing unnecessary computations.

Benefits of technology

This approach enhances efficiency by reducing hardware resource utilization and computational overhead, improving processing power usage in transformer-based models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2026017559_19112026_PF_FP_ABST
    Figure US2026017559_19112026_PF_FP_ABST
Patent Text Reader

Abstract

Systems and techniques are provided for key-value (KV) selection for token generation. A sequence of a plurality of tokens input to a machine learning network with a plurality of transformer layers including first and second subsets of layers can be obtained, where the first subset of layers includes a configured early exit layer. A first value of an early exit prediction can be determined from a hidden state vector output generated for a respective token of the sequence by the early exit layer, and causes the second subset of layers to skip processing the hidden state vector. Based on the first value of the early exit prediction, KV selection information can be determined for a transformer layer of the second subset and a subsequent token in the sequence, and can comprise the hidden state vector output generated for the respective token by the early exit layer.
Need to check novelty before this filing date? Find Prior Art

Description

PATENTQualcomm Docket No 2502632WO1KEY- VALUE SELECTION FOR EARLY EXIT OF A MACHINE LEARNING NETWORKINTRODUCTION

[0001] The present disclosure generally relates to machine learning networks. For example, aspects of the present disclosure relate to systems and techniques for early exit machine learning models (e.g.. including language models and / or large language models, etc.) during key -value (KV) processing for one or more machine learning layers.BACKGROUND

[0002] Many devices and systems allow video data to be processed and output for consumption. Digital video data includes large amounts of data to meet the demands of consumers and video providers. For example, consumers of video data desire high quality video, including high fidelity, resolutions, frame rates, and the like. As a result, the large amount of video data that is required to meet these demands places a burden on communication networks and devices that process and store the video data.

[0003] An artificial neural network attempts to replicate, using computer technology, logical reasoning performed by the biological neural networks that constitute animal brains. Deep neural networks, such as convolutional neural networks, are widely used for numerous applications, such as object detection, object classification, object tracking, big data analysis, among others. For example, convolutional neural networks are able to extract high-level features, such as facial shapes, from an input image, and use these high-level features to output a probability that, for example, an input image includes a particular object.SUMMARY

[0004] The following presents a simplified summary relating to one or more aspects disclosed herein. Thus, the following summary should not be considered an extensive overview relating to all contemplated aspects, nor should the following summary be considered to identify key or critical elements relating to all contemplated aspects or to delineate the scope associated with any particular aspect. Accordingly, the following summary7has the sole purpose to present certain concepts relating to one or more aspects relating to the mechanisms disclosed herein in a simplified form to precede the detailed description presented below.PATENTQualcomm Docket No 2502632WO2

[0005] Disclosed are systems, methods, apparatuses, and computer-readable media for early exit configurations for machine learning networks and / or models (e g., including language models, and / or large language models (LLMs), among various others), for example based on using difficulty-aware key-value (KV) selection and / or processing. In some aspects, a method is provided. The method includes: obtaining a sequence of a plurality of tokens corresponding to an input to a machine learning network including a plurality of transformer layers, the plurality of transformer layers including a first subset of layers and a second subset of layers, wherein the first subset of layers includes a configured early exit layer; determining an early exit prediction for a respective token included in the sequence of the plurality of tokens, the early exit prediction determined based on a hidden state vector generated as output for the respective token by the configured early exit layer, wherein a first value of the early exit prediction corresponds to skipping processing of the hidden state vector for the respective token by the second subset of layers; and determining, based on the early exit prediction for the respective token, key-value (KV) selection information corresponding to one or more transformer layers of the second subset of layers and a subsequent token included in the sequence of the plurality of tokens after the respective token, wherein the KV selection information comprises the hidden state vector generated as output for the respective token by the configured early exit layer, based on the first value of the early exit prediction.

[0006] In some aspects, an apparatus is provided. The apparatus includes at least one memory and at least one processor coupled to the at least one memory and configured to: obtain a sequence of a plurality of tokens corresponding to an input to a machine learning network including a plurality of transformer layers, the plurality of transformer layers including a first subset of layers and a second subset of layers, wherein the first subset of layers includes a configured early exit layer; determine an early exit prediction for a respective token included in the sequence of the plurality of tokens, the early exit prediction determined based on a hidden state vector generated as output for the respective token by the configured early exit layer, wherein a first value of the early exit prediction causes the at least one processor to skip processing of the hidden state vector for the respective token by the second subset of layers; and determine, based on the early exit prediction for the respective token, key-value (KV) selection information corresponding to one or more transformer layers of the second subset of layers and a subsequent token included in the sequence of the pl urality of tokens after the respectivePATENTQualcomm Docket No 2502632WO3token, wherein the KV selection information comprises the hidden state vector generated as output for the respective token by the configured early exit layer, based on the first value of the early exit prediction.

[0007] In some aspects, a non-transitory computer-readable storage medium is provided including instructions stored thereon which, when executed by at least one processor, causes the at least one processor to: obtain a sequence of a plurality of tokens corresponding to an input to a machine learning network including a plurality of transformer layers, the plurality’ of transformer layers including a first subset of layers and a second subset of layers, wherein the first subset of layers includes a configured early exit layer; determine an early exit prediction for a respective token included in the sequence of the plurality7of tokens, the early exit prediction determined based on a hidden state vector generated as output for the respective token by the configured early exit layer, wherein a first value of the early exit prediction causes the at least one processor to skip processing of the hidden state vector for the respective token by the second subset of layers; and determine, based on the early exit prediction for the respective token, keyvalue (KV) selection information corresponding to one or more transformer layers of the second subset of layers and a subsequent token included in the sequence of the plurality of tokens after the respective token, wherein the KV selection information comprises the hidden state vector generated as output for the respective token by7the configured early7exit layer, based on the first value of the early exit prediction.

[0008] In some aspects, an apparatus is provided. The apparatus includes: means for obtaining a sequence of a plurality of tokens corresponding to an input to a machine learning network including a plurality of transformer layers, the plurality of transformer layers including a first subset of layers and a second subset of layers, wherein the first subset of layers includes a configured early exit layer; means for determining an early exit prediction for a respective token included in the sequence of the plurality of tokens, the early exit prediction determined based on a hidden state vector generated as output for the respective token by the configured early exit layer, wherein a first value of the early exit prediction causes at least one processor of the apparatus to skip processing of the hidden state vector for the respective token by the second subset of layers: and means for determining, based on the early exit prediction for the respective token, key-value (KV) selection information corresponding to one or more transformer layers of the secondPATENTQualcomm Docket No 2502632WO4subset of layers and a subsequent token included in the sequence of the plurality of tokens after the respective token, wherein the KV selection information comprises the hidden state vector generated as output for the respective token by the configured early exit layer, based on the first value of the early exit prediction.

[0009] In some aspects, one or more of the apparatuses described herein is, is part of, or includes a mobile device (e.g., a mobile telephone or so-called “smart phone”, a tablet computer, or other type of mobile device), a wearable device, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a vehicle (or a computing device of a vehicle), a personal computer, a laptop computer, a video server, a television (e.g., a network-connected television), or other device. In some aspects, the apparatus includes at least one camera for capturing one or more images or video frames. For example, the apparatus(es) can include a camera (e.g., a red-green-blue (RGB) camera) or multiple cameras for capturing one or more images and / or one or more videos including video frames. In some aspects, the apparatus(es) includes a display for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the apparatus(es) includes at least one transmitter (or at least one transceiver) configured to transmit one or more video frame and / or syntax data over a transmission medium to at least one device. In some aspects, the at least one processor of the apparatus noted above includes a neural processing unit (NPU), a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), or other processing device or component.

[0010] Aspects generally include a method, apparatus, system, computer program product, non-transitory computer-readable medium, user device, user equipment, wireless communication device, and / or processing system as substantially described with reference to and as illustrated by the ddrawings and specification.

[0011] Some aspects include a device having a processor configured to perform one or more operations of any of the methods summarized above. Further aspects include processing devices for use in a device configured with processor-executable instructions to perform operations of any of the methods summarized above. Further aspects include a non-transitory processor-readable storage medium having stored thereon processor-executable instructions configured to cause a processor of a device to perform operations of any of the methods summarized above. Further aspects include a device having meansPATENTQualcomm Docket No 2502632WO3for performing functions of any of the methods summarized above.

[0012] The foregoing has outlined rather broadly the features and technical advantages of examples according to the disclosure in order that the detailed description that follows may be better understood. Additional features and advantages will be described hereinafter. The conception and specific examples disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. Characteristics of the concepts disclosed herein, both their organization and method of operation, together with associated advantages will be better understood from the following description when considered in connection with the accompanying figures. Each of the figures is provided for the purposes of illustration and description, and not as a definition of the limits of the claims. The foregoing, together with other features and aspects, will become more apparent upon referring to the following specification, claims, and accompanying drawings.

[0013] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all ddrawings, and each claim. The foregoing, together with other features and aspects, will become more apparent upon referring to the following specification, claims, and accompanying ddrawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying ddrawings are presented to aid in the description of various aspects of the disclosure and are provided solely for illustration of the aspects and not limitation thereof. So that the above-recited features of the present disclosure can be understood in detail, a more particular description, briefly summarized above, may be had by reference to aspects, some of which are illustrated in the appended ddrawings. It is to be noted, however, that the appended ddrawings illustrate only certain ty pical aspects of this disclosure and are therefore not to be considered limiting of its scope, for the description may admit to other equally effective aspects. The same reference numbers in different ddrawings may identify the same or similar elements.PATENTQualcomm Docket No 2502632WO6

[0015] FIG. 1 illustrates an example implementation of a system-on-a-chip (SOC), in accordance with some examples;

[0016] FIGS. 2A-C illustrate respective examples of a fully connected, locally connected, and convolutional neural network, in accordance with some examples;

[0017] FIG. 3 is a diagram illustrating an example of a Transformer-based machine learning architecture, in accordance with some examples;

[0018] FIG. 4 is a diagram illustrating an example of a Key -Value (KV) cache that can be associated with a Transformer-based machine learning architecture and / or attention determination, in accordance with some examples;

[0019] FIG. 5 is a diagram illustrating token sequences within an example of a KV cache associated with example early exit configurations, in accordance with some examples;

[0020] FIG. 6 is a diagram illustrating an example machine learning architecture configured with early exiting for one or more layers, in accordance with some examples;

[0021] FIG. 7 is a diagram illustrating an example of difficulty-aware KV selection that can be performed using an early exit router for one or more Transformer layers, in accordance with some examples;

[0022] FIG. 8 is a diagram illustrating an example of autoregressive token generation using a plurality of Transformer machine learning layers, in accordance with some examples;

[0023] FIG. 9 is a diagram illustrating an example of autoregressive token generation using a plurality of Transformer machine learning layers configured for early exit and KV selection, in accordance with some examples;

[0024] FIG. 10A is a diagram illustrating an example of pre-filling and autoregressive token generation in a vanilla Transformer machine learning architecture, in accordance with some examples;

[0025] FIG. 10B is a diagram illustrating an example of pre-filling and autoregressive token generation in an early exit KV selection Transformer machine learning architecture, in accordance with some examples;PATENTQualcomm Docket No 2502632WO7

[0026] FIG. 11 is a flow chart diagram illustrating an example of a process for simulation parameter optimization, in accordance with some examples;

[0027] FIG. 12 is a block diagram illustrating an example of a deep learning network, in accordance with some examples;

[0028] FIG. 13 is a block diagram illustrating an example of a convolutional neural network, in accordance with some examples;

[0029] FIG. 14 is a block diagram illustrating a deep convolutional network (DCN), in accordance with some examples; and

[0030] FIG. 15 illustrates an example computing device architecture of an example computing device which can implement the various techniques described herein.DETAILED DESCRIPTION

[0031] Certain aspects of this disclosure are provided below for illustration purposes. Alternate aspects may be devised without departing from the scope of the disclosure. Additionally, well-known elements of the disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of the disclosure. Some of the aspects described herein may be applied independently and some of them may be applied in combination as would be apparent to those of skill in the art. In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of aspects of the application. However, it will be apparent that various aspects may be practiced without these specific details. The figures and description are not intended to be restrictive.

[0032] The ensuing description provides example aspects, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the example aspects will provide those skilled in the art with an enabling description for implementing an example aspect. It should be understood that various changes may be made in the function and arrangement of elements without departing from the scope of the application as set forth in the appended claims.

[0033] Neural network (NN) machine learning models have advanced and evolved rapidly, and continue to increase in complexity. Various neural network machine learning architectures can include convolutional neural networks (CNNs), deep neural networksPATENTQualcomm Docket No 2502632WO8(DNNs), recurrent neural networks (RNNs), and / or Transformer-based model architectures, etc., among various others. In some examples, increases in accuracy and / or other performance metrics measured for various neural network architectures and tasks can be associated with a corresponding increase in hardware resource utilization when running the model inference. For example, the hardware resource utilization of a neural network machine learning model can correspond to information such as memory footprint, number of model parameters (e.g., model size), operations count, batch size, maximum net memory utilization for batches of different size, inference time, power consumption, etc., among various others.

[0034] Transformer-based machine learning architectures and / or models can implement an attention mechanism, which can determine various types of attention (e.g., self-attention, cross-attention, windowed attention, etc.) to differentially weight the relative importance of various portions of an input sequence for generating one or more outputs. For example, transformer attention can use a framework of Query-Key -Value (KQV) computations, where an attention function maps a query' (Q) and a set of Key-Value (KV) pairs to an output. Outputs can be determined as weighted sums of the values (V), with a respective weight associated with each value determined based on a function of the query (Q) with the corresponding key (K). For example, given a query matrix Q, a key matrix K, and a value matrix V, transformers may determine a scaled dot-product where dkrepresents thedimensions of the keys.

[0035] Input embeddings are linearly projected to obtain the queries, keys, and values for determining attention, using the respective learned weight matrices WQ (e.g., corresponding to learned weights for the Queries matrix), WK (e.g., corresponding to learned weights for the Keys matrix), and Wy (e.g., corresponding to learned weights for the Values matrix). Q, K, and V can then be packed together into matrices, where each row of a Q, K or V matrix corresponds to a different token position. For example, a Q, K, or V matrix with 100 rows may correspond to an input sequence with 100 different token positions, etc.

[0036] In at least some transformer architectures, including for decoder-only models and / or transformers configured for autoregressive decoding (e.g., such as for variousPATENTQualcomm Docket No 2502632WO9language model (LM) and / or large language model (LLM) implementations), a prefdl phase may be performed to process the input sequence prior to autoregressive token-bytoken generation. For example, a machine learning network can include a plurality of transformer layers, where each transformer layer implements an attention mechanism using the output of the previous transformer layer of the machine learning network. During inference, attention is computed over all previous tokens that are before the currently processed token (e.g., for each token, the model attends to all previous tokens). In decoder-only language models (LMs) and / or LLMs, the self-attention block included in each transformer layer may be configured to derive one set of query vectors (Q), key vectors (K), and value vectors (V) for each input token t, from a set of linear proj ections of the corresponding hidden state output from the preceding transformer layer of the LM or LLM machine learning model. When the tokens corresponding to the prompt are presented to the model, all prompt tokens must be processed by the entire decoder (e.g., each layer of a plurality of transformer layers in the decoder architecture) to compute key vectors and value vectors needed by the last input token of the prompt, to thereby perform the next work prediction. Processing all of the preceding input prompt tokens to compute the key vectors and value vectors (e.g., KV vector or tensor, etc.) for the final input prompt token before predicting the next token or word prediction of the output represents a large overhead in processing power during the prefilling phase of KV vectors for the different transformer layers / and tokens t corresponding to the prompt token sequence.

[0037] KV caching can be used to cache respective key-value information for the respective transformer layers of the plurality of transformer layers included in the machine learning network. For example, the initial token generation can be performed at a first time step for a first (e.g., initial) token of an input sequence. For each layer, the corresponding K and V can be calculated from the input token and stored. At subsequent steps for later tokens of the input sequence. Q, K, and V can be calculated only for the new token at each transformer layer. The new K and V calculated for the new token can be concatenated to cached KV tensors from previous steps, using a respective KV cache associated with and / or maintained for each respective transformer layer included in the plurality of transformer layers of the machine learning network. For example, the KV cache for each transformer layer can store (e.g., cache) the KV tensor from the previous state(s), step(s), and / or token(s), etc., processed by the particular transformer layer. When a next (e.g., new) token from the input sequence is processed, the particular layer canPATENTQualcomm Docket No 2502632WO10determine the K and V for the new token, and can append the determined K and V to the cached KV tensor for the previous tokens that is stored within the KV cache for the particular transformer layer. The full concatenated KV tensor can then be used to compute attention for the current step (e.g., current token) at the particular transformer layer. In some examples, at each layer I of a plurality of transformer layers of a machine learning network, the KV cache for each transformer layer I at the time t (e.g., the time where token t of the input sequence is processed) can hold Ki = [Ki i, Kij, Kij,..., Kijt-ii\ and Vi = [Vi j, Vi_2, Vij,.... Vi_(t-i)\, and the Query Qi _tcan be newly computed and used with the KV cache to determine attention for the token t for the layer I.

[0038] The KV cache prefdl phase is used to process all input prompt tokens together to generate the K and V token embeddings (e.g., KV tokens) for each transformer layer, and to initialize the KV cache with these resulting KV tokens. After prefill, the full K and V matrices for all layers are cached, and can be used for more efficient future token generation by avoiding recomputation of past activations. In some cases, prefdl for KV processing and / or KV cache uses dense computation that can scale quadratically with the length n of the input prompt sequence. The KV cache size per layer may increase linearly with n, as each transformer layer is associated with a corresponding KV cache that stores K and V information for each token position within the input prompt sequence length n. Each token generation may be performed using each layer of the plurality of transformer layers included in the machine learning network. Each token is processed by all transformer layers. It may be beneficial for systems and techniques that can be used to improve the efficiency of KV processing and / or the KV cache, for example based on using a lesser number of the plurality of transformer layers (e.g., a subset of the plurality of transformer layer) to predict a first subset of output tokens, while using the full plurality of transformer layers to predict a second subset of output tokens. For example, it may be beneficial to configure an early exit architecture for transformer-based machine learning models and / or networks. It may further be beneficial to provide an early exit transformer architecture that can be configured to reuse the same language model head for all early exit points from within the plurality of transformer layers.

[0039] Systems, apparatuses, methods (also referred to as processes), and computer-readable media (collectively referred to as "‘systems and techniques”) are described herein that can be used to perform difficulty-aware key-value (KV) selection for early exit of aPATENTQualcomm Docket No 2502632WO11configured transformer layer of a plurality of transformer layers included in a machine learning network. For example, the difficulty-aware KV selection can be implemented based on an early exit prediction determined for each token of a sequence of a plurality of tokens processed by the plurality of transformer layers. The early exit prediction can be determined based on an output of hidden state vectors (e.g., hidden state embeddings) generated from an attention calculation for a respective token of the sequence, where the hidden state vectors are generated as output of a configured early exit layer. The configured early exit layer can be a particular transformer layer connected between a first subset of transformer layers of the plurality of transformer layers, and a second subset of transformer layers of the plurality of transformer layers. For example, the configured early exit layer can be a last transformer layer included in the first subset of transformer layers. The configured early exit layer can be connected to a first transformer layer included in the second subset of transformer layers. For example, the input of the configured early exit layer can be the output of the second to last transformer layer of the first subset. The output of the configured early exit layer (e.g., the output of the last transformer layer of the first subset) can be connected to the input of the first transformer layer of the second subset of transformer layers. In some cases, the early exit prediction can be determined as a prediction output by an early exit router connected between the configured early exit layer and the second subset of transformer layers (e.g., connected between the configured early exit layer and the first layer of the second subset of transformer layers).

[0040] In some cases, a sequence of a plurality of tokens corresponding to an input to a machine learning network including a plurality of transformer layers can be obtained, where the plurality of transformer layers includes the first subset of layers and the second subset of layers, with the configured early exit layer included in the first subset of layers and connected to a first layer of the second subset of layers. An early exit prediction can be determined (e.g., by the early exit router) for a respective token included in the sequence of the plurality of tokens. For example, the early exit prediction can be determined based on a hidden state vector generated as output for the respective token by the configured early exit layer. A first value of the early exit prediction corresponds to an early exit of processing for the respective token, where the machine learning network performs the early exit to skip processing of the hidden state vector for the respective token by the second subset of layers.PATENTQualcomm Docket No 2502632WO12

[0041] In some examples, to skip processing of the hidden state vector for the respective token by the second subset of layers, the first value of the early exit prediction causes the machine learning network to provide the hidden state vector generated as output for the respective token by the configured early exit layer to a language model head connected to an output of a final transformer layer of the second subset of layers. The language model head and / or a softmax and token sampling block connected to the output of the language model head can be used to determine sampling information corresponding to a predicted next token for the sequence of the plurality of tokens using the hidden state vector generated as output for the respective token by the configured early exit layer.

[0042] A second value of the early exit prediction can cause the machine learning network to not skip processing of the hidden state vector for the respective token by the second subset of layers. For example, based on the second value of the early exit prediction, the hidden state vector for the respective token can be processed by each respective transformer layer included in the second subset of layers.

[0043] In some examples, the first subset of layers comprises a first subset of transformer layers of the plurality of transformer layers, and the second subset of layers comprises a second subset of transformer layers of the plurality of transformer layers. The first subset of transformer layers can be configured to determine self-attention for the respective token, in some cases. In some examples, the second subset of transformer layers can be configured to determine cross-attention for the subsequent token based on the first value of the early exit prediction for the respective token and using the KV selection information. For example, the cross-attention for the subsequent token can be determined by the second subset of layers by using the KV selection information to determine a respective keys (K) vector input and a respective values (V) vector input for each transformer layer of the second subset of transformer layers. The respective K vector input and the respective V vector input can be determined from a cached representation of the hidden state vector generated as output for the respective token by the configured early exit layer. In some cases, KV selection information can be determined based on the early exit prediction for the configured early exit layer and / or the hidden state output generated for the respective token by the configured early exit layer. The determination of the KV selection information can be associated with and / or may correspond to a prefill phase used to configure KV vectors within a respective KV cache associated with eachPATENTQualcomm Docket No 2502632WO13transformer layer of the second subset of layers. The configured early exit layer can be associated with an early exit router configured to determine the early exit prediction based on the hidden state vector generated as output for the respective token by the configured early exit layer.

[0044] Various aspects of the present disclosure will be described with respect to the figures.

[0045] FIG. 1 illustrates an example implementation of a system-on-a-chip (SOC) 100, which may include a central processing unit (CPU) 102 or a multi-core CPU, configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computational device (e.g., neural network with weights), delays, frequency bin information, task information, among other information may be stored in a memory block associated with a neural processing unit (NPU) 108, in a memory block associated with a CPU 102, in a memory block associated with a graphics processing unit (GPU) 104, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 118, and / or may be distributed across multiple blocks. Instructions executed at the CPU 102 may be loaded from a program memory associated with the CPU 102 or may be loaded from a memory block 118.

[0046] The SOC 100 may also include additional processing blocks tailored to specific functions, such as a GPU 104, a DSP 106, a connectivity block 110, which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, and the like, and a multimedia processor 112 that may, for example, detect and recognize gestures. In some implementations, the NPU is implemented in the CPU 102, DSP 106, and / or GPU 104. The SOC 100 may also include a sensor processor 114, image signal processors (ISPs) 116, and / or storage 120.

[0047] The SOC 100 may be based on an ARM instruction set. In an aspect of the present disclosure, the instructions loaded into the CPU 102 may comprise code to search for a stored multiplication result in a lookup table (LUT) corresponding to a multiplication product of an input value and a filter weight. The instructions loaded into the CPU 102 may also comprise code to disable a multiplier during a multiplication operation of thePATENTQualcomm Docket No 2502632WO14multiplication product when a lookup table hit of the multiplication product is detected. In addition, the instructions loaded into the CPU 102 may comprise code to store a computed multiplication product of the input value and the filter weight when a lookup table miss of the multiplication product is detected.

[0048] SOC 100 can be part of a computing device or multiple computing devices. In some examples, SOC 100 can be part of an electronic device (or devices) such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), a telephone system (e.g., a smartphone, a cellular telephone, a conferencing system, etc.), a desktop computer, an XR device (e.g., a head-mounted display, etc.), a smart wearable device (e.g., a smart watch, smart glasses, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a television, a display device, a system-on-chip (SoC), a digital media player, a gaming console, a video streaming device, a server, a drone, a computer in a car, an Internet-of-Things (IoT) device, or any other suitable electronic device(s).

[0049] In some implementations, the CPU 102, the GPU 104, the DSP 106, the NPU 108, the connectivity block 110, the multimedia processor 112, the sensor processor 114, the ISPs 116, the memory block 118 and / or the storage 120 can be part of the same computing device. For example, in some cases, the CPU 102, the GPU 104, the DSP 106, the NPU 108, the connectivity block 110, the multimedia processor 112, the sensor processor 114, the ISPs 116, the memory block 118 and / or the storage 120 can be integrated into a smartphone, laptop, tablet computer, smart wearable device, video gaming system, server, and / or any other computing device. In other implementations, the CPU 102, the GPU 104, the DSP 106, the NPU 108, the connectivity block 110, the multimedia processor 112, the sensor processor 114, the ISPs 116, the memory block 118 and / or the storage 120 can be part of two or more separate computing devices.

[0050] Machine learning (ML) can be considered a subset of artificial intelligence (Al). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inference, without the use of explicit instructions. An example of a ML system is a neural network (also referred to as an artificial neural network), which may include an interconnected group of artificial neurons (e.g., neuron models). Neural networks may be used for various applications and / or devices, such as image and / or video coding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices.PATENTQualcomm Docket No 2502632WO15autonomous vehicles, service robots, among others.

[0051] Individual nodes in a neural network may emulate biological neurons by taking input data and performing simple operations on the data. The results of the simple operations performed on the input data are selectively passed on to other neurons. Weight values are associated with each vector and node in the network, and these values constrain how input data is related to output data. For example, the input data of each node may be multiplied by a corresponding weight value, and the products may be summed. The sum of the products may be adjusted by an optional bias, and an activation function may be applied to the result, yielding the node’s output signal or ‘'output activation” (sometimes referred to as a feature map or an activation map). The weight values may initially be determined by an iterative flow of training data through the network (e.g., weight values are established during a training phase in which the network learns how to identify particular classes by their typical input data characteristics).

[0052] Different types of neural networks exist, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), multilayer perceptron (MLP) neural networks, transformer neural networks, among others. For instance, convolutional neural networks (CNNs) are a type of feed-forward artificial neural network. Convolutional neural networks may include collections of artificial neurons that each have a receptive field (e.g., a spatially localized region of an input space) and that collectively tile an input space. RNNs work on the principle of saving the output of a layer and feeding the output back to the input to help in predicting an outcome of the layer. A GAN is a form of generative neural network that can learn patterns in input data so that the neural network model can generate new synthetic outputs that reasonably could have been from the original dataset. A GAN can include two neural networks that operate together, including a generative neural network that generates a synthesized output and a discriminative neural network that evaluates the output for authenticity. In MLP neural networks, data may be fed into an input layer, and one or more hidden layers provide levels of abstraction to the data. Predictions may then be made on an output layer based on the abstracted data.

[0053] Deep learning (DL) is an example of a machine learning technique and can be considered a subset of ML. Many DL approaches are based on a neural network, such as an RNN or a CNN, and utilize multiple layers. The use of multiple layers in deep neuralPATENTQualcomm Docket No 2502632WO16networks can permit progressively higher-level features to be extracted from a given input of raw data. For example, the output of a first layer of artificial neurons becomes an input to a second layer of artificial neurons, the output of a second layer of artificial neurons becomes an input to a third layer of artificial neurons, and so on. Layers that are located between the input and output of the overall deep neural network are often referred to as hidden layers. The hidden layers learn (e.g., are trained) to transform an intermediate input from a preceding layer into a slightly more abstract and composite representation that can be provided to a subsequent layer, until a final or desired representation is obtained as the final output of the deep neural network.

[0054] As noted above, a neural network is an example of a machine learning system, and can include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes of the input layer, processing is performed by hidden nodes of the one or more hidden layers, and an output is produced through output nodes of the output layer. Deep learning networks typically include multiple hidden layers. Each layer of the neural network can include feature maps or activation maps that can include artificial neurons (or nodes). A feature map can include a filter, a kernel, or the like. The nodes can include one or more weights used to indicate an importance of the nodes of one or more of the layers. In some cases, a deep learning network can have a series of many hidden layers, with early layers being used to determine simple and low-level characteristics of an input, and later layers building up a hierarchy of more complex and abstract characteristics.

[0055] A deep learning architecture may leam a hierarchy of features. If presented with visual data, for example, the first layer may learn to recognize relatively simple features, such as edges, in the input stream. In another example, if presented with auditory data, the first layer may learn to recognize spectral power in specific frequencies. The second layer, taking the output of the first layer as input, may learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For instance, higher layers may learn to represent complex shapes in visual data or words in auditory data. Still higher layers may learn to recognize common visual objects or spoken phrases. Deep learning architectures may perform especially well when applied to problems that have a natural hierarchical structure. For example, the classification of motorized vehicles may benefit from first learning to recognize wheels, windshields, andPATENTQualcomm Docket No 2502632WO17other features. These features may be combined at higher layers in different ways to recognize cars, trucks, and airplanes.

[0056] Neural networks may be designed with a variety of connectivity patterns. In feed-forward networks, information is passed from lower to higher layers, with each neuron in a given layer communicating to neurons in higher layers. A hierarchical representation may be built up in successive layers of a feed-forward network, as described above. Neural networks may also have recurrent or feedback (also called top-down) connections. In a recurrent connection, the output from a neuron in a given layer may be communicated to another neuron in the same layer. A recurrent architecture may be helpful in recognizing patterns that span more than one of the input data chunks that are delivered to the neural network in a sequence. A connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. A network with many feedback connections may be helpful when the recognition of a high-level concept may aid in discriminating the particular low-level features of an input.

[0057] The connections between layers of a neural network may be fully connected or locally connected. FIG. 2A illustrates an example of a fully connected neural network 202. In a fully connected neural network 202, a neuron in a first hidden layer may communicate its output to every neuron in a second hidden layer, so that each neuron in the second layer will receive input from every neuron in the first layer. FIG. 2B illustrates an example of a locally connected neural network 204. In a locally connected neural network 204, a neuron in a first hidden layer may be connected to a limited number of neurons in a second hidden layer. More generally, a locally connected layer of the locally connected neural network 204 may be configured so that each neuron in a layer will have the same or a similar connectivity pattern, but with connections strengths that may have different values (e.g., 210, 212, 214, and 216). The locally connected connectivity pattern may give rise to spatially distinct receptive fields in a higher layer, because the higher layer neurons in a given region may receive inputs that are tuned through training to the properties of a restricted portion of the total input to the network.

[0058] An example of a locally connected neural network is a convolutional neural network. FIG. 2C illustrates an example of a convolutional neural network 206. The convolutional neural network 206 may be configured such that the connection strengths associated with the inputs for each neuron in the second layer are shared (e.g., 208).PATENTQualcomm Docket No 2502632WO18Convolutional neural networks may be well suited to problems in which the spatial location of inputs is meaningful.

[0059] FIG. 3 is a diagram illustrating an example of a Transformer-based machine learning architecture 300 including a plurality of layers, in accordance with some examples. In some aspects, the Transformer-based machine learning architecture 300 can be used to process an input 302 (e.g., an input comprising a prompt and / or generated text used for a query or other input to an LLM, among various other machine learning models or architectures that may be implemented using the Transformer-based machine learning architecture 300). The Transformer-based machine learning architecture 300 can be used to process the input 302 (e.g., prompt and generated text), to generate a corresponding output 375. For example, the corresponding output 375 generated from the input 302 can be an output indicative of a next token probability, etc.

[0060] The input 302 can be processed to generate a corresponding input embedding 305, where the input embedding 305 comprises a feature vector or other latent space representation associated with the Transformer-based machine learning architecture 300 and representative of the input 302 prompt and text. In some examples, a plurality of Transformer layers (e.g., including Transformer machine learning layers and / or Transformer-based machine learning layers, etc.) can process the input embedding 305 to generate an intermediate output that is provided to a language model head 360 (e.g., a linear projection layer) and a softmax layer 370 that generate the output 375 indicative of the next token probability.

[0061] The plurality of Transformer layers included in the Transformer-based machine learning architecture 300 can include a first set of layers 310 (e.g., Layer 1, Layer 2..... Layer / -I), a Layer / 320, a Layer / +1 330, and a set of layers 350 (e.g., Layer l+2, Layer l+3,..., Layer N). The plurality of Transformer layers can include N individual Transformer layers with a connection pattern therebetween. Each of the N Transformer layers may include a Transformer block, for example corresponding to and / or the same as or similar to the example Transformer block 340 included in Layer l+1 330.

[0062] The Transformer block 340 includes a layer norm 342, which for example can be implemented as a root-mean-square (RMS) Norm, etc. configured to receive the input to the current Transformer layer (e.g., Layer l+1 330) comprising the output of thePATENTQualcomm Docket No 2502632WO19previous Transformer layer (e.g., Layer l 320). The output of the RMS Norm (e.g., layer norm) 342 can be represented as Ho.m, and can be processed as the input to a self-attention component 344. The self-attention component 344 can also be referred to as a “selfattention layer,’’ and / or an “attention layer,” etc. In some aspects, the self-attention layer 344 may be a multi-head attention (MHA) attention layer 344 configured to calculate multi-headed attention over the input comprising HQ,™. The output of the MHA attention layer 344 can be provided as input to a second layer norm 346 (e.g., which may also be an RMS Norm, etc.), followed by a multi-layer perceptron (MLP) 348, among various other feed-forward neural network components or blocks, etc.

[0063] The MHA attention layer 344 can be used by the Transformer block 340 and / or the corresponding Transformer layer including the Transformer block 340, to attend to information from different representation subspaces at different positions. For example, Transformer architectures can implement MHA based on attending to different portions of an input sequence simultaneously, using multiple heads. Each head of the multiple heads used to implement MHA may focus (e.g., attend to) different characteristics and / or features of the input, with the combination of the attention across the multiple heads capturing more diverse relationships and features than may be provided by single-headed attention (e.g., using a single attention function over the input sequence).

[0064] For example, rather than performing a single attention function, the MHA attention layer 344 can be used to project query (Q), key (K), and value (V) information a plurality of times for the different portions of the input sequence, where the Q, K, V information is projected multiple times into the multiple heads using learned linear projections. The multiple heads attend separately to their respective projection of Q, K, V information, and each generate a respective attention output or attention context vector, etc. The respective attention outputs from each head of the multiple heads of the MHA attention layer 344 are concatenated and linearly transformed to obtain the output (e.g., Hout) for the Transformer block 340.

[0065] For example, the input 302 to the Transformer block 340 of Layer / +1 330 can be a sequence of tokens that are generated by a tokenizer at the output of the previous layer (e.g., Layer I 320). The input token sequence can comprise a plurality of tokens in a sequence or arrangement where each token ttis associated with a respective index and / or position. For example, the token ttcan represent the / -th token output by aPATENTQualcomm Docket No 2502632WO20tokenizer, and obtained as the input to a respective Transformer block and / or Transformer layer.

[0066] The input embedding 305 can be represented as xt, which is the input embedding for the Lth token tt. For a sequence of a plurality of tokens (e.g., a total of T tokens) where t = 0, 1, 2,..., T-2, T-J, a matrix X = [ o x1...xT-2Xr-i] can include all input embedding vectors corresponding to all tokens. For example, x0is the input embedding vector for token t0. x1is the input embedding vector for token t1..., xT-2is the input embedding vector for token tT-2, and xT-1is the input embedding vector for token tT-1.

[0067] The term htcan be used to represent the hidden state vector or “embedding” at the output of the l-th Transformer layer corresponding to the t-th token tt. For example, h01is the hidden state vector or embedding for token t0at the output of the l=1 first Transformer layer (e.g., Layer 1 included in the set of layers 310), h11is the hidden state vector or embedding for token t1at the output of the l=1 first Transformer layer,..., etc. Each Transformer layer generates a respective hidden state vector output for each different token ttincluded in the sequence of T tokens within the input sequence (e g., and within the matrix X of all input embedding vectors for all tokens). The hidden states for the T tokens after the first Transformer layer Z=1 comprises the hidden state vectors [h01h11h21... hT-11].

[0068] The respective hidden state vectors for each of the T tokens, at the output of each Transformer layer l, can be represented as a matrix of hidden state vectors for the T tokens. For example, the matrix Hl= [h0lh1l... hT-2lhT-1l] is the matrix including all hidden state vectors corresponding to all tokens (e.g., each of the T total tokens tt), after (e.g., output from) the Transformer layer I. In one illustrative example, the output of the l=2 second Transformer layer (e.g., Layer 2 included in the set of layers 310) is the T hidden state vectors [h02h12h22h32... hT-12]. The output of the Layer I Transformer layer 320 is the set of T hidden state vectors [h0lh1lh2l... hT-1l]. The output of the Layer Z+l Transformer layer 330 is the set of T hidden state vectorsThe output of the final Layer TV (e.g., of the set of layers 350) is the set of T hidden state vectors [h0Nh1Nh2N... hT-1N]PATENTQualcomm Docket No 2502632WO21

[0069] In some aspects, a Transformer layer and / or Transformer block 340 can be represented as a function that takes two hidden state inputs and generates one hidden state output:

[0070] Here, Houtis the hidden state output from the transformer, and the two hidden state inputs to the transformer are the first hidden state input HQ in(e.g., including and / or indicative of the respective Query (Q) matrix information), and the second hidden state input HKV in(e.g., including and / or indicative of the respective Key (K) matrix information and Value (V) matrix information). In Eq. (1), the subscript w represents a neural network parameterized by w. Based on the representation and notation of the Transformer representation in Eq. (1):H° = X Eq. (2)H1= TW1(H°, H°) Eq. (3)

[0071] Using the K, Q, and V matrices included in the two hidden state inputs HQ inand HKV inof Eq. (1), and also shown in FIG. 3 as the two inputs to the MHA attention layer 344, the MHA attention layer 344 can determine scaled dot-product attention (e.g., self- attention) as:Attention Q, K, V) = softmax Eq. (5)

[0072] Here, the Queries matrix Q = Wq- X, the Keys matrix K = Wk■ X, and the Values matrix V — Wv■ X. As noted above, the MHA attention layer 344 and / or the Transformer block 340 can perform learned linear projections of the same input A using different weight matrices. The terms Wq, Wk, and Wvrepresent trainable parameters of weight matrices for the Queries matrix Q. the Keys matrix K. and the Values matrix V (respectively). As also noted above, X represents the input matrix for a sequence of token embeddings or hidden states. Each of the weighted relationships from the learned linear projections Q = Wq■ X, K = Wk■ X, and V = Wv■ X maps the input matrix X into a different subspace optimized for the respective role of each of the Q, K, V matrices (e.g., querying, matching, or carrying content, respectively).PATENTQualcomm Docket No 2502632WO22

[0073] The separation of atention using Q, K, V can correspond to the MHA atention layer 344 and / or Transformer block 340, etc., such as determining the MHA atention at block 344 to independently learn how to best form atention queries, keys, and values rather than forcing identical copies of the input. In some examples, the MHA atention layer 344 can learn what to look for (via the queries Q), where to look (via the keys K), and what information to retrieve (via the values F) from the input sequence A of the input embeddings 305 (e.g., at Layer 1) or the input sequence comprising the matrix of all hidden state vectors for the set of tokens represented as the matrix

[0074] As noted above, a prefdl phase may be performed to process an input sequence to generate respective KV tokens for initializing a respective KV cache associated with each transformer layer of a plurality of transformer layers included in a machine learning network or model. For example, the prefill phase can be used to perform KV processing to initialize the KV cache of KV tokens (e.g., KV tensors) for each transformer layer, where the KV cache processing is used to process the input sequence prior to autoregressive token-by -token generation.

[0075] For example, FIG. 4 is a diagram illustrating an example of KV cache processing 400 that can be used to initialize Key-Value tensors within respective KV cache(s) implemented for each transformer layer of a plurality of transformer layers, in accordance with some examples. For example, a first transformer layer can be included in a plurality of transformer layers, and may include a first transformer block 430-1. A second transformer layer of the plurality of transformer layers can include a second transformer block 430-2. A third transformer layer of the plurality of transformer layers can include a third transformer block 430-3, etc. In some aspects, the transformer blocks 430-1, 430-2, 430-2, etc., can correspond to and / or may be the same as or similar to the transformer block layer 330 (or transformer block) of FIG. 3, etc.

[0076] An input embedding 405-1 to the first transformer block 430-1 can be used to obtain a corresponding Query 422-1 for processing by the MHA block of the first transformer block 430-1. During the prefill phase, the corresponding Key (K) information 427-1 (also referred to as “Key 427-1”) and the corresponding Value (V) information 429-1 (also referred to as “Value 429-1”) can be determined for the Query 422-1 for the inputPATENTQualcomm Docket No 2502632WO23embedding 405-1. The Key 427-1 and Value 429-1 information can be stored in the KV cache 427-1 for the transformer block 430-1.

[0077] An input embedding 405-2 to the second transformer block 430-2 corresponds to the output 407-1 from the first transformer block 430-1. and is used to obtain the corresponding Query7422-2 for processing by the MHA block of the second transformer block 430-2. During the prefill phase, the corresponding Key (K) information 427-2 (also referred to as “Key 427-2”) and the corresponding Value (V) information 429-2 (also referred to as “Value 429-2”) can be determined for the Query 422-2 for the input embedding 405-2. The Key 427-2 and Value 429-2 information can be stored in the KV cache 425-2 for the transformer block 430-2. The KV cache 425-2 for the second transformer layer that includes the second transformer block 430-2 additionally includes the same KV cache information as the preceding, first transformer layer with the first transformer block 430-1. The new Key 427-2 and Value 429-2 information can be appended to the KV tensor that is cached at the previous transformer layer with the first transformer block 430-1. For example, the second KV cache 425-2 includes the KV tensors of the previous layer (e.g., Keys 427-1 and Values 429-1), and appends the new KV information (e.g., Keys 427-2 and Values 429-2) determined for the Query 422-2 processed by the second transformer block 430-2.

[0078] An input embedding 405-3 to the third transformer block 430-3 corresponds to the output 407-2 from the second transformer block 430-2, and is used to obtain the corresponding Query 422-3 for processing by the MHA block of the third transformer block 430-3. During the prefill phase, the corresponding Key (K) information 427-3 (also referred to as “Key 427-3”) and the corresponding Value (V) information 429-3 (also referred to as “Value 429-3”) can be determined for the Query 422-3 for the input embedding 405-3. The Key 427-3 and Value 429-3 information can be stored in the KV cache 425-3 for the transformer block 430-3. The KV cache 425-3 for the third transformer layer that includes the third transformer block 430-3 additionally includes the same KV cache information as the preceding second transformer layer with the second transformer block 430-1. The new Key 427-3 and Value 429-3 information can be appended to the KV tensor that is cached at the previous transformer layer with the second transformer block 430-2.PATENTQualcomm Docket No 2502632WO24

[0079] For example, the third KV cache 425-3 includes the KV tensors of the second KV cache 425-2 (e.g., the Keys 427-1 from the first transformer block 430-1, the Keys 427-2 from the second transformer block 430-2; and the Values 429-1 of the first transformer block 430-1 and the Values 429-2 of the second transformer block 430-2), and appends the new KV information comprising Keys 427-3 and Values 429-3, determined for the Query 422-3 processed by the third transformer block 430-3, to obtain the cached KV tensor stored within the third KV cache 425-3.

[0080] In some aspects, KV caching can be used to cache respective key-value information for each transformer layer, where initial token generation can be performed at a first time step for a first (e.g., initial) token of an input sequence, with the corresponding K and V can be calculated from the input token and stored. Subsequently, for each layer and each additional token in the input sequence (e.g., at subsequent steps for later tokens of the input sequence), Q, K, and V can be calculated only for the new token at each transformer layer, with the calculated KV for the new token concatenated to cached KV tensors from previous steps within the respective KV cache associated with and / or maintained for each respective transformer layer.

[0081] For example, the KV cache for each transformer layer can store (e.g., cache) the KV tensor from the previous state(s), step(s), and / or token(s), etc., processed by the particular transformer layer. When a next (e g., new) token from the input sequence is processed, the particular layer can determine the K and V for the new token, and can append the determined K and V to the cached KV tensor for the previous tokens that is stored within the KV cache for the particular transformer layer. The full concatenated KV tensor can then be used to compute attention for the current step (e.g., current token) at the particular transformer layer. In some examples, at each layer / of a plurality of transformer layers of a machine learning network, the KV cache for each transformer layer I at the time t (e.g.. the time where token t of the input sequence is processed) can hold A / = [Ki j. Ki 2, Ki j, Ki_(t.i)\ and Vi = [Vi i. Vi_2, V1. V t-rt], and the QueryQi _tcan be newly computed and used with the KV cache to determine attention for the token t for the layer / .

[0082] The KV cache prefill phase is used to process all input prompt tokens together to generate the K and V token embeddings (e.g., KV tokens) for each transformer layer, and to initialize the KV cache with these resulting KV tokens. After prefill, the full K andPATENTQualcomm Docket No 2502632WO25V matrices for all layers are cached, and can be used for more efficient future token generation by avoiding recomputation of past activations. In some cases, prefill for KV processing and / or KV cache uses dense computation that can scale quadratically with the length n of the input prompt sequence. The KV cache size per layer may increase linearly with n, as each transformer layer is associated with a corresponding KV cache that stores K and V information for each token position within the input prompt sequence length n. Each token generation may be performed using each layer of the plurality of transformer layers included in the machine learning network. Each token is processed by all transformer layers. It may be beneficial for systems and techniques that can be used to improve the efficiency of KV processing and / or the KV cache, for example based on using a lesser number of the plurality of transformer layers (e.g., a subset of the plurality of transformer layer) to predict a first subset of output tokens, while using the full plurality of transformer layers to predict a second subset of output tokens.

[0083] For example, the first subset of output tokens can be the subset (e g., portion) of the generated tokens that are “easy” to generate, and the second subset of output tokens can be the subset (e.g., portion) of the generated tokens that are “hard” to generate. In some aspects, the first subset of output tokens can be output tokens that are generated using early exiting from processing by the entire plurality of transformer layers V, and the second subset of output tokens can be output tokens that are generated without early exiting (e.g., generated using processing by the entire plurality of transformer layers V). For example, in the transformer-based machine learning architecture 300 of FIG. 3, the first subset of tokens can be output tokens that are early exited at an intermediate transformer layer i of the plurality of transformer layers N, where i < N. The intermediate transformer layer i used for the early exit of the first subset of output tokens can also be referred to as an early exit layer and / or an early exiting layer, etc.

[0084] In some aspects, the first subset of tokens can be referred to as early exit tokens, and may be processed using the respective hidden state vectors Hlfor / < i. For example, the early exit layer may be the transformer layer 320 of FIG. 3, where I = i for the transformer layer 320. The first subset of tokens (e.g., early exit tokens) can be generated by the early exit transformer layer 320 based on using the hidden state vectors in Hl, which includes the respective hidden state vector information from layers 1, 2,..., l-l (e.g., the set of layers 310). The early exit tokens may be generated as output tokensPATENTQualcomm Docket No 2502632WO26without processing by the transformer layers that are after the early exit transformer layer 320. For example, the early exit tokens can skip processing by the remaining transformer layers / +1 (e.g., z+1, based on l=i in the instant example) and the set of layers 1+2, 1+3,..., N included in the set of transformer layers 350.

[0085] The second subset of output tokens (e.g., non-early exit tokens) can be generated from processing by each respective transformer layer of the plurality of transformer layers 1 - N included in the example machine learning architecture 300 of FIG. 3. For example, the non-early exit tokens (e.g., "‘hard” tokens) can be generated using the hidden states vectors in HN, which is generated based on fully processing the tokens using each respective transformer layer of the plurality of transformer layers N.

[0086] In some aspects, when a token t is an “easy” token (e.g., when token t is included in the first subset of output tokens, and / or when token t is an early exit token, etc.), the hidden state vector for the token t at the output of the / -th transformer layer used for early exit can be approximated as being equal to and / or similar to the hidden state vector for the token t at the output of the V-th transformer layer when no early exit is performed. For example, when the early exit layer is represented by the transformer layer / = i (e.g., transformer layer 320, with / = z, etc.), the hidden state vector for an early exit token t is hltat the output from the early exit layer / = i. In some aspects, the tokens selected for inclusion in the first subset of early exit tokens can be the subset of tokens with a respective hidden state vector at the early exit layer output (e g., htl) that is equal or similar to the respective hidden state vector at the output of the / V-th layer (e.g., h^, the output if the early exit token t w ere to be processed by the full plurality of transformer layers / V). In some aspects, early exit tokens can be the subset of tokens t w here htlh*, where htlis the hidden state vector at the output of the early exit transformer layer i (e.g., layer 320)is the hidden state vector at the output of the final transformer layer N (e.g., the last layer of the set of transformer layers 350).

[0087] In some aspects, for the subset of early exit tokens, the output of hidden state vectors htlfrom the early exit layer z can be used as an approximation for the subsequent transformer layers I > i for KV computation corresponding to KV selection and / or KV cache computation, for example associated with one or more of a prefill phase and / or token generation. For example, while token t may early exit without processing by the subsequent transformer layers / > i, the KV computation for prefilling and / or tokenPATENTQualcomm Docket No 2502632WO27generation is performed over the hidden state information for all tokens, and the KV computation at each transformer layer still requires as input the hidden states for the transformer layers I > i that are skipped for an early exit token.

[0088] In some aspects, the systems and techniques described herein can be used to implement difficulty-aware KV selection for KV cache processing and / or KV cache generation and / or initializing the KV cache for each transformer layer of a plurality of transformer layers, based on using the early exit layer output hidden state vectors htas an approximation for the hidden state vectors htlat the output of each subsequent transformer layer / > i that is skipped for a respective early exiting token t (e.g., hlt= hlt, for I > i). The approximation can be based on the early exiting tokens being selected as the subset of output tokens with early convergence to the state vector outputobtained by processing with the full plurality of N transformer layers, where the early convergence occurs by the output from the early exit transformer layer htl.

[0089] For non-early exiting tokens (e.g., tokens that are “hard’' to compute or generate as output tokens, and which are processed using each layer of the plurality of N transformer layers), the intermediate hidden state vector at the output of the intermediate, early exit transformer layer 1=1 (e.g., ht) is different from the final hidden state vector output of the final transformer layer l=N (e.g., h ), represented ashlt. For non-early exiting tokens, the early exit approximation of htl(1 > i) by htlcan cause a performance gap in the accuracy of “hard” tokens that are early exited, versus “hard” tokens that are processed by the plurality of transformer layers N.

[0090] The systems and techniques described herein can be used for difficult-aware KV selection for early exit of a transformer-based machine learning model, including an LM and / or LLM model (e.g., among various others), based at least in part on selecting the hidden state vectors for KV computation based on an early exit prediction for each respective token t of a plurality of tokens Tin the input and / or output sequences associated with the model. For example, the hidden state vectors for KV selection and / or computation (e.g., corresponding to token generation, and / or a prefill phase for KV caching, etc.) can be selected for each respective token t and layer I based on a prediction of whether the respective token t will be an early exit token that is early exited at an early exit layer l=i. The layer immediately after the early exit layer l=i is the layer ( / +7>z), andPATENTQualcomm Docket No 2502632WO28can correspond to the transformer layer having an input connected to the output from the early exit transformer layer. In some aspects, the hidden state vectors for KV computation at the next layer after the early exit layer (e.g., the hidden state vectors for KV computation at a layer l+l >i), can be selected as the hidden state vectors hLtif the token t is predicted and / or determined to be an “easy” token (e.g., an early exit token), and can be selected as the hidden state vectors htlif the token t is a “hard” token (e.g.. is not an early exit token).

[0091] In some cases, a Boolean early exit indication can be predicted and / or generated for each respective token t of a sequence of T tokens. For example, etcan represent the Boolean early exit indication for the / -th token, where a value of et— 1 corresponds to an “easy” and / or early exiting token, and a value of et= 0 corresponds to a “hard” and / or non-early exiting token, or vice versa. The respective Boolean early exit indication etfor each respective token t of the sequence of T tokens can be represented within a matrix E of early exit token information and / or early exit token predictions:

[0092] In some aspects, the difficulty -aware KV selection can be implemented according to:H° = X Eq. (8)

[0093] Here, Eq. (9) corresponds to the KV selection for the transformer layers I that are before the early exit layer l=i. At the early exit layer l=i, the difficulty-aware KVPATENTQualcomm Docket No 2502632WO29selection can be implemented according to Eq. (10). For the subsequent layers after the early exit layer l=i, the di ffi cull -aware KV selection can be implemented according to Eq. (11). The terms E and 1 in Eq. (11) can correspond to Eqs. (6) and (7). respectively, and with:

[0094] For the subset of tokens (e.g., of the plurality of tokens T) that are early exited (e.g., “easy” tokens), the term HlE in Eq. (11) and Eq. (12) is equal to Hl, and the term Wl(l — £) is equal to 0. For the subset of tokens that are not early exited (e.g., “hard” tokens), the term HlE in Eq. (11) and Eq. (12) is equal to 0, and the term Hl(l — E) is equal to Hl.

[0095] FIG. 5 is a diagram illustrating example token sequences within an example of a KV cache associated with example early exit configurations, in accordance with some examples. For example, a first token sequence 500a can correspond to a KV selection token sequence for a normal (e.g., vanilla, etc.) transformer architecture that does not perform early exiting for one or more tokens and / or that does not include an early exit router for one or more early exit transformer layers. In the example of the first token sequence 500a, KV selection and / or KV cache computation can be performed for all tokens using the KV tensors computed for all of the preceding layers up to the current layer I for which KV selection and processing for KV cache computation is performed. For example, each token position to, ti. to,.., to. tioo of the first token sequence 500a can be performed according to Eq. (9).

[0096] The second example token sequence 500b and the third example token sequence 500c can correspond to KV selection for a transformer architecture that can perform early exiting for one or more tokens of the sequence of tokens to. ti, to,..., to, tioo. For example, tokens that are early exited are represented in the second example token sequence 500b and the third example token sequence 500c as the grayscale shaded boxes (e.g., including the token positions to, ti, to, etc. as the early exit tokens). In the second token sequence 500b and the third token sequence 500c, the early exit tokens (e.g.., represented by the grayscale shaded boxes within the sequence) can be processed using KV selection according to one or more of Eqs. (10) and / or (11), corresponding to the early exit processing noted above.PATENTQualcomm Docket No 2502632WO30

[0097] In the second token sequence 500b, token 99 (e.g., fo>) is not early exited, and is processed according to the KV-Z processing of Eq. (9). In the third token sequence 500c, token 99 (e.g., Q?) is early exited, and is processed according to the KV-Z processing of Eq. (10) and / or Eq. (11). In each example token sequence (e.g., first token sequence 500a, second token sequence 500b, third token sequence 500c), the processing for token 100 (e.g., tioo) is performed based on information previously determined for the prior tokens 0-99 (e.g., to - too). The early exit tokens identified by the grayscale shading may each be early exited at the same respective early exit layer l=i within the transformer architecture including a plurality of transformer layers N. Increasing the number of early exited tokens included in the set of prior tokens in a token sequence used as context for processing of a subsequent token in the sequence can reduce the KV processing and KV cache size and / or complexity. For example, the third token sequence 500c with the early exit of token 99 too can correspond to reduced KV processing for the KV cache initialization at token 100 tioo, relative to an increased KV processing for the KV cache initialization at token 100 tioo for the second token sequence 500b with no early exit of token 99 too. The KV processing for the KV cache initialization at token 100 tioo for the first token sequence 500a without early exit of any tokens can be higher than that associated with either the second token sequence 500b and the third token sequence 500c.

[0098] FIG. 6 is a diagram illustrating an example machine learning architecture 600 configured with early exiting for one or more layers, in accordance with some examples. In some aspects, the machine learning architecture 600 can correspond to the machine learning architecture 300 of FIG. 3, where the machine learning architecture 600 of FIG.6 further includes an early exit router 630 provided on the output of an early exit transformer layer i.

[0099] For example, an input embedding 605 of FIG. 6 can correspond to the input embedding 305 of FIG. 3. A set of i transformer layers 620 of FIG. 6 can correspond to the first set of transformer layers 320 of FIG. 3. For example, the set of i transformer layers 620 (e.g., also referred to herein as the set of transformer layers 620) can include a first transformer layer 1, a second transformer layer 2, a third transformer layer 3,..., a transformer layer i-1. and an z-th transformer layer i. In some aspects, the z-th transformer layer i within the set of transformer layers 620 can be configured as an early exit transformer layer, and may also be referred to as an early exit-capable transformer layer,PATENTQualcomm Docket No 2502632WO31etc.

[0100] The input embedding 605 can correspond to an input sequence of tokens to, ti, t2...., tt-2, tt-i, tt, tt+i,.... etc. Processing of the input embedding for token ttby the machine learning architecture 600 can be used to generate next token sampling information 675 for the next token tt+i of the sequence. The next token sampling information 675 can include a next token probability (or next token prediction), which may also be referred to as a sampled next token and / or next token sampling information. Processing can be performed for each respective token ttof the sequence, where each respective token ttis either early exited by the early exit router 630 to skip processing by the second set of N-i transformer layers 650, or is not early exited and continues from the early exit router 630 on to processing by the second set of N-i transformer layers 650.

[0101] An early exit router 630 can be provided between the set of transformer layers 620 and a second set of N-i transformer layers 650. In some examples, the early exit router 630 is connected to the output of the z-th, early exit transformer layer (e.g., the final layer of the first set of i transformer layers 620), and is connected to the input of the z+l-th transformer layer (e.g., the first layer of the second set of N-i transformer layers 650).

[0102] In some examples, the second set of N-i transformer layers 650 can correspond to the second set of transformer layers 350 of FIG. 3. In some cases, the language model head 660 of FIG. 6 can correspond to the language model head 360 of FIG. 3, and the softmax sampling block 670 (also referred to as a softmax token sampling block) of FIG.6 can correspond to the softmax layer 370 of FIG. 3. The output of the machine learning architecture 600 of FIG. 6 can be a next token sampling information 675 (e.g., a next token probability or prediction), which may also be referred to as a sampled next token and / or next token sampling information, etc. The output of the next token sampling information 675 (e.g., a next token probability or prediction) can correspond to the output 375 providing the next token probability of FIG. 3.

[0103] In some aspects, the early exit router 630 can be included in the machine learning architecture 600, with the early exit router 630 provided between the z-th layer and the z+l-th layer. The early exit router 630 can be configured to receive the output hidden state htlof the z-th layer of the first set of transformer layers 620. For example, the input to the early exit router 630 can be the hidden state vectors htlgenerated as outputPATENTQualcomm Docket No 2502632WO32from the early exit / -th layer of the first set of transformer layers 620. The hidden state vectors htlobtained as input to the early exit router 630 can correspond to the selfattention calculated by the z-th layer (e.g.. early exit layer in the first set of transformer layers 620).

[0104] Based on the hidden state vectors htloutput by the early exit layer / , the early exit router 630 can be configured to determine and / or generate an early exit prediction etfor each token t in the sequence of tokens of the input embedding 605. For example, the early exit router 630 can generate the respective early exit prediction etfor each token t, where etis indicative of a prediction that the token t will early exit after the early exit layer / (e.g., an early exit prediction of YES, and / or et= 1, etc.) or a prediction that the token t will not early exit after the early exit layer z (e.g., an early exit prediction of NO. and / or et= 0, etc.). In some examples, the respective early exit predictions etfor the plurality of tokens t in the input embedding 605 can correspond to the matrix E of early exit token information and / or early exit token predictions used in Eq. (6), above, where etrepresents a prediction of the actual early exit indicator etfor the token position t during inference of the machine learning architecture 600.

[0105] In cases where the early exit router 630 determines an early exit prediction of YES (e.g., et= 1) for a particular token t, the particular token t can be early exited from processing by the remaining N-i transformer layers of the second set of transformer layers 650, and can be routed directly from the early exit router 630 to the language model head 660 as the output hidden state information for the early exited token h = hlt. Processing for the particular token t is early exited based on skipping the remaining N-i transformer layers included in the second set of transformer layers 650, and generating the corresponding next token sampling information 675 (e.g., a next token probability) by the language model head 660 and softmax sampling block 670 using the early exited hidden state information h = htlfor the particular token position t.

[0106] In cases where the early exit router 630 determines an early exit prediction of NO (e.g.. et= 0) for a particular token t, the particular token it does not early exit and proceeds to processing by the remaining N-i transformer layers of the second set of transformer layers 650. The output hidden state information for the particular token t (e.g., hLt) is routed to the N-i remaining layers in the second set of transformer layers 650 andPATENTQualcomm Docket No 2502632WO33is processed in succession by each respective layer in the second set of transformer layers 650, with the final output of hidden states h* obtained as the output of the final (e.g., N-th) layer of the second set of transformer layers 650 used by the language model head 660 and softmax sampling block 670 to determine the next token sampling information 675 (e.g., a next token probability or prediction) for the next token in the sequence (e.g., token ti+r).

[0107] In some examples, the early exit router 630 can be configured to generate as output the early exit prediction etas a prediction of the early exit indicator etfor each token t during inference using the machine learning architecture 600. In some aspects, the machine learning architecture 600 can be implemented as a transformer and / or transformer-based machine learning architecture. A binary classification head can be used for the early exit predictor, and in some examples the binary classification head can use a number of layers configured as 1 or 2.

[0108] FIG. 7 is a diagram illustrating an example of a machine learning architecture 700 configured to implement difficulty -aware KV selection using an early exit router 730 for one or more Transformer layers, in accordance with some examples. The machine learning architecture 700 includes a set of transformer layers 720 that may include a quantity i of a plurality of transformer layers 1, 2,..., i. The set of transformer layers 720 can correspond to and / or may be the same as or similar to the set of transformer layers 320 of FIG. 3 and / or the set of transformer layers 620 of FIG. 6, etc.

[0109] An early exit router 730 can correspond to and / or may be the same as or similar to the early exit router 630 of FIG. 6, and can be provided between the output of an early exit transformer layer z and the input of the next transformer layer z+1 that is the next layer after the early exit transformer layer i. The next transformer layer z+1 is shown in FIG. 7 as the (z+l)thtransformer layer 742. The machine learning architecture 700 can include a second set of transformer layers, comprising the N-i transformer layers that are included in the plurality of transformer layers and are after the early exit zth transformer layer (e.g., the last layer of the set of transformer layers 720). For example, the machine learning architecture 700 can include a second set of N-i transformer layers corresponding to the second set of transformer layers 650 of FIG. 6 and / or the second set of transformer layers 350 of FIG. 3, etc. The second set of N-i transformer layers included in the machine learning architecture 700 can include the (z+l)thtransformer layer 742, an (z+2)thPATENTQualcomm Docket No 2502632WO34transformer layer 744,..., and an Vhtransformer layer 758.

[0110] In some examples, difficulty-aware KV selection can be performed by the machine learning architecture 700, based on Houtinformation corresponding to the hidden state vectors corresponding to the respective transformer layers 1, 2,..., i included in the set of transformer layers 720. For example, the hidden state vector output for the i-th transformer layer of the set of transformer layers 720 can be the hidden state output vectors htl, which may be input to the early exit router 730 (e.g., corresponding to the input hltto the early exit router 630 of FIG. 6).[OHl] The early exit router 730 can determine the early exit prediction etfor the current token t (e.g., token ti) at the output of the early exit / -th layer within the set of transformer layers 720. An early exit prediction et~ = 1 can correspond to a prediction that the token ttwill be early exited after the / -th layer, and an early exit prediction et~ = 0 can correspond to a prediction that the token ttwill not be early exited after the / -th layer.

[0112] An early exit prediction et— 1 for a current token ttcan cause the early exit router 730 to route the hidden state vectors h for the current token ltto the language model head 760. bypassing (e.g.. skipping, early exiting from, etc.) processing of the hidden state vectors hltfor the early exited token ttby the remaining N-i transformer layers that are after the z-th layer (e.g., the (z+l)thtransformer layer 742, an (z+2)thtransformer layer 744,..., and an Vthtransformer layer 758).

[0113] The hidden state vectors htlfor the early exited token ttcan be provided as inputs to one or more KV selection blocks associated with subsequent transformer layers included in the set of N-i transformer layers that are after the early exit / -th layer. For example, the early exit information htletcorresponding to the early exited token ttcan be provided as input to the (z+2)thtransformer layer 744,..., and as input to the Vthtransformer layer 758. The early exit information htletcorresponding to the early exited token ttcan further be provided to the language model head 760. When the token ttis early- exited, the early exit prediction et= 1, and the early exit information htet= htl. When the token ttis not early exited, the early exit prediction et= 0, and the early exit information htlet~ = 0.

[0114] For KV processing of each respective transformer layer of the machine learningPATENTQualcomm Docket No 2502632WO35architecture 700, the (z+l)thtransformer layer 742 can receive inputs HQ inand HKV inbased on information corresponding to the z-th (e.g., early exit) transformer layer, provided from the early exit router 730 for the KV processing of the remaining layers after the early exit layer z for the early exited token tt. For example, using the hidden state output vectors htl, et— 0 for the token ttfrom the early exit z-th layer (e.g., obtained from the early exit router 730), the (z+l)thlayer 742 can determine the corresponding Houtinformation including hLt+et= 0.

[0115] The hidden state vector output for the z i 7 layer 742 (e.g., h^1 1^ = 0) can be provided to the i+2 layer KV selection block 734, and can additionally be provided as an input to the i+2 transformer layer 744 with the HQ ininput.

[0116] The z +2 transformer layer 744 can obtain the HKV lninput from a respective KV cache for the i+2 transformer layer 744, where the respective KV cache implements and / or is associated with the i+2 layer KV selection block 734. The i+2 layer KV selection block 734 can determine the HKV ininput for the i+2 transformer layer 744 based on and / or using the hidden state vector information h0lfig, h e,etfor the early exit z-th transformer layer from the tokens indexed by t = 0, 1,..., t. and the hidden state vector information hg+1(l — e^), h+1(l — el),..., hf+1(l — e)) for the i+1 transformer layer 742 over the tokens indexed by t = 0, 1,..., t. For example, the HKV ininput for the i+2 transformer layer 744 can comprise h.0l^v in= fto+1(l — e0) +

[0117] When a preceding token (e.g., represented as a token ts, s + t), is an early exited token, the HKV ininput for the i+2 transformer layer 744 comprises the KV cached hshidden state information for the early exited token tswith es= 1, obtained using the i+2 layer KV selection block 734.

[0118] When a preceding token ts, s < t is not an early exited token, the early exit prediction for the preceding token is es= 0, and the HKV ininput for the i+2 transformer layer 744 comprises the KV cached hsl+1hidden state information from the output of the previous, i+1 transformer layer 742.

[0119] When the current token ttis an early exited token, the Hl(V minput for the i+2 transformer layer 744 comprises the KV cached hthidden state information for the earlyPATENTQualcomm Docket No 2502632WO36exited token ttwith et= 1, obtained using the i+2 layer KV selection block 734.

[0120] When the current token tt is not an early exited token et= 0, the HKV ininput for the i+2 transformer layer 744 comprises the KV cached )+1hidden state information from the output of the previous, i+l transformer layer 742.

[0121] Each remaining transformer layer I in the N-i transformer layers after the early exit transformer layer i can use a corresponding KV selection block (e.g., N-th layer KV selection block 738, etc.) to obtain the HKV ininput as the hidden state vector output hslescached for an early exiting token tswith es= 1, or as the hidden state vector output of the previous transformer layer— e / ) + hsles— hs~ l — 0) + 0 = h^1 1for cases where the token tsis a non-early exiting token (e.g., cases where token tsdoes not early exit after layer i) with es= 0. Here, tsmay represent either a preceding token or the current token, e.g. s < t.

[0122] The language model head 760 can generate an output prediction for the next token tt+i based on an input of the cached KV information of the hidden state vectors output for the token ttby the early exiting / -th layer (e.g., the input hltwhen the token ttis predicted to early exit based on et= 1), or based on an input of the hidden state vectors output by the JV-th layer 758 for the token itwhen not early exiting (e.g., the inputwhen the token tt is predicted to not early exit based on et= 0).

[0123] FIG. 8 is a diagram illustrating an example of autoregressive token generation 800 using a plurality of Transformer machine learning layers, in accordance with some examples. In some cases, the autoregressive token generation 800 can be the autoregressive token generation performed by a vanilla transformer machine learning architecture that does not include an early exit router or an early exit transformer layer, and that does not implement difficulty-aware KV selection. In the example of autoregressive token generation 800 by a vanilla transformer, an input embedding includes and / or corresponds to a sequence of 100 prefix tokens (t0, ty,, t99) in an input prompt to the LM or LLM that uses the autoregressive token generation 800.

[0124] In one illustrative example of the autoregressive token generation 800 by the vanilla transformer architecture, to determine the embedding99(e.g., the output embedding of hidden state vectors generated by the Transformer Layer N 858 for thePATENTQualcomm Docket No 2502632WO37token tgg) needed for sampling the next token t100, all N transformer layers of the vanilla transformer architecture must be invoked (e.g., Transformer Layer 1 811. Transformer Layer 2812, Transformer Lay er N-l 857, and Transformer Lay er N 858)to obtain the embeddings {hklI = 1, 2,..., N] for all 99 of the preceding prefix tokens (k = 0, 1, 98). That is, to determine the embedding h99needed to sample the next token tioo, all N transformer layers must be invoked for each different one of the 99 previous tokens k in the prompt sequence (k = 0, 1,..., 98), corresponding to NK = N-99 transformer layer operations over different portions or combinations of the prefix tokens in the prompt.

[0125] In some cases, invoking all N transformer layers to obtain the embeddings (hklI = 1, 2, N for all 99 of the preceding prefix tokens (k = 0, 1,..., 98) corresponds to performing the prefill phase to obtain the resulting keys {kkl; I = 1,2, N and values {vklI — 1,2,...,1V} corresponding to the 99 prefix tokens (k = 0, 1,...,98), where the keys {kkl} and the values {vkl} are computed and stored in the KV cache for each respective layer I, for I = 1, 2,..., N, to be used in the sampling of t100by the language model head 860 and the softmax next token sampling block 870 that generates as output the sampled next token 875 (e g., the sampled next token / Too).

[0126] After the keys {kkl} and the values {vkl] are computed and stored in the KV cache for the 99 preceding prefix tokens (k = 0, 1,..., 98), the KV cached information of the keys {kkl} and the values {v^} at each respective layer I — 1, 2,..., N can be used in the sampling of the next token / Too in conjunction with the keys {k9l9} and the values {V99} corresponding to the current token t99at each respective layer I = 1, 2,..., N.

[0127] The resulting keys from the sampling of the next token two are the keys {kg9; I = 1,2,..., N and values {vl99; I = 1,2,..., 1V} calculated by the plurality of machine learning layers and the respective hidden state outputs {hklI = 1, 2,..., V} for k=99. The resulting keys {kg9; I = 1,2,..., N and values9l9; I = 1,2,..., V} are also added to the KV cache, to be used in the sampling of the next output token t101,..., etc. In some examples, the autoregressive token generation 800 associated with the vanilla transformer implementation can be represented as:PATENTQualcomm Docket No 2502632WO38

[0128] Here, P(t1Oo = t|G,is the output distribution sampled for the next token / Too, given the previous 100 prefix tokens t0,..., t99. The autoregressive token generation 800 corresponding to an example vanilla transformer architecture shown in FIG. 8 can use an amount of computation for sampling the next tokens two, fioi, etc. that is the same as if generating 101 tokens from scratch, which may be inefficient as 100 KV tokens are already available from the KV cache information and / or hidden state outputs associated with the earlier tokens of the prompt sequence.

[0129] FIG. 9 is a diagram illustrating an example of autoregressive token generation 900 using a plurality of Transformer machine learning layers configured for early exit and KV selection, in accordance with some examples. In some cases, the autoregressive token generation 900 can be implemented using a transformer and / or transformer-based machine learning architecture configured with difficulty-aware key-value selection for early exiting, such as the example machine learning architecture 600 of FIG. 6, the example machine learning architecture 700 of FIG. 7, etc.

[0130] In some cases, the input embedding 905 can be the same as or similar to the input embedding 805 of FIG. 8, and for example input embedding 90 may be an input embedding that includes and / or corresponds to a sequence of 100 prefix tokens (t0, G,..., t99) in an input prompt to the LM or LLM that uses the autoregressive token generation 900 of FIG. 9. A plurality of transformer layers in the example of FIG. 9 can include a first Transformer Layer 1 911, a second Transformer Layer 2 912, a third Transformer Layer 3 913,..., an (N-l)-th Transformer Layer 957, and an N-th Transformer Layer 958. The plurality of N transformer layers of FIG. 9 can correspond to one or more of the transformer layers of the preceding FIGS. 1-8. The 1-th Transformer Layer 958 can provide a respective inputof the output hidden state vectors from the 1-th Transformer Layer 958 for each token k = 0, 1,..., 98, 9 of the 100 prefix tokens (t0, G,..., t99) in the input prompt corresponding to the input embedding 905.

[0131] In the example of autoregressive token generation 900 with early exiting, the early exiting layer is configured as the first Transformer Layer 1 911. For example, each respective token tthas a corresponding input token embedding xtin the input embedding 905. The input token embedding xtis processed by the first Transformer Layer 1 911. for each token k = 0, 1,..., 98, 99 of the 100 prefix tokens (t0, G, ■■■> G9) inthe input prompt corresponding to the input embedding 905.PATENTQualcomm Docket No 2502632WO39

[0132] An early exit router (e.g., such as the early exit router 630 of FIG. 6, 730 of FIG.7, etc.) can be provided between the output of Transformer Layer 1 911 (e.g., the early exit layer) and the input of Transformer Layer 2 912 (e.g., the first non-early exit layer after the early exiting layer), and the early exit router can determine and / or generate a corresponding early exit prediction etfor each token k = 0, 1,, 98, 99 of the 100 prefix tokens (t0, tq,..., t99) in the input prompt corresponding to the input embedding 905.

[0133] For example, an early exit prediction of et= 1 corresponds to an early exit prediction of YES, and the respective hidden state vector output h, for each early exited token in the set of tokens k= 0, 1,..., 98 can be provided to the respective KV cache and KV selection block corresponding to the subsequent layers in the plurality of N layers.

[0134] In some aspects, when the early exit layer is the Transformer Layer 1 911, the remaining transformer layers 2-N may each be associated with a respective KV selection block for implementing the prefill and KV processing operations to initialize the KV cache for the particular remaining transformer layer. For example, Transformer Layer 2 912 can be associated with a KV selection block 932 and a corresponding KV cache for Transformer Layer 2. Transformer Layer 3 913 can be associated with a KV selection block 933 and a corresponding KV cache for Transformer Layer 3..., Transformer Layer N-l 957 can be associated with a KV selection block 937 and a corresponding KV cache for Transformer Layer N-l, and Transformer Layer N 958 can be associated with a KV selection block 938 and a corresponding KV cache for Transformer Layer N.

[0135] In an illustrative example, the early exit layer (e.g.. transformer layer associated with an early exit router, etc.) is Transformer Layer 1 911, and tokens t0, t2,..., t96, t98are each early exited after Transformer Layer 1 911. The remaining tokens that are not early exited are the tokens Q and t97. The final input sequence token, t99, is not early exited, nor is the sampled output token t100(e.g., predicted at the output of softmax and next token sampling block 970 as the predicted next output token 975 based on the embedding h99).

[0136] The outputs of hidden state vectors h. from Transformer Layer 1 (e.g., early exit layer) 911 for the prefix tokens k = 0, 1,..., 99 are routed according to the respective early exit prediction etdetermined for each token. The early exited tokens tQ, t2,..., t96, t98have their respective hidden state vector outputsft2, ■■■ > ^96< ^98 generatedPATENTQualcomm Docket No 2502632WO40and / or determined by Transformer Layer 1 (e.g., early exit layer) 911 routed to the respective KV selection blocks 932, 033. 937, and 938 for each of the remaining transformer layers 2-N (e.g., Transformer Layer 2 912, Transformer Layer 3 913,..., Transformer Layer N-l 957, and Transformer Layer N 958) after the early exit Transformer Layer 1.

[0137] The hidden state vectors h^ from Transformer Layer 1 (e.g., early exit layer) 911 corresponding to the non-early exited tokens h and I97 are provided as input to the next Transformer Layer 2 as the respective hidden state vector outputs h and hj7. Using the KV cache and KV selection block 932 for Transformer Layer 2 912, the updated hidden state vector outputs h2and h27are generated by the Transformer Layer 2912 for the non-early exited tokens ti and tg. The updated hidden state vector outputs h and h7from Transformer Layer 2 are output and propagated to the respective KV caches and KV selection blocks 933,..., 937, and 938 corresponding to the remaining Transformer Layer 3 913,..., Transformer Layer N-l 957, and Transformer layer N 958 (respectively).

[0138] The updated hidden state vector outputs h2and h27from Transformer Layer 2 are additionally provided as input to the next Transformer Layer 3 913. Using the KV cache and KV selection block 933 for Transformer Layer 3 913, which includes the hidden state information h8, h,h,...,hg6, hg8and / or h2and h27from the preceding Transformer Layer 1 911 and / or Transformer Layer 2912 (respectively), the Transformer Layer 3913 generates and outputs the updated hidden state vector outputs hl and hl7for the non-early exited tokens ti and tg. The updated hidden state vector outputs h and / tg7can be provided from Transformer Layer 3 913 as input to the next layer (e.g., a transformer layer 4, etc.), and can additionally be output and propagated to the respective KV caches and KV selection blocks of the next transformer layer.

[0139] The Transformer Layer N-l 957 receives as input the updated hidden states for the non-early exited tokens ti and Q? generated as output by the previous transformer layer N-2, for example the two hidden state vector outputs h^~2and h^2, which are also provided to the KV cache and KV selection blocks 937 and 938. Using the KV cache information and KV selection block 937, the Transformer Layer N-l 957 generates the updated hidden state vector outputs and h.^1, which are output to the KV selection block 938 and KV cache for the next transformer layer (e.g., the final transformer layer N 958) and are additionally provided as inputs to the transformer layer N 958. Using thePATENTQualcomm Docket No 2502632WO41KV cache information and KV selection block 938, the transformer Layer N 958 generates the output hidden state vectors h.^ and h97corresponding to the non-early exited tokens ti and Qz. which can be provided to the language model head 960 and to the softmax and next token sampling block 970 for prediction of one or more output tokens 975, etc.

[0140] In some aspects, the difficulty-aware KV selection machine learning architecture used for the autoregressive token generation with early exiting described with respect to the autoregressive token generation 900 of FIG. 9 can reduce the number of token processing operations relative to the transformer architecture used to perform the autoregressive token generation 800 of FIG. 8 with no early exit or di fficul ty-aware KV selection. For example, a token processing operation can correspond to processing one token t by one transformer layer I.

[0141] In the example of the early exiting, difficulty-aware KV selection-based autoregressive token generation 900 of FIG. 9, the 97 early exited tokens t0, t2,..., t96, t98are only processed through one single transformer layer (e.g., based on the early exit layer being configured as the Transformer Layer 1 911), and the non-early exited tokens ty and t97are processed through the N transformer layers, for a total number of token processing operations for the input sequence of the 99 prompt or prefix tokens (e.g., the token sequence t0, t1(t2, t96, t97, t98) equal to 97-1 + 2-V= 2V+97 « 98V, where 98V is the number of token processing operations associated with processing each of the 99 prefix tokens using all V transformer layers, as in the example of the autoregressive token generation 800 of FIG. 8 using a vanilla Transformer machine learning architecture.

[0142] In some aspects, the difficulty-aware KV selection transformer layers can be trained concurrently across the plurality of transformer layers V. For example, for a vocabulary size of AL, the terms lt l, lt 2, ■■■ > lt, Mcanrepresent the logits output of a target model at token position t. where the target model does not use early exiting (e.g., such as for a target model corresponding to the example autoregressive token generation 800 of FIG. 8 without early exiting, etc.). The terms le t 1(Ze 12,, le,t, Mcanrepresent the logits output of the early exit model including an early exit router and KV selection blocks for the transformer layers that are after the early exit layer coupled to the early exit router (e.g., corresponding to the early exit machine learning model architecture 600 of FIG. 6, 700 of FIG. 7, 900 of FIG. 9, etc.).PATENTQualcomm Docket No 2502632WO42

[0143] The loss function for training of the difficulty-aware KV selection transformer layers of the early exit machine learning model can be represented according to:L = WLMHNEq. (14)

[0144] Here, Eq. (14) and Eq. (15) correspond to a loss function for the target model associated with the logits output lt 1(lt 2, ■■■, lt, M °f the target model at token position t. Eq. (16) and Eq. (17) can correspond to a loss function for the early exit model associated with the logits output■■■ > le,t, M of the early exit model including an early exit router and KV selection blocks at token position t.

[0145] A per-token (e.g., per token position t) cross entropy loss can be determined, assuming the label token is k, as the cross entropy loss CEtfor the target model of Eqs. (14)-(15), and as the cross entropy loss ( Vty, for the early exit model of Eqs. (16)-(17):CEt= - Z + log (^=1eh.;) Eq. (18)CEe,t= - le,t,k + logEq. (19)

[0146] An average cross entropy loss can be determined as:

[0147] Here, the average cross entropy loss CE of Eq. (20) is an average cross entropy loss for the target model of Eqs. (14)-(15) and the target model cross entropy loss CEtofPATENTQualcomm Docket No 2502632WO43Eq. (18). The average cross entropy loss CEeof Eq. (21) is an average cross entropy loss for the early exit model of Eqs. (16)-(l 7) and the early exit model c ross entropy loss CEe,tof Eq. (19).

[0148] In some aspects, a multitask learning framework can be used with a composite loss function / .. where:CE I CE

[0149] In some examples, the term — - — - within the composite loss function of Eq.(22) can be used to cause the machine learning system to leam both the full model and the early exit model at the same time. The termcorresponds to a router weight that is used to train an early exit predictor to generate the early exit prediction etfor each respective token tt(e.g., corresponding to the early exit prediction etof the early exit router 630 of FIG. 6 and / or the early exit router 730 of FIG. 7, etc.).

[0150] The early exit router can be trained using a per-token (position z) KL divergence loss between the target mode and the early exit model, given as:

[0151] Early exit labeling-KL divergence criteria can be provided for each token t, based on assigning for each token t a positive early exit label (e.g., et= 1) if KLt< ThKL. and assigning a negative early exit label (e.g., et= 0) if KLt> ThKL. The term ThKLcan represent a tunable hyperparameter configured to control the tradeoff between quality and early exit. An early exit w eighted binary cross entropy loss can be implemented for leXit,t,o and representative of the two output logits of the early exit predictor corresponding to the two possible early exit decisions, and:^e,t 0.9( lexit,t,0 d” log(e log(exp

[0152] Here, the early w eighted binary cross entropy loss of Eq. (24) can give more weight to the negative early exit label to discourage false early exit decision(s).PATENTQualcomm Docket No 2502632WO44

[0153] In some cases, adaptation of a foundation, pre-trained machine learning model can be performed. For example, configuring the early exit prediction to always predict 6i = 0 can correspond to a KV hidden state difficulty-aware selection implemented by the systems and techniques herein that architecturally mirrors a vanilla and / or non-early exiting transformer architecture. In some aspects, a pre-trained foundation model can be annealed to be an early exit-capable machine learning model with an early exit predictor for an early layer. A foundation model and a target early exit model can be selected for the annealing. The target early exit model can correspond to the target early exit layer of the pre-trained foundation model used as an input to the annealing. For example, a foundation model can be a pre-trained model with 70B parameters and 80 layers, and the selection of the target early exit model can correspond to an early exit at layer 8 (e.g., early exit layer 1=1=8). which can provide a reduction in parameters on the order of 7-8B for the early exit model. The selection of foundation model and target early exit model (e.g., target early exit layer) can then be used in combination with a selected scheduler for the hyperparameter ThKLto anneal the pre-trained foundation model to generate the annealed early exit with prediction model. For example, a value of the hyperparameter ThKL= 0 can correspond to adapting the foundation model weights to the combined loss= (EE + CEe) / 2. In some cases, for relatively small values of the hyperparameter ThKL, the updated model weights can begin to be adapted to the combined KV hidden state architecture. Slow increases in the value of the hyperparameter ThKLcan then be used until a desired tradeoff between early exit computational savings and quality difference is achieved.

[0154] As noted above, transformer-based machine learning architectures (e.g., decoder-only and / or autoregressive token generation using an LM and / or LLM, etc.) a prefill phase may be performed to process the input sequence prior to autoregressive token-by-token generation. For example, a machine learning network can include a plurality of transformer layers, where each transformer layer implements an attention mechanism using the output of the previous transformer layer of the machine learning network. During inference, attention is computed over all previous tokens that are before the currently processed token (e.g., for each token, the model attends to all previous tokens). In decoder-only language models (LMs) and / or LLMs, the self-attention block included in each transformer layer may be configured to derive one set of query vectors (Q), key vectors (K), and value vectors (V) for each input token t, from a set of linearPATENTQualcomm Docket No 2502632WO45projections of the corresponding hidden state output from the preceding transformer layer of the LM or LLM machine learning model. When the tokens corresponding to the prompt are presented to the model, all prompt tokens must be processed by the entire decoder (e.g., each layer of a plurality of transformer layers in the decoder architecture) to compute key vectors and value vectors needed by the last input token of the prompt, to thereby perform the next work prediction. Processing all of the preceding input prompt tokens to compute the key vectors and value vectors (e.g., KV vector or tensor, etc.) for the final input prompt token before predicting the next token or word prediction of the output represents a large overhead in processing power during the prefilling phase of KV vectors for the different transformer layers I and tokens t corresponding to the prompt token sequence.

[0155] For example, FIG. 10A is a diagram illustrating an example of a pre-filling technique 1000a for autoregressive token generation in a vanilla Transformer machine learning architecture, in accordance with some examples. In some cases, the pre-filling technique 1000a can correspond to the autoregressive token generation 800 of FIG. 8 for a vanilla transformer machine learning architecture. FIG. 10B is a diagram illustrating an example of a pre-filling technique 1000b for autoregressive token generation in an early exit KV selection Transformer machine learning architecture, in accordance with some examples. For example, the pre-filling technique 1000b can correspond to the autoregressive token generation with early exiting described with respect to the autoregressive token generation 900 of FIG. 9 for a difficulty -aware KV selection-based machine learning architecture, such as the machine learning architecture 600 of FIG. 6, 700 of FIG. 7, etc.

[0156] In the example of the pre-filling technique 1000a for autoregressive token generation in a vanilla transformer architecture without early exiting prediction or routing for KV selection, an input embedding corresponds to 100 prefix tokens (t0, t1;■■■, t99) in an input prompt to the underlying LM or LLM machine learning model using the vanilla transformer architecture. To compute the embedding / i99at the output of the last transformer layer N for the last prefix token / 99 that is needed to sample the next token t100, each of the N transformer layers must be invoked to obtain the embeddings {hkl; I = 1, 2,..., N for each transformer layer I across each of the 99 prefix tokens ( = 0, 1,...,98). The resulting keys {kkl; I = 1,2, N and values {vkl; I = 1,2,..., NPATENTQualcomm Docket No 2502632WO46corresponding to the 99 prefix tokens (k = 0, 1,...,98) are computed and stored in the KV cache, to be used in the sampling of t100. The resulting keys {fcg9; I = 1,2,..., N and values {v; I = 1,2,..., N are also added to the KV cache, to be used in the sampling of t101. In some aspects, the amount of computation for the pre-filling technique 1000a corresponding to the autoregressive token generation in a vanilla transformer architecture is approximately equal to that for generating 101 tokens from scratch, even though 100 tokens are already available.

[0157] In transformer architectures without early exiting and / or difficulty -aware KV selection for early exit layers, each transformer layer of a plurality of transformer layers includes a self-attention block (e.g., such as the MHA attention layer 344 of the transformer block 340 of FIG. 3, etc.). Queries, keys, and values are input vectors used in the self-attention block of each respective transformer layer. The Q, K, and V vectors for the self-attention (e.g., MHA attention layer 344 of FIG. 3, etc.) calculated by the first transformer layer / =1 can be computed using the input embedding 305 of FIG. 3, and do not reference Q, K. and / or V vectors associated with additional transformer layers of the plurality of transformer layers. However, the Q, K, and V vectors used in the remaining self-attention blocks for each transformer layer after the first layer (e.g., each transformer layer / >1) are computed using the output hidden state vectors from the preceding transformer layer, for each token of an input token sequence to, ti, tr. For example, the output hidden state vectors hlkcan be the respective hidden state vectors output by the transformer layer i for token k. The hidden state vectors hklcan be used to compute the queries, keys, and values (e.g., Q, K, and V vectors) for the self-attention calculation of the MHA attention layer 344 of the next transformer layer i+1. For example, the next transformer layer i+1 can include an RMS Norm (e.g., such as the RMS Norm 342 of FIG. 3, etc.) that receives as input the output hidden state vectors hklfrom the preceding transformer layer i. and generates the computed queries (e.g., Q vector(s)), keys (e.g., K vector(s)), and values (e.g., V vector(s)) for the self-attention computation by the MHA attention layer 344 of the next transformer layer i-1.

[0158] In some aspects, the systems and techniques provided for difficulty-aware KV selection for early exit from a plurality of transformer layers associated with a machine learning model (e g., LM and / or LLM, etc.) can be configured with a first set of transformer layers that use self-attention for the attention calculation for each transformerPATENTQualcomm Docket No 2502632WO47layer up to and including the early exit layer i. Each of the subsequent N-i transformer layers after the early exit layer i can be implemented as a transformer layer configured to calculate cross-attention. For example, the keys and values (e.g., KV vectors, tensors, etc.) used in the cross-attention calculation of the subsequent N-i transformer layers after the early exit layer i can be calculated using the output hidden state vectors hklfrom the z-th transformer layer (e.g., the early exit transformer layer, which is the last transformer layer to perform a self-attention calculation).

[0159] For example, the first set of z transformer layers 620 of FIG. 6 can be implemented as transformer layers configured to calculate self-attention over Q, K, V vectors corresponding to output hidden state vectors from the preceding transformer layer in the set of transformer layers 620. The last transformer layer z in the set of transformer layers 620 of FIG. 6 is the early exit transformer layer, which can early exit based on an early exit prediction for the output hidden state vectors hklfor the self-attention calculation performed by the early exit transformer layer z. The second set of the remaining, N-i transformer layers 650 of FIG. 6 can be implemented as transformer layers configured to calculate cross-attention using the output hidden state vectors from the early exit transformer layer z (e.g., configured to calculate cross-attention using the output hidden state vectors hkl, which can be stored in the KV cache and / or KV selection block configured for each of the remaining N-i transformer layers 650 that are configured to implement cross-attention. For example, the first set of z transformer layers 620 can implement self- attend on based on obtaining and / or determining the Q, K, V vectors all from the same source, which is the input to each transformer layer in the first set of transformer layers 620 comprising the hidden state vectors output by the preceding transformer layer. Each token in the input sequence for self-attention attends to all preceding tokens in the same sequence. The second set of N-i transformer layers 650 can implement cross-attention based on obtaining Q from a first source (e.g., decoder hidden states), and obtaining KV from a second source (e.g., the output hidden state vectors hklin the KV cache, stored after the early exiting of the z-th layer).

[0160] For example, the pre-filling technique 1000b of FIG. 10B includes a first set of 4 transformer layers with self-attention for z=4, which can correspond to the first set of z transformer layers 620 of FIG. 6 and / or the first set of z transformer layers 720 of FIG. 7, etc. A second set of (AM) transformer layers with cross-attention can correspond to thePATENTQualcomm Docket No 2502632WO48second set of A- / transformer layers 650 of FIG. 6 and / or the second set of N-i transformer layers (e.g., transformer layer i+1 742, transformer layer i+2744,..., transformer layer A 758) of FIG. 7, etc.

[0161] In some examples of pre-filling for autoregressive generation associated with a KV transformer, the input embedding for the pre-filling technique 1000b of FIG. 10B can correspond to 100 prefix tokens (t0, t1(, t99) in the prompt, where each input prompt token tkis mapped to an input embedding xkfor all k. A number of transformer layers with self-attention can be set as z=4. index k.

[0162] To compute the hidden state embeddings { I99; I = 1,2,..., N for the last prefix token tgg, which are needed to sample the next token t100, the pre-filling technique 1000b of FIG. 10B can be performed to include a first step of computing {7i9g; I = 1, 2,..., 4}, where only the first i = 4 transformer layers are invoked to obtain the embeddings {hkl; Z = 1, 2,..., 4} for all the 100 prefix tokens (k = 0, 1,...,99), for computing the keys and values needed in the transformer self-attention layers.

[0163] In a second step of the example pre-filling technique 1000b for autoregressive generation associated with a KV transformer, to compute {h99; Z = 5, 10,..., ZV}, {hk; k = 0, 1,...,99} obtained in the first step can be used to compute the respective keys and values in the cross-attention layers of the subsequent N-4 transformer layers configured with cross-attention. Unlike in the vanilla transformer implementation and pre-filling technique 1000a of FIG. 10A, the example pre-filling technique 1000b of FIG.10B can skip the computation of {hklI = 9, 10,..., N for the first 99 prefix tokens.

[0164] In some aspects, the systems and techniques for difficulty-aware KV selection for early exit of a machine learning network can correspond to a reduction in the computational cost and time associated with prefilling for KV cache and / or KV processing across a plurality of transformer layers of the machine learning network. For example, as computing the key vectors {kkl} and the value vectors {vkl} for the transformer layers after the early exit layer i (e.g., the second set of transformer layers configured with cross-attention, comprising the transformer layers I G {Z + 1, Z + 2,..., ZV} with Z « N ) only requires the cached hidden state embeddings hkgenerated as output from the early exit layer i, as the input for determining the respective KV vectors at each subsequent transformer layer after the early exit layer Z, the input tokens in thePATENTQualcomm Docket No 2502632WO49prompt can skip most of the processing applied in each layer I after the early exit layer i.

[0165] For example, processing for the transformer layers after the early exit layer i (e.g., processing for the transformer layers configured with cross-attention using the cached hidden state embeddings hklfrom the early exit layer z) can skip the processing steps at each subsequent layer that are associated with the two normalization modules per layer, the WQ projection applied for computing query vectors Q in the MHA attention per layer, the softmax V1computation for each head in the MHA attention per layer,the W0lprojection applied in the MHA attention for combining the results from the multiple heads per layer, and / or the position-wise Multiplayer Perceptron MLP applied at the output per layer. In some aspects, the systems and techniques can provide further increased prefill efficiency improvements for input prompts that are relatively long, and / or when group query attention is used. In group query attention, multiple attention heads of an MHA computation for a transformer layer and / or transformer block can share the same K and V vectors. For example, a model may have 32 attention heads and a number of KV groups equal to 8, with 32 sets of Q vectors (e.g., one Q vector per attention head). For the 8 KV groups, there exist a correspond 8 pairs of KV vectors, and the group query attention can be configured for every 4 attention heads (e.g., out of the total of 32 attention heads) to share the same pair of KV vectors selected out of the 8 pairs of KV vectors available.

[0166] FIG. 11 is a flowchart diagram illustrating an example of a process 1100 that can be used for difficult -aware key -value selection for early exit of a machine learning network including a plurality of transformer layers and / or attention calculation blocks. Although the example process 1100 depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the process 1100. In other examples, different components of an example device or system that implements the process 1100 may perform functions at substantially the same time or in a specific sequence.

[0167] In some examples, the process 1100 can be performed by a computing device or apparatus or a component or system (e.g., one or more chipsets, one or more processorsPATENTQualcomm Docket No 2502632WO50such as one or more CPUs, DSPs, NPUs, NSPs, microcontrollers, ASICs, FPGAs, programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc., any combination thereof, and / or other component or system) of the computing device or apparatus. The operations of the process 1100 may be implemented as software components that are executed and run on one or more processors (e.g., processor 1510 of FIG. 15 or other processor(s)). In some examples, the process 1100 can be performed by a machine learning network, including any of the machine learning networks and / or neural networks described herein, etc. In some aspects, the process 1100 can be performed by a UE, smartphone, mobile computing device, user computing device, etc. The process 1100 may be performed by an apparatus that may be a mobile device (e.g., a mobile phone), a network-connected wearable such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or augmented reality (AR) device, a vehicle or component or system of a vehicle, or other type of computing device. The operations of the process 1100 may be implemented as software components that are executed and run on one or more processors (e.g., processor 1510 of FIG. 15, and / or other processor(s)).

[0168] At block 1102, the apparatus (or component thereof) can obtain a sequence of a plurality of tokens corresponding to an input to a machine learning network including a plurality of transformer layers, the plurality of transformer layers including a first subset of layers and a second subset of layers, wherein the first subset of layers includes a configured early exit layer. For example, the plurality of transformer layers can correspond to the one or more transformer layers associated with the Transformer-based machine learning architecture 300 of FIG. 3, and / or the first set of layers 310, the layer / 320, the layer l+l 330, and / or the set of layers 350, etc., of FIG. 3. In some cases, the plurality of transformer layers can correspond to one or more of the transformer blocks 430-1, 430-2, 430-3 of FIG. 4. In some cases, the plurality of transformer layers can correspond to one or more of the set of transformer layers 620 of FIG. 6, the set of transformer layers 650 of FIG. 6, the transformer layers of FIG. 7, the transformer layers of FIG. 8, the transformer layers of FIG. 9, the transformer layers of FIG. 10A and / or FIG. 10B, etc.

[0169] In some examples, the first subset of layers comprises a first subset of transformer layers of the plurality of transformer layers, and the second subset of layersPATENTQualcomm Docket No 2502632WO51comprises a second subset of transformer layers of the plurality of transformer layers. For example, the first subset of layers can correspond to the set of layers 310 and the layer 320 of FIG. 3, and the second subset of layers can correspond to the set of layers 350 of FIG. 3. In some examples, the first subset of layers can correspond to the set of layers 620 of FIG. 6, and the second subset of layers can correspond to the set of layers 650 of FIG.6. In some examples, the first subset of layers can correspond to the set of layers 720 of FIG. 7, and the second subset of layers can correspond to the layers 742, 744,..., 758 of FIG. 7. In some cases, the first subset of layers can correspond to a first subset of the layers of FIG. 8, and the second subset of layers can correspond to a second subset of the layers of FIG. 8. In some examples, the first subset of layers can correspond to a first subset of the layers of FIG. 9, and the second subset of layers can correspond to a second subset of the layers of FIG. 9, etc. In some cases, the first subset of transformer layers can be configured to determine self-attention, and the second subset of transformer layers can be configured to determine cross attention.

[0170] In some examples, the configured early exit layer can be a last layer of the first subset of layers. In some cases, the configured early exit layer can correspond to the layer 320 of FIG. 3. In some cases, the configured early exit layer can correspond to a last layer of the set of transformer layers 620 of FIG. 6, and / or may be associated with the early exit router 630 of FIG. 6. In some cases, the configured early exit layer can correspond to a last layer of the set of transformer layers 720 of FIG. 7, and / or can be associated with the early exit router 730 of FIG. 7.

[0171] At block 1104, the apparatus (or component thereof) can determine an early exit prediction for a respective token included in the sequence of the plurality of tokens, the early exit prediction determined based on a hidden state vector generated as output for the respective token by the configured early exit layer. A first value of the early exit prediction corresponds to skipping processing of the hidden state vector for the respective token by the second subset of layers. For example, the first value of the early exit prediction can cause at least one processor of the apparatus (or component thereof) to skip processing of the hidden state vector for the respective token by the second subset of layers.

[0172] For example, the early exit prediction can be determined based on a hidden state vector generated as output for the respective token by the configured early exit layerPATENTQualcomm Docket No 2502632WO52comprising a last layer of the first subset of layers, and / or the configured early exit layer corresponding to the layer 320 of FIG. 3, and / or the configured early exit layer corresponding to a last layer of the set of transformer layers 620 of FIG. 6, and / or the configured early exit layer corresponding to a last layer of the set of transformer layers 720 of FIG. 7. In some examples, the early exit prediction can be determined by the early exit router 630 of FIG. 6. In some cases, the early exit prediction can be determined by the early exit router 730 of FIG. 7. In some cases, the first subset of transformer layers can be configured to determine self-attention for the respective token. In some examples, the hidden state vector for the respective token can be generated as a self-attention output associated with a self-attention calculated for the respective token by the configured early exit layer.

[0173] In some cases, to skip processing of the hidden state vector for the respective token by the second subset of layers, the first value of the early exit prediction further causes the apparatus (or component thereof) to provide the hidden state vector generated as output for the respective token by the configured early exit layer to a language model head connected to an output of a final transformer layer of the second subset of layers. For example, the language model head can correspond to one or more of the language model head 360 of FIG. 3, the language model head 660 of FIG. 6, the language model head 760 of FIG. 7, the language model head 860 of FIG. 8, and / or the language model head 960 of FIG. 9, etc.

[0174] In some cases, the first value of the early exit prediction further causes the apparatus (or component thereof) to determine sampling information corresponding to a predicted next token for the sequence of the plurality of tokens using the hidden state vector generated as output for the respective token by the configured early exit layer. For example, the sampling information corresponding to the predicted next token can correspond to one or more of the corresponding output 375 of FIG. 3, the next token sampling information 675 (e.g., a next token probability or prediction) of FIG. 6, the sampled next token 875 of FIG. 8, and / or the predicted next output token 975 of FIG. 9, etc.

[0175] In some cases, the early exit prediction can be determined using an early exit router connected between the first and second subset of layers, for example the early exit router 630 of FIG. 6 and / or the early exit router 730 of FIG. 7, etc. In some cases, thePATENTQualcomm Docket No 2502632WO53early exit router may be connected between an output of the configured early exit layer included in the first subset of layers, and an input of a first transformer layer included in the second subset of layers. In some examples, a second value of the early exit prediction causes the apparatus (or component thereof) to not skip processing of the hidden state vector for the respective token by the second subset of layers. For example, based on the second value of the early exit prediction, KV selection information for a respective transformer layer of the second subset of layers can comprise a hidden state vector generated as output by a preceding transformer layer connected to the respective transformer layer, the preceding transformer layer included in the second subset of layers.

[0176] At block 1106, the apparatus (or component thereof) can determine, based on the early exit prediction for the respective token, key-value (KV) selection information corresponding to one or more transformer layers of the second subset of layers and a subsequent token included in the sequence of the plurality of tokens after the respective token, wherein the KV selection information comprises the hidden state vector generated as output for the respective token by the configured early exit layer, based on the first value of the early exit prediction.

[0177] In some examples, the KV selection information can be associated with a KV cache, such as the KV cache 425-1, the KV cache 425-2. and / or the KV cache 425-3 of FIG. 4. In some cases, the KV selection information can correspond to one or more of the layer KV selection block 734 and / or the N-th layer KV selection block 738 of FIG. 7, etc. In some examples, the KV selection information can correspond to one or more of the KV selection block 932, the KV selection block 933, the KV selection block 937, and / or the KV selection block 938 of FIG. 9, etc.

[0178] In some cases, the first subset of transformer layers can be configured to determine self-attention for the respective token, and the second subset of transformer layers can be configured to determine cross-attention for the subsequent token based on the first value of the early exit prediction for the respective token and using the KV selection information. In some cases, the cross-attention for the subsequent token can be determined based on using the KV selection information to determine a respective keys (K) vector input and a respective values (V) vector input for each transformer layer of the second subset of transformer layers. In some cases, the respective K vector input and the respective V vector input can be determined from a cached representation of the hiddenPATENTQualcomm Docket No 2502632WO54state vector generated as output for the respective token by the configured early exit layer.

[0179] In some examples, the apparatus (or component thereof) can be configured to store the hidden state vector generated as output for the respective token by the configured early exit layer within a respective KV cache associated with each transformer layer of the second subset of layers. The apparatus (or component thereof) can be configured to process the subsequent token the second subset of layers, where each transformer layer of the second subset of layers is configured to determine cross-attention for the subsequent token using the hidden state vector stored within the respective KV cache associated with each transformer layer of the second subset of layers. In some examples, the cross-attention for the subsequent token can be determined using a query (Q) vector associated with the subsequent token, and a KV vector determined based on the hidden state vector generated as output for the respective token by the configured early exit layer and stored within the respective KV cache associated with each transformer layer of the second subset of layers.

[0180] In some examples, the KV selection information can correspond to a token generation phase to configure KV vectors within a respective KV cache associated with each transformer layer of the second subset of layers. In some examples, the configured early exit layer is associated with an early exit router configured to determine the early exit prediction based on the hidden state vector generated as output for the respective token by the configured early exit layer.

[0181] In some examples, the processes described herein (e.g., the process 1100 and / or any other process described herein) may be performed by a computing device or apparatus. In some aspects, the process 1100 and / or other technique or process described herein can be performed by a computing system having an architecture according to any of FIGS. 1-10 and / or FIGS. 12-15. In another example, the process 1100 and / or other technique or process described herein can be performed by the computing device architecture 1500 shown in FIG. 15. In some examples, the computing device can include a mobile device (e.g., a mobile phone, a tablet computing device, etc.), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television, a vehicle (or a computing device of a vehicle), robotic device, and / or any other computing device with the resource capabilities to perform the processesPATENTQualcomm Docket No 2502632WO55described herein.

[0182] In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more transmitters, receivers or combined transmitter-receivers (e.g., referred to as transceivers), one or more cameras, one or more sensors, and / or other component(s) that are configured to carry out the steps of processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive the data, any combination thereof, and / or other component(s). The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other type of data.

[0183] The components of the computing device can be implemented in circuitry. For example, the components can include and / or can be implemented using electronic circuits or other electronic hardware, which can include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), neural processing units (NPUs), and / or other suitable electronic circuits), and / or can include and / or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein.

[0184] The processes described herein may be illustrated or described as a logical flow diagrams, the operation of which represents a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the processes.

[0185] Additionally, the processes described herein may be performed under the controlPATENTQualcomm Docket No 2502632WO56of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0186] As noted above, neural networks may be designed with a variety of connectivity patterns. In feed-forward networks, information is passed from lower to higher layers, with each neuron in a given layer communicating to neurons in higher layers. A hierarchical representation may be built up in successive layers of a feed-forward network, as described above. Neural networks may also have recurrent or feedback (also called top-down) connections. In a recurrent connection, the output from a neuron in a given layer may be communicated to another neuron in the same layer. A recurrent architecture may be helpful in recognizing patterns that span more than one of the input data chunks that are delivered to the neural network in a sequence. A connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. A network with many feedback connections may be helpful when the recognition of a high-level concept may aid in discriminating the particular low-level features of an input.

[0187] FIG. 12 is an illustrative example of a deep learning neural network 1200. An input layer 1220 includes input data. In some cases, the input layer 1220 can include data representing the pixels of an input video frame. The neural network 1200 includes multiple hidden layers 1222a, 1222b, through 1222n. The hidden layers 1222a, 1222b, through 1222n include “n’‘ number of hidden layers, where “n” is an integer greater than or equal to one. The number of hidden layers can be made to include as many layers as needed for the given application. The neural network 1200 further includes an output layer 1224 that provides an output resulting from the processing performed by the hidden layers 1222a, 1222b, through 1222n. In some aspects, the output layer 1224 can provide a classification for an object in an input video frame. The classification can include a class identify ing the type of object (e.g.. a person, a dog, a cat, or other object).PATENTQualcomm Docket No 2502632WO57

[0188] The neural network 1200 is a multi-layer neural network of interconnected nodes. Each node can represent a piece of information. Information associated with the nodes is shared among the different layers and each layer retains information as information is processed. In some cases, the neural network 1200 can include a feedforward network, in which case there are no feedback connections where outputs of the network are fed back into itself. In some cases, the neural network 1200 can include a recurrent neural network, which can have loops that allow information to be carried across nodes while reading in input.

[0189] Information can be exchanged between nodes through node-to-node interconnections between the various layers. Nodes of the input layer 1220 can activate a set of nodes in the first hidden layer 1222a. For example, as shown, each of the input nodes of the input layer 1220 is connected to each of the nodes of the first hidden layer 1222a. The nodes of the hidden layers 1222a, 1222b, through 1222n can transform the information of each input node by applying activation functions to the information. The information derived from the transformation can then be passed to and can activate the nodes of the next hidden layer 1222b, which can perform their own designated functions. Example functions include convolutional, up-sampling, data transformation, and / or any other suitable functions. The output of the hidden layer 1222b can then activate nodes of the next hidden layer, and so on. The output of the last hidden layer 1222n can activate one or more nodes of the output layer 1224, at which an output is provided. In some cases, while nodes (e.g., node 1226) in the neural network 1200 are shown as having multiple output lines, a node has a single output and all lines shown as being output from a node represent the same output value.

[0190] In some cases, each node or interconnection between nodes can have a weight that is a set of parameters derived from the training of the neural network 1200. Once the neural network 1200 is trained, the neural network 1200 may be referred to as a trained neural network, which can be used to classify one or more objects. For example, an interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnection can have a tunable numeric w eight that can be tuned (e.g., based on a training dataset), allowing the neural network 1200 to be adaptive to inputs and able to learn as more and more data is processed.PATENTQualcomm Docket No 2502632WO58

[0191] The neural network 1200 is pre-trained to process the features from the data in the input layer 1220 using the different hidden layers 1222a, 1222b, through 1222n in order to provide the output through the output layer 1224. In an example in which the neural network 1200 is used to identify objects in images, the neural network 1200 can be trained using training data that includes both images and labels. For instance, training images can be input into the network, with each training image having a label indicating the classes of the one or more objects in each image (basically, indicating to the network what the objects are and what features they have). In some examples, a training image can include an image of a number 2, in which case the label for the image can be [00 1 0 000 000],

[0192] In some cases, the neural network 1200 can adjust the weights of the nodes using a training process called backpropagation. Backpropagation can include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update is performed for one training iteration. The process can be repeated for a certain number of iterations for each set of training images until the neural network 1200 is trained well enough so that the weights of the layers are accurately tuned.

[0193] For the example of identifying objects in images, the forward pass can include passing a training image through the neural network 1200. The weights are initially randomized before the neural network 1200 is trained. The image can include, for example, an array of numbers representing the pixels of the image. Each number in the array can include a value from 0 to 255 describing the pixel intensity at that position in the array. In some examples, the array can include a 28 x 28 x 3 array of numbers with 28 row s and 28 columns of pixels and 3 color components (such as red, green, and blue, or luma and two chroma components, or the like).

[0194] For a first training iteration for the neural network 1200, the output will likely include values that do not give preference to any particular class due to the weights being randomly selected at initialization. For example, if the output is a vector with probabilities that the object includes different classes, the probability value for each of the different classes may be equal or at least very similar (e.g., for ten possible classes, each class may- have a probability value of 0.1). With the initial weights, the neural network 1200 is unable to determine low level features and thus cannot make an accurate determinationPATENTQualcomm Docket No 2502632WO59of what the classification of the object might be. A loss function can be used to analyze error in the output. Any suitable loss function definition can be used. An example of a loss function includes a mean squared error (MSE). The MSE is defined as Etotal=— output)2, which calculates the sum of one-half times a ground truthoutput (e.g., the actual answer) minus the predicted output (e.g., the predicted answer) squared. The loss can be set to be equal to the value of Etotal.

[0195] The loss (or error) will be high for the first training images since the actual values will be much different than the predicted output. The goal of training is to minimize the amount of loss so that the predicted output is the same as the training label. The neural network 1200 can perform a backward pass by determining which inputs (weights) most contributed to the loss of the network, and can adjust the weights so that the loss decreases and is eventually minimized.

[0196] A derivative of the loss with respect to the weights (denoted as dLldW, where W are the weights at a particular layer) can be computed to determine the weights that contributed most to the loss of the network. After the derivative is computed, a weight update can be performed by updating all the weights of the filters. For example, the weights can be updated so that they change in the opposite direction of the gradient. Theweight update can be denoted as w =where w denotes a weight, ir denotes the initial weight, and r] denotes a learning rate. The learning rate can be set to any suitable value, with a high learning rate including larger weight updates and a lower value indicating smaller weight updates.

[0197] The neural network 1200 can include any suitable deep network. An example includes a convolutional neural network (CNN), which includes an input layer and an output layer, with multiple hidden layers between the input and out layers. An example of a CNN is described below with respect to FIG. 13. The hidden layers of a CNN include a series of convolutional, nonlinear, pooling (for downsampling), and fully connected layers. The neural network 1200 can include any other deep network other than a CNN, such as an autoencoder, a deep belief nets (DBNs), a Recurrent Neural Networks (RNNs), among others.

[0198] FIG. 13 is an illustrative example of a convolutional neural network 1300 (CNN 1300). The input layer 1320 of the CNN 1300 includes data representing an image. ForPATENTQualcomm Docket No 2502632WO60example, the data can include an array of numbers representing the pixels of the image, with each number in the array including a value from 0 to 255 describing the pixel intensity at that position in the array. Using the previous example from above, the array can include a 28 x 28 x 3 array of numbers with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or luma and two chroma components, or the like). The image can be passed through a convolutional hidden layer 1322a, an optional non-linear activation layer, a pooling hidden layer 1322b, and fully connected hidden layers 1322c to get an output at the output layer 1324. While only one of each hidden layer is shown in FIG. 13, one of ordinary skill will appreciate that multiple convolutional hidden layers, non-linear layers, pooling hidden layers, and / or fully connected layers can be included in the CNN 1300. As previously described, the output can indicate a single class of an object or can include a probability of classes that best describe the object in the image.

[0199] The first layer of the CNN 1300 is the convolutional hidden layer 1322a. The convolutional hidden layer 1322a analyzes the image data of the input layer 1320. Each node of the convolutional hidden layer 1322a is connected to a region of nodes (pixels) of the input image called a receptive field. The convolutional hidden layer 1322a can be considered as one or more filters (each filter corresponding to a different activation or feature map), with each convolutional iteration of a filter being a node or neuron of the convolutional hidden layer 1322a. For example, the region of the input image that a filter covers at each convolutional iteration would be the receptive field for the filter. In some aspects, if the input image includes a 28x28 array, and each filter (and corresponding receptive field) is a 5x5 array, then there will be 24x24 nodes in the convolutional hidden layer 1322a. Each connection between a node and a receptive field for that node learns a weight and, in some cases, an overall bias such that each node learns to analyze its particular local receptive field in the input image. Each node of the hidden layer 1322a will have the same weights and bias (called a shared weight and a shared bias). For example, the filter has an array of weights (numbers) and the same depth as the input. A filter will have a depth of 3 for the video frame example (according to three color components of the input image). An illustrative example size of the filter array is 5 x 5 x 3. corresponding to a size of the receptive field of a node.PATENTQualcomm Docket No 2502632WO61

[0200] The convolutional nature of the convolutional hidden layer 1322a is due to each node of the convolutional layer being applied to its corresponding receptive field. For example, a filter of the convolutional hidden layer 1322a can begin in the top-left comer of the input image array and can convolve around the input image. As noted above, each convolutional iteration of the filter can be considered a node or neuron of the convolutional hidden layer 1322a. At each convolutional iteration, the values of the filter are multiplied with a corresponding number of the original pixel values of the image (e.g., the 5x5 filter array is multiplied by a 5x5 array of input pixel values at the top-left comer of the input image array). The multiplications from each convolutional iteration can be summed together to obtain a total sum for that iteration or node. The process is next continued at a next location in the input image according to the receptive field of a next node in the convolutional hidden layer 1322a.

[0201] For example, a filter can be moved by a step amount to the next receptive field. The step amount can be set to 1 or other suitable amount. For example, if the step amount is set to 1, the filter will be moved to the right by 1 pixel at each convolutional iteration. Processing the filter at each unique location of the input volume produces a number representing the filter results for that location, resulting in a total sum value being determined for each node of the convolutional hidden layer 1322a.

[0202] The mapping from the input layer to the convolutional hidden layer 1322a is referred to as an activation map (or feature map). The activation map includes a value for each node representing the filter results at each locations of the input volume. The activation map can include an array that includes the various total sum values resulting from each iteration of the filter on the input volume. For example, the activation map will include a 24 x 24 array if a 5 x 5 filter is applied to each pixel (a step amount of 1) of a 28 x 28 input image. The convolutional hidden layer 1322a can include several activation maps in order to identify multiple features in an image. The example shown in FIG. 13 includes three activation maps. Using three activation maps, the convolutional hidden layer 1322a can detect three different kinds of features, with each feature being detectable across the entire image.

[0203] In some examples, a non-linear hidden layer can be applied after the convolutional hidden layer 1322a. The non-linear layer can be used to introduce nonlinearity to a system that has been computing linear operations. An example of a non-PATENTQualcomm Docket No 2502632WO62linear layer is a rectified linear unit (ReLU) layer. A ReLU layer can apply the function f(x) = max((). x) to all of the values in the input volume, which changes all the negative activations to 0. The ReLU can thus increase the non-linear properties of the CNN 1300 without affecting the receptive fields of the convolutional hidden layer 1322a.

[0204] The pooling hidden layer 1322b can be applied after the convolutional hidden layer 1322a (and after the non-linear hidden layer when used). The pooling hidden layer 1322b is used to simplify the information in the output from the convolutional hidden layer 1322a. For example, the pooling hidden layer 1322b can take each activation map output from the convolutional hidden layer 1322a and generates a condensed activation map (or feature map) using a pooling function. Max-pooling is an example of a function performed by a pooling hidden layer. Other forms of pooling functions be used by the pooling hidden layer 1322a, such as average pooling, L2-norm pooling, or other suitable pooling functions. A pooling function (e.g., a max-pooling filter, an L2-norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 1322a. In the example show n in FIG. 13, three pooling filters are used for the three activation maps in the convolutional hidden layer 1322a.

[0205] In some examples, max-pooling can be used by applying a max-pooling filter (e.g., having a size of 2x2) with a step amount (e.g.. equal to a dimension of the filter, such as a step amount of 2) to an activation map output from the convolutional hidden layer 1322a. The output from a max-pooling filter includes the maximum number in every sub-region that the filter convolves around. Using a 2x2 filter as an example, each unit in the pooling layer can summarize a region of 2x2 nodes in the previous layer (with each node being a value in the activation map). For example, four values (nodes) in an activation map will be analyzed by a 2x2 max -pooling filter at each iteration of the filter, with the maximum value from the four values being output as the “max” value. If such a max-pooling filter is applied to an activation filter from the convolutional hidden layer 1322a having a dimension of 24x24 nodes, the output from the pooling hidden layer 1322b will be an array of 12x12 nodes.

[0206] In some examples, an L2-norm pooling filter could also be used. The L2-norm pooling filter includes computing the square root of the sum of the squares of the values in the 2x2 region (or other suitable region) of an activation map (instead of computingPATENTQualcomm Docket No 2502632WO63the maximum values as is done in max -pooling), and using the computed values as an output.

[0207] Intuitively, the pooling function (e g., max-pooling, L2-norm pooling, or other pooling function) determines whether a given feature is found anywhere in a region of the image, and discards the exact positional information, for example without affecting results of the feature detection because, once a feature has been found, the exact location of the feature is not as important as its approximate location relative to other features. Max-pooling (as well as other pooling methods) offer the benefit that there are many fewer pooled features, thus reducing the number of parameters needed in later layers of the CNN 1300.

[0208] The final layer of connections in the network is a fully-connected layer that connects every node from the pooling hidden layer 1322b to every one of the output nodes in the output layer 1324. Using the example above, the input layer includes 28 x 28 nodes encoding the pixel intensities of the input image, the convolutional hidden layer 1322a includes 3 *24*24 hidden feature nodes based on application of a 5*5 local receptive field (for the filters) to three activation maps, and the pooling layer 1322b includes a layer of 3*12*12 hidden feature nodes based on application of max-pooling filter to 2*2 regions across each of the three feature maps. Extending the example, the output layer 1324 can include ten output nodes. In such an example, every node of the 3x12x12 pooling hidden layer 1322b is connected to every node of the output layer 1324.

[0209] The fully connected layer 1322c can obtain the output of the previous pooling layer 1322b (which should represent the activation maps of high-level features) and determines the features that most correlate to a particular class. For example, the fully connected layer 1322c layer can determine the high-level features that most strongly correlate to a particular class, and can include weights (nodes) for the high-level features. A product can be computed between the weights of the fully connected layer 1322c and the pooling hidden layer 1322b to obtain probabilities for the different classes. For example, if the CNN 1300 is being used to predict that an object in a video frame is a person, high values will be present in the activation maps that represent high-level features of people (e.g., two legs are present, a face is present at the top of the object, two eyes are present at the top left and top right of the face, a nose is present in the middle ofPATENTQualcomm Docket No 2502632WO64the face, a mouth is present at the bottom of the face, and / or other features common for a person).

[0210] In some examples, the output from the output layer 1324 can include an M-dimensional vector (in the prior example. M=10), where M can include the number of classes that the program has to choose from when classifying the object in the image. Other example outputs can also be provided. Each number in the N-dimensional vector can represent the probability the object is of a certain class. In some cases, if a 10-dimensional output vector represents ten different classes of obj ects is [000.05 0.800.15 000 0], the vector indicates that there is a 5% probability that the image is the third class of object (e.g., a dog), an 80% probability that the image is the fourth class of object (e.g., a human), and a 15% probability that the image is the sixth class of object (e.g., a kangaroo). The probability for a class can be considered a confidence level that the object is part of that class.

[0211] One type of convolutional neural network is a deep convolutional network (DCN).

[0212] A DCN may be trained with supervised learning. During training, the DCN may be presented with an image or other input, and a forward pass may then be computed to produce an output. The DCN may include a feature extraction section and a classification section. Upon receiving the image (or other input), a convolutional layer may apply convolutional kernels to the image to generate a first set of feature maps. As an example, the convolutional kernel for the convolutional layer may be a 5x5 kernel that generates 28x28 feature maps. In the present example, because four different feature maps are generated in the first set of feature maps, four different convolutional kernels were applied to the image at the convolutional layer. The convolutional kernels may also be referred to as filters or convolutional filters.

[0213] The first set of feature maps may be subsampled by a max pooling layer to generate a second set of feature maps. The max pooling layer reduces the size of the first set of feature maps. That is, a size of the second set of feature maps, such as 14x14, is less than the size of the first set of feature maps, such as 28x28. The reduced size provides similar information to a subsequent layer while reducing memory consumption. The second set of feature maps may be further convolved via one or more subsequentPATENTQualcomm Docket No 2502632WO65convolutional layers (not show n) to generate one or more subsequent sets of feature maps (not shown).

[0214] The second set of feature maps can be convolved to generate a first feature vector. Furthermore, the first feature vector is further convolved to generate a second feature vector. Each feature of the second feature vector may include a number that corresponds to a possible feature of the image. A softmax function may convert the numbers in the second feature vector to a probability'. As such, an output of the DCN is a probability of the image including one or more features. Before training, the output produced by the DCN may be incorrect. Thus, an error may be calculated between the output and a target output. The target output is the ground truth of the image features that are extracted or extractable by the DCN. The weights of the DCN may then be adjusted so the output of the DCN is more closely aligned with the target output. To adjust the weights, a learning algorithm may compute a gradient vector for the weights. The gradient may indicate an amount that an error would increase or decrease if the weight were adjusted. At the top layer, the gradient may correspond directly to the value of a weight connecting an activated neuron in the penultimate layer and a neuron in the output layer. In lower layers, the gradient may depend on the value of the weights and on the computed error gradients of the higher layers. The weights may then be adjusted to reduce the error. The manner of adjusting the w eights as noted in the example(s) above may be referred to as “back propagation,” for example based on involving a "backward pass” through the neural network. In practice, the error gradient of weights may be calculated over a small number of examples, so that the calculated gradient approximates the true error gradient. The approximation method may be referred to as stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the entire system has stopped decreasing or until the error rate has reached a target level. After learning, the DCN may be presented with new images and a forward pass through the network may yield an output that may be considered an inference or a prediction of the DCN.

[0215] Deep belief networks (DBNs) are probabilistic models comprising multiple layers of hidden nodes. DBNs may be used to extract a hierarchical representation of training data sets. ADBN may be obtained by stacking up layers of Restricted Boltzmann Machines (RBMs). An RBM is a type of artificial neural network that can learn a probability distribution over a set of inputs. Because RBMs can learn a probabilityPATENTQualcomm Docket No 2502632WO66distribution in the absence of information associated with the class to which each input should be categorized, RBMs are often used in unsupervised learning. Using a hybrid unsupervised and supervised paradigm, the bottom RBMs of a DBN may be trained in an unsupervised manner and may serve as feature extractors, and the top RBM may be trained in a supervised manner (on a joint distribution of inputs from the previous layer and target classes) and may serve as a classifier.

[0216] Deep convolutional networks (DCNs) are networks of convolutional networks, configured with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning in which both the input and output targets are known for many exemplars and are used to modify the w eights of the network by use of gradient descent methods. DCNs may be feed-forward networks. In addition, as described above, the connections from a neuron in a first layer of a DCN to a group of neurons in the next higher layer are shared across the neurons in the first layer. The feed-forward and shared connections of DCNs may be exploited for fast processing. The computational burden of a DCN may be much less, for example, than that of a similarly sized neural network that comprises recurrent or feedback connections. The processing of each layer of a convolutional network may be considered a spatially invariant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then the convolutional network trained on that input may be considered three-dimensional, with two spatial dimensions along the axes of the image and a third dimension capturing color information. The outputs of the convolutional connections may be considered to form a feature map in the subsequent layer, with each element of the feature map receiving input from a range of neurons in the previous layer and from each of the multiple channels. The values in the feature map may be further processed with a non-linearity, such as a rectification, max(0,x). Values from adjacent neurons may be further pooled, which corresponds to down sampling, and may provide additional local invariance and dimensionality reduction.

[0217] FIG. 14 is a block diagram illustrating an example of a deep convolutional network (DCN) 1450. The deep convolutional network 1450 may include multiple different types of layers based on connectivity and weight sharing. As shown in FIG. 14, the deep convolutional network 1450 includes the convolution blocks 1454A, 1454B.PATENTQualcomm Docket No 2502632WO67Each of the convolution blocks 1454A, 1454B may be configured with a convolution layer(CONV) 1456, a normalization layer (LNorm) 1458, and a max pooling layer (MAX POOL) 1460.

[0218] The convolution layers 1456 may include one or more convolutional filters, which may be applied to the input data 1452 to generate a feature map. Although only two convolution blocks 1454A, 1454B are shown, the present disclosure is not so limiting, and instead, any number of convolution blocks (e g., blocks 1454A, 1454B) may be included in the deep convolutional network 1450 according to design preference. The normalization layer 1458 may normalize the output of the convolution filters. For example, the normalization layer 1458 may provide whitening or lateral inhibition. The max pooling layer 1460 may provide down sampling aggregation over space for local invariance and dimensionality reduction.

[0219] The parallel filter banks, for example, of a deep convolutional network may be loaded on a CPU or GPU of an SOC (e.g., such as the CPU 102 or GPU 104 of the SOC 100 of FIG. 1, etc.) to achieve high performance and low power consumption. In alternative aspects, the parallel filter banks may be loaded on the DSP 106 or an ISP 116 of the SOC 100 of FIG. 1. In addition, the deep convolutional network 1450 may access other processing blocks that may be present on the SOC 100 of FIG. 1. such as sensor processor 114 and storage 120, etc.

[0220] The deep convolutional network 1450 may also include one or more fully connected layers, such as layer 1462A (labeled “FC1”) and layer 1462B (labeled “FC2”). The deep convolutional network 1450 may further include a logistic regression (LR) layer 1464. Between each layer 1456, 1458, 1460. 1462A, 1462B. 1464 of the deep convolutional network 1450 are weights (not shown) that are to be updated. The output of each of the layers (e.g., 1456, 1458, 1460, 1462A, 1462B, 1464) may serve as an input of a succeeding one of the layers (e.g., 1456, 1458, 1460, 1462A, 1462B, 1464) in the deep convolutional network 1450 to learn hierarchical feature representations from input data 1452 (e.g., images, audio, video, sensor data and / or other input data) supplied at the first of the convolution blocks 1454A. The output of the deep convolutional network 1450 is a classification score 1466 for the input data 1452. The classification score 1466 may¬ be a set of probabilities, where each probability is the probability of the input data including a feature from a set of features.PATENTQualcomm Docket No 2502632WO68

[0221] FIG. 15 illustrates an example computing device architecture 1500 of an example computing device which can implement the various techniques described herein. In some examples, the computing device can include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or computing device of a vehicle), or other device. In some examples, the computing device architecture 1500 can be included in and / or used to implement a computing system, which can be for example any computing device making up SOC 100 of FIG. 1, fully connected neural network 202, locally connected neural network 204, convolutional neural network 206, deep learning network 1200, CNN 1300, the deep convolutional network of FIG. 14 and / or component(s) thereof, the machine learning architecture 300 of FIG. 3, the machine learning architecture of the KV cache processing 400 of FIG. 4, the machine learning architecture 600 of FIG. 6, the machine learning architecture 700 of FIG. 7, a machine learning architecture corresponding to the autoregressive token generation 800 of FIG. 8, a machine learning architecture corresponding to the autoregressive token generation 900 of FIG. 9, a machine learning architecture corresponding to the prefilling technique 1000a of FIG. 10A, and / or a machine learning architecture corresponding to the prefilling technique 1000b of FIG.10B, etc., or any component thereof in which the components of the system are in communication with each other using connection 1505. In some examples, the computing device architecture 1500 can perform the process 1100 of FIG. 11.

[0222] The components of computing device architecture 1500 are shown in electrical communication with each other using connection 1505, such as a bus. The example computing device architecture 1500 includes a processing unit (CPU or processor) 1510 and computing device connection 1505 that couples various computing device components including computing device memory 1515, such as read only memory (ROM) 1520 and random access memory (RAM) 1525, to processor 1510.

[0223] Computing device architecture 1500 can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of processor 1510. Computing device architecture 1500 can copy data from memory 1515 and / or the storage device 1530 to cache 1512 for quick access by processor 1510. Copying data from memory to cache can allow the cache to provide a performance boost that avoidsPATENTQualcomm Docket No 2502632WO69processor 1510 delays while waiting for data. These and other modules can control or be configured to control processor 1510 to perform various actions. Other computing device memory 1515 may be available for use as well. Memory’ 1515 can include multiple different types of memory with different performance characteristics. Processor 1510 can include any general purpose processor and a hardware or software service, such as service 1 1532, sendee 2 1534, and service 3 1536 stored in storage device 1530, configured to control processor 1510 as well as a special-purpose processor where software instructions are incorporated into the processor design. Processor 1510 may be a self-contained system, containing multiple cores or processors, a bus, memory' controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

[0224] In some aspects, the processor 1510 (in some cases in combination with one or more other components, such as the ROM 1520 and / or RAM 1525) can perform any of the techniques described herein, such as the techniques described with respect to the machine learning architecture 300 of FIG. 3, the machine learning architecture of the KV cache processing 400 of FIG. 4, the machine learning architecture 600 of FIG. 6, the machine learning architecture 700 of FIG. 7, a machine learning architecture corresponding to the autoregressive token generation 800 of FIG. 8, a machine learning architecture corresponding to the autoregressive token generation 900 of FIG. 9, a machine learning architecture corresponding to the prefilling technique 1000a of FIG.10A, and / or a machine learning architecture corresponding to the prefilling technique 1000b of FIG. 10B, etc.; and / or the process 1100 of FIG. 11.

[0225] To enable user interaction with the computing device architecture 1500, input device 1545 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. Output device 1535 can also be one or more of a number of output mechanisms known to those of skill in the art, such as a display, projector, television, speaker device, etc. In some instances, multimodal computing devices can enable a user to provide multiple types of input to communicate with computing device architecture 1500. Communication interface 1540 can generally govern and manage the user input and computing device output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.PATENTQualcomm Docket No 2502632WO70

[0226] Storage device 1530 is a non-volatile memory7and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory7devices, digital versatile disks, cartridges, random access memories (RAMs) 1525, read only memory (ROM) 1520, and hybrids thereof. Storage device 1530 can include services 1532, 1534, 1536 for controlling processor 1510. Other hardware or software modules are contemplated. Storage device 1530 can be connected to the computing device connection 1505. In some aspects, a hardware module that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary7hardware components, such as processor 1510, connection 1505, output device 1535, and so forth, to carry out the function.

[0227] Aspects of the present disclosure are applicable to any suitable electronic device (such as security systems, smartphones, tablets, laptop computers, vehicles, drones, or other devices) including or coupled to one or more active depth sensing systems. While described below with respect to a device having or coupled to one light projector, aspects of the present disclosure are applicable to devices having any number of light projectors, and are therefore not limited to specific devices.

[0228] The term “device” is not limited to one or a specific number of physical obj ects (such as one smartphone, one controller, one processor and so on). As used herein, a device may be any electronic device with one or more parts that may implement at least some portions of this disclosure. While the below description and examples use the term “device” to describe various aspects of this disclosure, the term “device” is not limited to a specific configuration, type, or number of objects. Additionally, the term “system” is not limited to multiple components or specific aspects. For example, a system may be implemented on one or more printed circuit boards or other substrates, and may have movable or static components. While the below description and examples use the term “system” to describe various aspects of this disclosure, the term “system” is not limited to a specific configuration, type, or number of objects.

[0229] Specific details are provided in the description above to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood by one of ordinary skill in the art that the aspects may be practiced without these specific details. For clarity of explanation, in some instances the present technologyPATENTQualcomm Docket No 2502632WO71may be presented as including individual functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Additional components may be used other than those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the aspects in unnecessary7detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the aspects.

[0230] Individual aspects may be described above as a process or method which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed, but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination can correspond to a return of the function to the calling function or the main function.

[0231] Processes and methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer-readable media. Such instructions can include, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or a processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc.

[0232] The term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing, containing, or carrying instruction(s) and / or data. A computer-readable medium may include a non-transitory medium in which data can be stored and that does not include carrier waves and / or transitory7electronic signals propagating wirelessly or over wired connections. Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as flash memory, memoryPATENTQualcomm Docket No 2502632WO72or memory devices, magnetic or optical disks, flash memory USB devices provided with non-volatile memory, networked storage devices, compact disk (CD) or digital versatile disk (DVD), any suitable combination thereof, among others. A computer-readable medium may have stored thereon code and / or machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, or the like.

[0233] In some aspects the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy; carrier signals, electromagnetic waves, and signals per se.

[0234] Devices implementing processes and methods according to these disclosures can include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and can take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary' tasks (e.g., a computer-program product) may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Typical examples of form factors include laptops, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.

[0235] The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in the disclosure.

[0236] In the foregoing description, aspects of the application are described withPATENTQualcomm Docket No 2502632WO73reference to specific aspects thereof, but those skilled in the art will recognize that the application is not limited thereto. Thus, while illustrative aspects of the application have been described in detail herein, it is to be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. Various features and aspects of the above-described application may be used individually or jointly. Further, aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and ddrawings are, accordingly, to be regarded as illustrative rather than restrictive. For the purposes of illustration, methods were described in a particular order. It should be appreciated that in alternate aspects, the methods may be performed in a different order than that described.

[0237] One of ordinary skill will appreciate that the less than (“<”) and greater than (“>”) symbols or terminology used herein can be replaced with less than or equal to (“<”) and greater than or equal to (“>”) symbols, respectively, without departing from the scope of this description.

[0238] Where components are described as being "‘configured to" perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., microprocessors, or other suitable electronic circuits) to perform the operation, or any combination thereof.

[0239] The phrase “coupled to” refers to any component that is physically connected to another component either directly or indirectly, and / or any component that is in communication with another component (e.g., connected to the other component over a wired or wireless connection, and / or other suitable communication interface) either directly or indirectly.

[0240] Claim language or other language reciting “at least one of’ a set and / or “one or more” of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting “at least one of A and B” or “at least one of A or B” means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C” or “at least one of A, B, or C” means A, B,PATENTQualcomm Docket No 2502632WO74C, or A and B, or A and C, or B and C, A and B and C, or any duplicate information or data (e.g., A and A, B and B, C and C, A and A and B, and so on), or any other ordering, duplication, or combination of A, B, and C. The language "‘at least one of’ a set and / or ■‘one or more” of a set does not limit the set to the items listed in the set. For example, claim language reciting “at least one of A and B” or “at least one of A or B” may mean A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases “at least one” and “one or more” are used interchangeably herein.

[0241] Claim language or other language reciting “at least one processor configured to,” “at least one processor being configured to,” “one or more processors configured to,” “one or more processors being configured to,” or the like indicates that one processor or multiple processors (in any combination) can perform the associated operation(s). For example, claim language reciting “at least one processor configured to: X, Y, and Z” means a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each tasked with a certain subset of operations X, Y, and Z such that together the multiple processors perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, claim language reciting “at least one processor configured to: X, Y. and Z” can mean that any single processor may only perform at least a subset of operations X, Y, and Z.

[0242] Where reference is made to one or more elements performing functions (e.g., steps of a method), one element may perform all functions, or more than one element may collectively perform the functions. When more than one element collectively performs the functions, each function need not be performed by each of those elements (e.g., different functions may be performed by different elements) and / or each function need not be performed in whole by only one element (e.g., different elements may perform different sub-functions of a function). Similarly, where reference is made to one or more elements configured to cause another element (e.g.. an apparatus) to perform functions, one element may be configured to cause the other element to perform all functions, or more than one element may collectively be configured to cause the other element to perform the functions.

[0243] Where reference is made to an entity7(e.g., any entity or device described herein) performing functions or being configured to perform functions (e.g., steps of a method), the entity may be configured to cause one or more elements (individually or collectively)PATENTQualcomm Docket No 2502632WO75to perform the functions. The one or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more (or all) of the functions, and / or any combination thereof. Where reference to the entity performing functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to collectively perform the functions. When the entity is configured to cause more than one component to collectively perform the functions, each function need not be performed by each of those components (e.g., different functions may be performed by different components) and / or each function need not be performed in whole by only one component (e.g., different components may perform different sub-functions of a function).

[0244] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality’. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0245] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices such as general purposes computers, wireless communication device handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, performs one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may includePATENTQualcomm Docket No 2502632WO76packaging materials. The computer-readable medium may comprise memory or data storage media, such as random access memory' (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as propagated signals or waves.

[0246] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein.

[0247] Illustrative aspects of the disclosure include:

[0248] Aspect 1. An apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: obtain a sequence of a plurality of tokens corresponding to an input to a machine learning network including a plurality of transformer layers, the plurality of transformer layers including a first subset of layers and a second subset of layers, wherein the first subset of layers includes a configured early exit layer; determine an early exit prediction for a respective token included in the sequence of the plurality of tokens, the early exit prediction determined based on a hidden state vector generated as output for the respective token by the configured early exit layer, wherein a first value of the early exit prediction causes the atPATENTQualcomm Docket No 2502632WO77least one processor to skip processing of the hidden state vector for the respective token by the second subset of layers; and determine, based on the early exit prediction for the respective token, key-value (KV) selection information corresponding to one or more transformer layers of the second subset of layers and a subsequent token included in the sequence of the plurality of tokens after the respective token, wherein the KV selection information comprises the hidden state vector generated as output for the respective token by the configured early exit layer, based on the first value of the early exit prediction.

[0249] Aspect 2. The apparatus of Aspect 1, wherein the hidden state vector for the respective token is generated as a self-attention output associated with a self-attention calculated for the respective token by the configured early exit layer.

[0250] Aspect 3. The apparatus of any of Aspects 1 to 2, wherein a second value of the early exit prediction causes the at least one processor to not skip processing of the hidden state vector for the respective token by the second subset of layers.

[0251] Aspect 4. The apparatus of Aspect 3. wherein, based on the second value of the early exit prediction, the KV selection information for a respective transformer layer of the second subset of layers comprises a hidden state vector generated as output by a preceding transformer layer connected to the respective transformer layer, the preceding transformer layer included in the second subset of layers.

[0252] Aspect 5. The apparatus of any of Aspects 1 to 4, wherein, to skip processing of the hidden state vector for the respective token by the second subset of layers, the first value of the early exit prediction further causes the at least one processor to: provide the hidden state vector generated as output for the respective token by the configured early exit layer to a language model head connected to an output of a final transformer layer of the second subset of layers; and determine sampling information corresponding to a predicted next token for the sequence of the plurality of tokens using the hidden state vector generated as output for the respective token by the configured early exit layer.

[0253] Aspect 6. The apparatus of any of Aspects 1 to 5, wherein the early exit prediction is determined using an early exit router connected between the first and second subset of layers.

[0254] Aspect 7. The apparatus of Aspect 6, wherein the early exit router is connected between an output of the configured early exit layer included in the first subset of layers,PATENTQualcomm Docket No 2502632WO78and an input of a first transformer layer included in the second subset of layers.

[0255] Aspect 8. The apparatus of any of Aspects 1 to 7. wherein: the first subset of layers comprises a first subset of transformer layers of the plurality of transformer layers, and the second subset of layers comprises a second subset of transformer layers of the plurality of transformer layers; the first subset of transformer layers are configured to determine self-attention for the respective token; and the second subset of transformer layers are configured to determine cross-attention for the subsequent token based on the first value of the early exit prediction for the respective token and using the KV selection information.

[0256] Aspect 9. The apparatus of Aspect 8, wherein the cross-attention for the subsequent token is based on using the KV selection information to determine a respective keys (K) vector input and a respective values (V) vector input for each transformer layer of the second subset of transformer layers, and wherein the respective K vector input and the respective V vector input are determined from a cached representation of the hidden state vector generated as output for the respective token by the configured early exit layer.

[0257] Aspect 10. The apparatus of any of Aspects 1 to 9, wherein the at least one processor is configured to: store the hidden state vector generated as output for the respective token by the configured early exit layer within a respective KV cache associated with each transformer layer of the second subset of layers; and process the subsequent token the second subset of layers, wherein each transformer layer of the second subset of layers is configured to determine cross-attention for the subsequent token using the hidden state vector stored within the respective KV cache associated with each transformer layer of the second subset of layers.

[0258] Aspect 11. The apparatus of Aspect 10, wherein the cross-attention for the subsequent token is determined using a query (Q) vector associated with the subsequent token, and a KV vector determined based on the hidden state vector generated as output for the respective token by the configured early exit layer and stored within the respective KV cache associated with each transformer layer of the second subset of layers.

[0259] Aspect 12. The apparatus of any of Aspects 1 to 11, wherein the KV selection information corresponds to a token generation phase to configure KV vectors within a respective KV cache associated wi th each transformer layer of the second subset of layers.PATENTQualcomm Docket No 2502632WO79

[0260] Aspect 13. The apparatus of any of Aspects 1 to 12, wherein the configured early exit layer is associated with an early exit router configured to determine the early exit prediction based on the hidden state vector generated as output for the respective token by the configured early exit layer.

[0261] Aspect 14. A method comprising: obtaining a sequence of a plurality of tokens corresponding to an input to a machine learning network including a plurality of transformer layers, the plurality of transformer layers including a first subset of layers and a second subset of layers, wherein the first subset of layers includes a configured early exit layer; determining an early exit prediction for a respective token included in the sequence of the plurality of tokens, the early exit prediction determined based on a hidden state vector generated as output for the respective token by the configured early exit layer, wherein a first value of the early exit prediction corresponds to skipping processing of the hidden state vector for the respective token by the second subset of layers; and determining, based on the early exit prediction for the respective token, key -value (KV) selection information corresponding to one or more transformer layers of the second subset of layers and a subsequent token included in the sequence of the plurality of tokens after the respective token, wherein the KV selection information comprises the hidden state vector generated as output for the respective token by the configured early exit layer, based on the first value of the early exit prediction.

[0262] Aspect 15. The method of Aspect 14, wherein the hidden state vector for the respective token is generated as a self-attention output associated with a self-attention calculated for the respective token by the configured early exit layer.

[0263] Aspect 16. The method of any of Aspects 14 to 15, wherein a second value of the early exit prediction corresponds to not skipping processing of the hidden state vector for the respective token by the second subset of layers.

[0264] Aspect 17. The method of Aspect 16, wherein, based on the second value of the early exit prediction, the KV selection information for a respective transformer layer of the second subset of layers comprises a hidden state vector generated as output by a preceding transformer layer connected to the respective transformer layer, the preceding transformer layer included in the second subset of layers.

[0265] Aspect 18. The method of any of Aspects 14 to 17, wherein skipping processingPATENTQualcomm Docket No 2502632WO80of the hidden state vector for the respective token by the second subset of layers based on the first value of the early exit prediction further includes: providing the hidden state vector generated as output for the respective token by the configured early exit layer to a language model head connected to an output of a final transformer layer of the second subset of layers; and determining sampling information corresponding to a predicted next token for the sequence of the plurality of tokens using the hidden state vector generated as output for the respective token by the configured early exit layer.

[0266] Aspect 19. The method of any of Aspects 14 to 18, wherein the early exit prediction is determined using an early exit router connected between the first and second subset of layers.

[0267] Aspect 20. The method of Aspect 19, wherein the early exit router is connected between an output of the configured early exit layer included in the first subset of layers, and an input of a first transformer layer included in the second subset of layers.

[0268] Aspect 21. The method of any of Aspects 14 to 20, wherein: the first subset of layers comprises a first subset of transformer layers of the plurality of transformer layers, and the second subset of layers comprises a second subset of transformer layers of the plurality of transformer layers; the first subset of transformer layers are configured to determine self-attention for the respective token; and the second subset of transformer layers are configured to determine cross-attention for the subsequent token based on the first value of the early exit prediction for the respective token and using the KV selection information.

[0269] Aspect 22. The method of Aspect 21, wherein the cross-attention for the subsequent token is based on using the KV selection information to determine a respective keys (K) vector input and a respective values (V) vector input for each transformer layer of the second subset of transformer layers, and wherein the respective K vector input and the respective V vector input are determined from a cached representation of the hidden state vector generated as output for the respective token by the configured early exit layer.

[0270] Aspect 23. The method of any of Aspects 14 to 22, further comprising: storing the hidden state vector generated as output for the respective token by the configured early exit layer within a respective KV cache associated with each transformer layer of the second subset of layers; and processing the subsequent token the second subset ofPATENTQualcomm Docket No 2502632WO81layers, wherein each transformer layer of the second subset of layers is configured to determine cross-attention for the subsequent token using the hidden state vector stored within the respective KV cache associated with each transformer layer of the second subset of layers.

[0271] Aspect 24. The method of Aspect 23, wherein the cross-attention for the subsequent token is determined using a query (Q) vector associated with the subsequent token, and a KV vector determined based on the hidden state vector generated as output for the respective token by the configured early exit layer and stored within the respective KV cache associated with each transformer layer of the second subset of layers.

[0272] Aspect 25. The method of any of Aspects 14 to 24, wherein the KV selection information corresponds to a token generation phase to configure KV vectors within a respecti ve KV cache associated with each transformer layer of the second subset of lay ers.

[0273] Aspect 26. The method of any of Aspects 14 to 25, wherein the configured early exit layer is associated with an early exit router configured to determine the early exit prediction based on the hidden state vector generated as output for the respective token by the configured early exit layer.

[0274] Aspect 27. A non-transitory computer-readable storage medium comprising instructions stored thereon which, when executed by at least one processor, causes the at least one processor to: obtain a sequence of a plurality of tokens corresponding to an input to a machine learning netw ork including a plurality of transformer layers, the plurality of transformer layers including a first subset of layers and a second subset of layers, wherein the first subset of layers includes a configured early exit layer; determine an early exit prediction for a respective token included in the sequence of the plurality of tokens, the early exit prediction determined based on a hidden state vector generated as output for the respective token by the configured early exit layer, wherein a first value of the early exit prediction causes the at least one processor to skip processing of the hidden state vector for the respective token by the second subset of layers; and determine, based on the early exit prediction for the respective token, key-value (KV) selection information corresponding to one or more transformer layers of the second subset of layers and a subsequent token included in the sequence of the plurality of tokens after the respective token, wherein the KV selection information comprises the hidden state vector generatedPATENTQualcomm Docket No 2502632WO82as output for the respective token by the configured early exit layer, based on the first value of the early exit prediction.

[0275] Aspect 28. A non-transitory computer-readable storage medium comprising instructions stored thereon which, when executed by at least one processor, causes the at least one processor to perform operations according to any of Aspects 1 to 13.

[0276] Aspect 29. An apparatus comprising one or more means for performing operations according to any of Aspects 1 to 13.

Claims

PATENTQualcomm Docket No 2502632WO83CLAIMSWhat is claimed is:

1. An apparatus comprising:at least one memory; andat least one processor coupled to the at least one memory and configured to: obtain a sequence of a plurality of tokens corresponding to an input to a machine learning network including a plurality of transformer layers, the plurality of transformer layers including a first subset of layers and a second subset of layers, wherein the first subset of layers includes a configured early exit layer;determine an early exit prediction for a respective token included in the sequence of the plurality of tokens, the early exit prediction determined based on a hidden state vector generated as output for the respective token by the configured early exit layer, wherein a first value of the early exit prediction causes the at least one processor to skip processing of the hidden state vector for the respective token by the second subset of layers; anddetermine, based on the early' exit prediction for the respective token, key -value (KV) selection information corresponding to one or more transformer layers of the second subset of layers and a subsequent token included in the sequence of the plurality of tokens after the respective token, wherein the KV selection information comprises the hidden state vector generated as output for the respective token by the configured early exit layer, based on the first value of the early exit prediction.

2. The apparatus of claim 1, wherein the hidden state vector for the respective token is generated as a self-attention output associated with a self-attention calculated for the respective token by the configured early exit layer.

3. The apparatus of claim 1, wherein a second value of the early exit prediction causes the at least one processor to not skip processing of the hidden state vector for the respective token by the second subset of layers.PATENTQualcomm Docket No 2502632WO844. The apparatus of claim 3, wherein, based on the second value of the early exit prediction, the KV selection information for a respective transformer layer of the second subset of layers comprises a hidden state vector generated as output by a preceding transformer layer connected to the respective transformer layer, the preceding transformer layer included in the second subset of layers.

5. The apparatus of claim 1, wherein, to skip processing of the hidden state vector for the respective token by the second subset of layers, the first value of the early exit prediction further causes the at least one processor to:provide the hidden state vector generated as output for the respective token by the configured early exit layer to a language model head connected to an output of a final transformer layer of the second subset of layers; anddetermine sampling information corresponding to a predicted next token for the sequence of the plurality of tokens using the hidden state vector generated as output for the respective token by the configured early exit layer.

6. The apparatus of claim 1, wherein the early exit prediction is determined using an early exit router connected between the first and second subset of layers.

7. The apparatus of claim 6, wherein the early exit router is connected between an output of the configured early exit layer included in the first subset of layers, and an input of a first transformer layer included in the second subset of layers.

8. The apparatus of claim 1, wherein:the first subset of layers comprises a first subset of transformer layers of the plurality of transformer layers, and the second subset of layers comprises a second subset of transformer layers of the plurality of transformer layers;the first subset of transformer layers are configured to determine self-attention for the respective token; andthe second subset of transformer layers are configured to determine crossattention for the subsequent token based on the first value of the early exit prediction for the respective token and using the KV selection information.PATENTQualcomm Docket No 2502632WO859. The apparatus of claim 8, wherein the cross-attention for the subsequent token is based on using the KV selection information to determine a respective keys (K) vector input and a respective values (V) vector input for each transformer layer of the second subset of transformer layers, and wherein the respective K vector input and the respective V vector input are determined from a cached representation of the hidden state vector generated as output for the respective token by the configured early exit layer.

10. The apparatus of claim 1, wherein the at least one processor is configured to: store the hidden state vector generated as output for the respective token by the configured early exit layer within a respective KV cache associated with each transformer layer of the second subset of layers; andprocess the subsequent token the second subset of layers, wherein each transformer layer of the second subset of layers is configured to determine crossattention for the subsequent token using the hidden state vector stored within the respective KV cache associated with each transformer layer of the second subset of layers.

11. The apparatus of claim 10, wherein the cross-attention for the subsequent token is determined using a query' (Q) vector associated with the subsequent token, and a KV vector determined based on the hidden state vector generated as output for the respective token by the configured early exit layer and stored within the respective KV cache associated with each transformer layer of the second subset of layers.

12. The apparatus of claim 1, wherein the KV selection information corresponds to a token generation phase to configure KV vectors within a respective KV cache associated with each transformer layer of the second subset of layers.

13. The apparatus of claim 1, wherein the configured early exit layer is associated with an early exit router configured to determine the early exit prediction based on the hidden state vector generated as output for the respective token by the configured early exit layer.PATENTQualcomm Docket No 2502632WO8614. A method comprising:obtaining a sequence of a plurality of tokens corresponding to an input to a machine learning network including a plurality of transformer layers, the plurality of transformer layers including a first subset of layers and a second subset of layers, wherein the first subset of layers includes a configured early exit layer;determining an early exit prediction for a respective token included in the sequence of the plurality of tokens, the early exit prediction determined based on a hidden state vector generated as output for the respective token by the configured early exit layer, wherein a first value of the early exit prediction corresponds to skipping processing of the hidden state vector for the respective token by the second subset of layers; anddetermining, based on the early exit prediction for the respective token, keyvalue (KV) selection information corresponding to one or more transformer layers of the second subset of layers and a subsequent token included in the sequence of the plurality of tokens after the respective token, wherein the KV selection information comprises the hidden state vector generated as output for the respective token by the configured early exit layer, based on the first value of the early exit prediction.

15. The method of claim 14, wherein the hidden state vector for the respective token is generated as a self-attention output associated with a self-attention calculated for the respective token by the configured early exit layer.

16. The method of claim 14, wherein a second value of the early exit prediction corresponds to not skipping processing of the hidden state vector for the respective token by the second subset of layers.

17. The method of claim 16, wherein, based on the second value of the early exit prediction, the KV selection information for a respective transformer layer of the second subset of layers comprises a hidden state vector generated as output by a preceding transformer layer connected to the respective transformer layer, the preceding transformer layer included in the second subset of layers.PATENTQualcomm Docket No 2502632WO8718. The method of claim 14, wherein skipping processing of the hidden state vector for the respective token by the second subset of layers based on the first value of the early exit prediction further includes:providing the hidden state vector generated as output for the respective token by the configured early exit layer to a language model head connected to an output of a final transformer layer of the second subset of layers; anddetermining sampling information corresponding to a predicted next token for the sequence of the plurality of tokens using the hidden state vector generated as output for the respective token by the configured early exit layer.

19. The method of claim 14, wherein the early exit prediction is determined using an early exit router connected between the first and second subset of layers.

20. A non-transitory computer-readable storage medium comprising instructions stored thereon which, when executed by at least one processor, causes the at least one processor to:obtain a sequence of a plurality of tokens corresponding to an input to a machine learning network including a plurality of transformer layers, the plurality of transformer layers including a first subset of layers and a second subset of layers, wherein the first subset of layers includes a configured early exit layer;determine an early exit prediction for a respective token included in the sequence of the plurality of tokens, the early exit prediction determined based on a hidden state vector generated as output for the respective token by the configured early exit layer, wherein a first value of the early exit prediction causes the at least one processor to skip processing of the hidden state vector for the respective token by the second subset of layers; anddetermine, based on the early exit prediction for the respective token, key -value (KV) selection information corresponding to one or more transformer layers of the second subset of layers and a subsequent token included in the sequence of the plurality of tokens after the respective token, wherein the KV selection information comprises the hidden state vector generated as output for the respective token by the configured early exit layer, based on the first value of the early exit prediction.