Scheduler for removing redundancy in quantized multiplications on inference accelerators

Abstract

Methods and systems, including computer-readable media, are described for a multiplication scheduler for quantized inference (MSQI) computations. The multiplication scheduler is implemented as a MSQI hardware unit on a compute tile / unit of a special-purpose hardware integrated circuit, such as an inference accelerator or tensor processing unit (TPU). The multiplication scheduler is designed to optimize multiplication operations in hardware ML inference accelerators by minimizing or eliminating redundant computations (e.g., multiplications) on prevalent quantized formats.

Description

ATTORNEY DOCKET NO. 56113-0834WO1SCHEDULER FOR REMOVING REDUNDANCY IN QUANTIZED MULTIPLICATIONS ON INFERENCE ACCELERATORS BACKGROUND

[0001] This specification generally relates to quantized multiplications performed on a hardware integrated circuit.

[0002] In the context of machine-learning, neural networks employ one or more layers of nodes to generate an output, e.g., a classification, for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to one or more other layers in the network, e.g., other hidden layers or the output layer of the network. A layer of the neural network can generate an output from a received input in accordance with current values of a respective set of parameters. Some neural networks can be convolutional neural networks (CNNs) configured for image processing or recurrent neural networks (RNNs) configured for speech and language processing. Other neural networks can have a transformer network architecture that is based on attention mechanisms, which could augment recurrent and convolutional networks.Different types of neural network architectures can be used to perform a variety' of tasks related to classification or pattern recognition, predictions that involve data modeling, and information clustering.

[0003] In general, the layers of these neural networks can have an associated set of kernels, which may correspond to parameters or weights. The associated set of kernels are used to process inputs (e.g., a batch of inputs) through the neural network layer to generate a corresponding output of the layer for computing a neural network inference. A batch of inputs and a set of kernels can be represented as a tensor, e.g., a multidimensional array of inputs and weights. A hardware circuit that implements a neural network includes memory' with locations that are identified by an address value. The memory locations can correspond to elements of a tensor and the tensor elements may be traversed or accessed using control logic of the circuit. For example, the control logic can determine or compute a memory address value of an element to load or store the corresponding data value of the element.SUMMARY

[0004] The specification describes techniques for a multiplication scheduler for quantized inference (MSQ1) computations. The multiplication scheduler is implemented as a MSQI hardware unit on a special-purpose hardware integrated circuit, such as machine-learningATTORNEY DOCKET NO. 56113-0834WO1(ML) hardware, tensor processing unit (TPU), neural network processor, or related specialpurpose processor. The multiplication scheduler is designed to optimize multiplication operations in hardware ML inference accelerators by minimizing or eliminating redundant computations (e.g., multiplications) on prevalent quantized formats. The formats include integer quantization used to execute Large Language Models (LLMs) and Generative Al (GenAI) applications in edge devices such as smartphones, tablets, and laptops.

[0005] Quantization can be used to scale high-resolution floating-point values to lower-precision quantized values to improve ML processing / computing efficiency in edge devices. Some integer quantization formats are limited in the number of unique values that can be expressed among a set of operands that are multiplied and accumulated to execute an ML inference. For neural network inference computations, the operands are: i) activations that represent inputs to a neural network layer and ii) a corresponding set of parameters (or weights) for the layer. For certain large ML models with millions or billions of parameters (e.g., weights), this limited number of unique values among the operands will yield frequent repetition of identical computations based on repeat values of input and weight operands.

[0006] The hardware MSQI unit is configured to detect redundant or repeat values among quantized integer values that are operands for neural network or ML computations executed on a special-purpose hardware inference processor / accelerator. The computations include matrix multiplications and / or vector math where products are computed from multiplying two quantized integer values that form an operand pair (e.g., activation & weight). The matrix multiplications and / or vector math are executed using hardware arithmetic and multiplier circuits that are integrated in computing units of the special-purpose inference processor.

[0007] The hardware MSQI unit implements a reorder buffer to store prior computed products and uses a corresponding product available flag to limit or prevent redundant multiplications on reoccurring quantized integer values for different sets of ML operands (e.g., operand pairs). For an initial occurrence of a particular set of quantized integer values in a given operand pair, the multiplication scheduler is configured to compute the multiplication product for that pair. The multiplication scheduler buffers that computed product and uses the product available flag to indicate a prior computed product is available for reuse. When the particular set of quantized integer values is repeated in a subsequent operand pair, the multiplication scheduler detects this recurrence, reads the product_available flag, and reuses the prior computed product corresponding to the flag.

[0008] One aspect of the subject matter described in this specification can be embodied in a method performed using a compute tile of a hardware integrated circuit configured toATTORNEY DOCKET NO. 56113-0834WO1implement a neural network. The method includes: i) receiving, at the compute tile, an operand pair that includes an activation and a weight and ii) identifying, in a product buffer of the compute tile, an existing operand pair comprising an existing activation and an existing weight stored in the product buffer. The method further includes: i) determining, by a comparator circuit of the compute tile, that the operand pair matches the existing operand pair; and ii) obtaining, at the compute tile, a product of multiplying the existing activation and the existing weight of the existing operand pair. The method also includes computing an output for a layer of the neural network using the obtained product.

[0009] These and other implementations can each optionally include one or more of the following features. For example, in some implementations, determining that the operand pair matches the existing operand pair includes: i) comparing, by the comparator circuit, the activation of the operand pair and the existing activation of the existing operand pair; and ii) based on the comparing of the activation and the existing activation, generating, by the comparator circuit, an activation compare signal indicating the activation matches the existing activation. Determining that the operand pair matches the existing operand pair further includes: i) comparing, by the comparator circuit, the weight of the operand pair and the existing weight of the existing operand pair; and ii) based on the comparing of the weight and the existing weight, generating, by the comparator circuit, a weight compare signal indicating the weight matches the existing weight.

[0010] In some implementations, determining that the operand pair matches the existing operand pair includes: i) executing, by the comparator circuit, compare operations between individual operands of the operand pair and corresponding individual operands of the existing operand pair; ii) computing a respective result for each of the compare operations; and iii) generating, by the comparator circuit, a compare output signal based on the respective result for each of the compare operations. The compare output signal can indicate that the individual operands of the operand pair match the corresponding individual operands of the existing operand pair. Executing compare operations can include executing each compare operation in parallel over a single clock cycle using a respective individual comparator of the comparator circuit.

[0011] Obtaining the product includes: i) detecting a product flag corresponding to the existing operand pair in response to determining that the operand pair matches the existing operand pair; and ii) obtaining the product corresponding to the existing operand pair based on the detected product flag. Obtaining the product corresponding to the existing operand pair based on the detected product flag includes generating, by a scheduler of the computeATTORNEY DOCKET NO. 56113-0834WO1tile, a product control signal that causes the product buffer to output a product of multiplying operands of the existing operand pair.

[0012] In some implementations, the product buffer includes multiple entries, and each entry of the multiple entries corresponds to a memory location configured to store an activation, a weight, a product flag, and a product of multiplying the activation and the weight. The operand pair can be a first operand pair among multiple operand pairs that are received at the compute tile, and the method can further include: i) selecting, by a scheduler of the compute tile, a second operand pair from among the multiple operand pairs in response to determining that the first operand pair matches the existing operand pair; and ii) generating, at the compute tile, a control signal that controls a multiplexer (MUX) of the compute tile.

[0013] The method also includes routing, by the MUX and based on the control signal, the second operand pair to a multiplier-and-accumulator cell (MAC) of the compute tile. The method also includes computing a product of multiplying an activation and a weight of the second operand pair using the MAC. The compute tile has a computing unit that can include multiple MACs. Each operand of the operand pair and each operand of the existing operand pair can be a quantized integer having a discrete set of unique integer combinations based on a bit-width used to represent the quantized integer. In some implementations, receiving the operand pair includes receiving multiple operand pairs, each of the multiple operand pairs including a respective activation and a corresponding weight.

[0014] One aspect of the subject matter described in this specification can be embodied in a hardware integrated circuit configured to implement a neural network. The hardware integrated circuit includes a compute tile that includes a scheduler configured to receive an operand pair that includes an activation and a weight, a product buffer, and a comparator circuit. The product buffer is configured to store: i) one or more existing operand pairs, each of the one or more existing operand pairs comprising a respective existing activation and a respective existing weight; and ii) for each of the one or more existing operand pairs: a respective product of multiplying the existing activation and the existing weight of the existing operand pair. The comparator circuit is configured to determine whether the operand pair matches an existing operand pair of the one or more existing operand pairs.

[0015] These and other implementations can each optionally include one or more of the following features. For example, in some implementations, the comparator circuit is configured to: i) compare the activation of the operand pair and the existing activation of the existing operand pair; ii) determine that the operand pair matches the existing operand pairATTORNEY DOCKET NO. 56113-0834WO1based on the compare of the activation and the existing activation; and iii) generate an activation compare signal indicating the activation matches the existing activation.Relatedly, the comparator circuit is further configured to: i) compare the weight of the operand pair and the existing weight of the existing operand pair; ii) determine that the operand pair matches the existing operand pair based on the compare of the weight and the existing weight; and iii) generate a weight compare signal indicating the weight matches the existing weight.

[0016] The comparator circuit is configured to execute compare operations between individual operands of the operand pair and corresponding individual operands of the existing operand pair and compute a respective result for each of the compare operations. In some implementations, the comparator circuit is configured to, for each of the compare operation, generate a compare output signal based on the respective result for the compare operation. The compare output signal indicates the individual operands of the operand pair matches the corresponding individual operands of the existing operand pair.

[0017] The comparator circuit can include at least a first comparator and a second comparator. Each of the first comparator and the second comparator can be configured to execute a respective compare operation and the comparator circuit can be configured to execute the respective compare operations in parallel over a single clock cycle using the first comparator and the second comparator. The product buffer can be configured to store, for each of the one or more existing operand pairs, a product flag corresponding to the existing operand pair. The compute tile can be configured to obtain the respective product of the existing operand pair based on the product flag corresponding to the existing operand pair when the received operand pair matches the existing operand pair.

[0018] The scheduler can be configured to generate a product control signal that causes the product buffer to output the respective product of the existing operand pair. The compute tile obtains the respective product of the existing operand pair based on product flag and the product control signal. In some implementations, the operand pair is a first operand pair among multiple operand pairs that are received by the scheduler and the scheduler is configured to select a second operand pair from among the multiple operand pairs.

[0019] The compute tile further includes a multiplexer (MUX) and is configured to generate a control signal that controls a data routing function of the MUX. The MUX is configured to route the second operand pair to a multiplier-and-accumulator cell (MAC) of the compute tile based on the control signal. The MAC is configured to compute a product of multiplying an activation and a weight of the second operand pair. Each of the activation andATTORNEY DOCKET NO. 56113-0834WO1the weight of the operand pair can be a quantized integer having a discrete set of unique integer combinations based on a bit-width used to represent the quantized integer.

[0020] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation causes the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0021] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages.

[0022] First, the techniques described herein reduce the power consumption of hardware multiplier circuitry of an inference accelerator that implements an ML neural network model. According to the techniques described herein, products that are computed to perform neural network inferences can be stored in one or more reorder buffers. These products can then be accessed for future neural network computations for which they are required without having to be computed again.

[0023] This buffering and reuse of the prior computed products reduces the overall quantity of multiplication operations that are performed by the hardware multiplier circuitry of an inference accelerator that implements an ML neural network model. For example, by reusing the prior computed product the multiplication scheduler avoids toggling the hardware multiplier circuits to perform needless, redundant multiply operations. Reducing the multiplication operations required to execute an inference workload leads to fewer circuit toggles and reduced power consumption, improving the power efficiency of the circuitry.

[0024] Second, the techniques described herein can also enhance multiplication throughput concurrent with optimizing inference compute for different ML accelerator applications. For example, the disclosed techniques reduce the overall quantity of redundant or repeat multiplication operations performed on reoccurring operands for a given neural network inference computation, as described above. This reduction allows the multiplication circuitry to perform more non-redundant multiplications (e.g., on non-reoccurring quantized operands), which enhances multiplication throughput of an inference accelerator. Relatedly, this reduction of repeat multiplies allows the inference accelerator to execute more neural network inference computations within a given time period, which optimizes inference compute for different ML use cases.ATTORNEY DOCKET NO. 56113-0834WO1

[0025] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Fig. 1 is a block diagram of an example computing system that includes compute tiles of an inference processor that implements neural network machine-learning models.

[0027] Fig. 2 show s an example hardware unit that represents a multiplication scheduler for quantized inference (MSQI) computations.

[0028] Fig. 3 shows an example quantized product entry and related aspects of a quantized product reorder buffer (QPRB) of the MSQI.

[0029] Fig. 4 shows an example operational flow for executing MSQI computations on a compute tile in the computing system of Fig. 1.

[0030] Fig. 5 is an example quantization format for operands stored as a quantized product entry in the QPRB.

[0031] Fig. 6 is an example process for removing redundancy in quantized multiplications executed on the compute tile of Fig. 1.

[0032] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0033] Fig. 1 is a block diagram of an example computing system 100 for implementing a neural network model at a hardware integrated circuit, such as a machine-learning hardware accelerator. Compute system 100 includes one or more compute tiles 101, a host 120, and a higher-level controller 125 (‘‘controller 125"’). As described in more detail below; the host 120 and the controller 125 cooperate to provide datasets and instructions to one or more compute tiles 101 of system 100.

[0034] In some implementations, the host 120 and the controller 125 are the same device. The host 120 and the controller 125 can also perform distinct functions but be integrated in a single device package. For example, the host 120 and controller 125 can form a central processing unit (CPU) that interacts or cooperates w ith a hardware accelerator, which includes the multiple compute tiles 101. In some implementations, the host 120, controller 125, and multiple compute tiles 101 are included or formed on a single integrated circuit die.ATTORNEY DOCKET NO. 56113-0834WO1For example, the host 120, controller 125, and multiple compute tiles 101 can form a specialpurpose System-on-Chip (SoC) that is optimized for executing neural network models for processing machine-learning workloads.

[0035] Each compute tile 101 generally includes a controller 103 that provides one or more control signals 105 to cause inputs (or activations) for an input vector 102 to be stored at, or accessed from, a memory location of a first memory 108 (“memory 108”). Likewise, the controller 103 can also provide one or more control signals 105 to cause weights (or parameters) for a matrix structure of weights 104 to be stored at, or accessed from, a memory location of a second memory 110 (“memory' 110”). The controller 103 is referred to alternatively herein as a microcontroller or a local microcontroller. In some implementations, the input vector 102 is obtained from an input tensor, whereas the matrix structure of weights 104 is obtained from a parameter tensor. Each of the input tensor and the parameter tensor may be multi-dimensional data structures, such as a multi-dimensional matrix or tensor.

[0036] Each memory' location of memory 108, 110 may be identified by a corresponding memory address. Each of memory’ 108, 110 can be implemented as a series of banks, units, or any other related storage medium or device. Each of memory 108, 110 can include one or more registers, buffers, or both. In general, controller 103 arbitrates access to each of memory' 108, 110. In some implementations, inputs or activations are stored at memory 108, memory 110. or both; and weights are stored at memory 110, memory 108. or both. For example, inputs and weights may be transferred between memory 108 and memory 110 to facilitate certain neural network computations.

[0037] Each compute tile 101 also includes an input activation bus 106, an output activation bus 116, and a computational unit 114 that includes multiply accumulate cells (MACs) 115-0, 115-1, and 115-«. As indicated in the example of Fig. 1, the computational unit 114 can include n multiply accumulate cells, where n is an integer greater than or equal to one.

[0038] Controller 103 can generate control signals 105 to initiate retrieving, obtaining, or otherwise routing operands stored at the memory of the compute tile 101. More specifically, controller 103 can generate control signals 105 to cause: i) inputs / activations of a vector 102 stored at memory' 108 to be read (or retrieved) from a memory location of memory 108 and routed to computational unit 114 and ii) weights 104 stored at memory 110 to be read (or retrieved) from a memory location of memory 110 and routed to computational unit 114. For example, each input obtained from memory 108 is provided to input activation bus 106 for routing (e.g., direct routing) to a compute cell 115 0 / 1 / M in the computational unit 114.ATTORNEY DOCKET NO. 56113-0834WO1Similarly, each weight obtained from memory 110 is routed to a cell 115 0 / 1 / M of the computational unit 114.

[0039] As described below, each cell 115 0 / 1 / z? performs computations that produce partial sums or accumulated values for generating outputs for a given neural network layer. An activation function may be applied to a set of outputs to generate a set or vector of activated values that represent an output(s) or output activations for the neural network layer. In some implementations, the outputs or output activations are routed for storage and / or transfer via output activation bus 116. For example, a set of output activations can be routed for storage at the memory 108, transferred to a different compute tile 101, or both. For example, a vector or matrix of output activations can be transferred from a first compute tile 101 to a second, different compute tile 101 for processing at the second compute tile 101 as input activations for a different layer of the neural network.

[0040] The compute tile 101 includes a non-linear unit 118 configured to apply an activation function (e.g., anon-linear function) to output or accumulated values computed at the compute tile. The non-linear unit 118. and its corresponding activation function(s), are used to capture complex, non-linear relationships among input and output data processed at the compute tile. The non-linear unit 118 can be a hardware circuit that is separate and / or distinct from the computational unit 114. In some implementations, the non-linear unit 118 is integrated in the computational unit 114.

[0041] The non-linear unit 118 is configured to apply an activation function to a set of accumulated values generated by the computational unit 114 and generate a set of output activations based on the activation function applied to the accumulated values. In some examples the activations can be routed directly from the non-linear unit 118 to memory 108 for storage at the memory, whereas in some other examples the activations can be passed back to the output activation bus 116 from the non-linear unit 118 and then routed to memory 108 for storage at the memory.

[0042] Controller 103 can also generate control signals 107 that are passed to a multiplication scheduler implemented as MSQI hardware unit 112 (“MSQI unit 112”). The controller 103 generates control signals 107 that are used to execute scheduling and buffering tasks that optimize multiplication operations of the compute tile 101. For example, the controller 103 uses the control signals 107 and MSQI unit 112 to minimize or eliminate redundant computations (e.g., multiplications) on prevalent quantized formats. The MSQI unit 112 is configured to detect redundant or repeat values among quantized integers that areATTORNEY DOCKET NO. 56113-0834WO1operands for neural network or ML computations executed on the compute tile 101. This is described in more detail below at least with reference to the example of Fig. 2.

[0043] In general, each compute tile 101 and system 100 can include additional hardware structures to perform computations associated with multi-dimensional data structures such as tensors, matrices and / or data arrays. In some implementations, inputs / activations for an input vector (or tensor) 102 and weights 104 for a parameter tensor can be pre-loaded into memory’ 108, 110 of the compute tiles 101. The inputs and weights are received as sets of data values that arrive at a particular compute tile 101 from a host 120 (e.g., an external host), via a host interface, or from a higher-level control such as controller 125.

[0044] Each of compute tile 101 and controller 103 can include one or more processors, processing devices, and various types of memory. In some implementations, processors of compute tile 101 and controller 103 include one or more devices such as microprocessors or central processing units (CPUs), application specific integrated circuits (ASICs), or a combination of different processors. Each of compute tile 101 and controller 103 can also include other computing and storage resources, such as buffers, registers, control circuitry, etc. These resources cooperate to provide additional processing options for performing one or more of the determinations, computations, and calculations described in this specification.

[0045] In some implementations, a processing unit(s) of controller 103 executes programmed instructions stored in memory to cause controller 103 and compute tile 101 to perform one or more functions described in this specification. The memory of controller 103 can include one or more non-transitory machine-readable storage mediums. The non-transitory machine-readable storage medium can include solid-state memory’, a read-only memory' (ROM), an erasable programmable read-only memory (e.g., EPROM, EEPROM, or Flash memory), or any other tangible medium capable of storing information, commands, programs, and / or instructions.

[0046] The system 100 receives instructions that define a particular compute operation to be performed by a compute tile 101. In some implementations, a host or other control device of system 100 (e.g., controller 125), can generate sets of compressed sparse parameters (CSP) comprising non-sparse weight values and corresponding mapping vectors, e.g.. anon-zero map (NZM), for a given operation. For example, the host 120 can send, via a host interface, the compressed set of non-sparse parameter values (e.g., non-sparse weights for a neural network layer) to a compute tile 101 for further processing at the tile. The controller 103 can execute programmed instructions to analyze a data stream associated with the receivedATTORNEY DOCKET NO. 56113-0834WO1weights and inputs, including the compressed non-sparse parameters and corresponding mapping vectors.

[0047] In some implementations, the compute tile 101 uses sparsity mitigation techniques, such as the CSP and NZM, to limit the number of sparse or zero value operands (e.g., activations or weights) that are processed at the compute tile 101. The compute tile 101 can execute these sparsity mitigation techniques concurrent with the disclosed techniques for removing or reducing redundancy in quantized multiplications. For example, the sparsity mitigation techniques limit or prevent wasted compute cycles that occur from multiplying sparse operands that have a zero value, while the techniques of the MSQI unit 112 mitigate or prevent redundant multiplication of operands that have certain reoccurring quantized integer values.

[0048] The controller 103 causes instructions, inputs / activations, and weights of an input data stream to be stored and / or processed at the compute tile 101. In some implementations, the controller 103 analyzes an instruction input to the compute tile 101, detects, based on the analysis, a particular operation code (‘"opcode”) encoded in the instruction, and generates, based on the detected opcode, a control signal 107 that triggers the MSQI unit 112 to mitigate or prevent redundant multiplication of certain quantized operands. The opcode can be a single encoded value that is read by the controller 103 to dynamically execute multiplication redundancy mitigation at the compute tile 101 for certain quantized inference workloads. Based on the opcode, the controller 103 can activate special-purpose data path logic that uses certain comparator logic / circuits and reorder buffers of the MSQI unit 112, compute cells 115 0 / 1 / n to perform multiplication redundancy mitigation at the compute tile 101.

[0049] Fig. 2 shows an example hardware MSQI unit 200 that includes (or represents) the MSQI unit 112 described above with reference to Fig. 1. The hardware MSQI unit 200 can be implemented as a MSQI hardware unit on a special-purpose hardware integrated circuit, such as machine-learning (ML) hardware, tensor processing unit (TPU), neural network processor, or related special-purpose processor. As will be described below, the hardware MSQI unit 200 is designed to optimize multiplication operations in hardware ML inference accelerators by minimizing or eliminating redundant computations (e.g.. multiplications) on prevalent quantized formats. The formats include integer quantization used to execute Large Language Models (LLMs) and / or Generative Al (GenAI) applications in edge devices such as smartphones, tablets, and laptops.

[0050] The hardware MSQI unit 200 includes a scheduler 202, a Quantized Product Reorder Buffer (QPRB) 204, one or more multiply accumulate cells (MACs) 115-0, 115-1,ATTORNEY DOCKET NO. 56113-0834WO1115-2, 115-3 (collectively “MACs 115-0 / 1 / 2 / 3’"), and an accumulator circuit 210. The accumulator circuit 210 is a partial sum adder that sums accumulated values or partial sums generated by the at least one of MACs 115-0 / 1 / 2 / 3. The accumulator circuit 210 can be a hardware circuit that is integrated in the computational unit 114, as indicated in the example of Fig. 2. In some implementations, the accumulator circuit 210 is separate and / or distinct from the computational unit 114.

[0051] The QPRB 204 can include one or more storage banks 206-0. 206-1, 206-2, 206-3 (collectively '‘storage banks 206-0 / 1 / 2 / 3”). In some implementations, each of the storage banks 206-0 / 1 / 2 / 3 is an individual buffer unit (e.g., a reorder buffer) that includes a predefined set of storage locations that correspond to individual entries of the storage bank 206- (or buffer). The buffer is configured to store distinct subsets of data items that each represent an individual entry of the buffer. Each subset of data items can include: i) a first operand of an operand pair (e.g., an activation); ii) a second operand of the operand pair (e.g., an activation); iii) a product of multiplying the first and second operands; iv) and a product flag indicating availability of the product. Each of the storage banks 206-0 / 1 / 2 / 3 can store multiple entries.

[0052] In the example of Fig. 2, each entry of a storage bank is represented as a row in the storage bank. Each entry includes an operand pair, a product (e.g., the product of the operand pair), and a product_available flag. The operand pair includes an activation that represents an input to a neural network layer and a weight that represents a parameter of the neural network layer with which the activation is to be multiplied in order to carry out neural network inference computations. In some implementations, each of the activation and the weight is quantized integer having a discrete set of unique integer combinations based on a bit-width used to represent the quantized integer. The product can be the product of multiplying the activation of the operand pair with the weight of the operand pair.

[0053] The product stored in a given entry can have been previously computed, for example, by one of the MACs 115-0 / 1 / 2 / 3 as will be described below. In some entries, the product may be an empty field or other value indicating that the product of the operand pair included in the entry has not yet been computed. The product available flag can be a binary indication of whether the product of the operand pair stored in the entry has already been computed and stored in the entry. For example, a value of 1 or “True” for the product_available flag can indicate that the product of the operand pair stored in the entry has already been computed and stored in the entry. Meanwhile, a value of 0 or “False” for the product_available flag can indicate that the product of the operand pair stored in the entry hasATTORNEY DOCKET NO. 56113-0834WO1not already been computed and stored in the entry. In the example of Fig. 2, one entry7of storage bank 206-0 is shown to include an operand pair including an activation Al and a weight Wl, a product Pl, and a product_available flag PA.

[0054] Each of the operand pair, the product, and the product available flag of a given entry can be stored in any suitable format. For example, the activation and the weight of the operand pair can be stored as either 4-bit integers or 8-bit integers. The product of the operand pair can be stored as an integer with a size equal to the sum of the size of the activation, the size of the weight, and one additional bit. The product_available flag can be stored as a 1-bit flag.

[0055] In some implementations, the storage banks 206-0 / 1 / 2 / 3 can be pre-populated with entries that include desired operand pairs. For example, the desired operand pairs can be operand pairs the products of which are useful for performing downstream neural network inference computations.

[0056] The QPRB 204 can also include a set of comparators 207 for each of the storage banks 206-0 / 1 / 2 / 3. In particular, each of storage banks 206-0 / 1 / 2 / 3 can have a comparator circuit associated with each of its entries. For each entry' in one of storage banks 206-0 / 1 / 2 / 3, the associated comparator 207 performs parallel comparisons between the stored values in the entry' and a broadcasted weight-activation pair (e.g., a pair of operands including an activation and a weight as described above). For example, the comparator 207 compares the broadcasted activation to the activation stored in the entry and compares the broadcasted weight to the weight stored in the entry. Each comparator 207 in a set of comparators can be implemented as an individual comparator circuit, logic gate, or both. Thus, comparator 207 is referred to alternatively herein as comparator circuit 207.

[0057] The QPRB 204 can also include a multiplexer (MUX) 214. The MUX 214 routes operand pairs to a corresponding MAC 115-0 / 1 / 2 / 3 based on control signals generated by the scheduler 202 (or the comparators) and transmitted or passed to the MUX 214. The comparator circuitry of the QPRB 204 can include one or more XOR gates 209 that generate a control signal 211, e.g., based on control logic of the scheduler 202. In some implementations, the control signal 211 is a multi-purpose HIT signal.

[0058] For example, the HIT signal 211 can be used as: i) a select or enable signal 214m that controls a signal routing function of MUX 214, or ii) a buffer push signal that causes or triggers an operand pair to be pushed to a storage bank 206-0 / 1 / 2 / 3 for storage as a new entry' in the storage bank. In some implementations, a comparator circuit 209 of the MSQI 112 determines that an operand pair does not match an existing operand pair in a product buffer ofATTORNEY DOCKET NO. 56113-0834WO1the compute tile 101. The comparator circuit 209 can then generate a buffer push signal 213 that causes the operand pair that did not match an existing operand pair to be pushed to a reorder buffer to be stored as a new entry in that buffer.

[0059] In some implementations, the HIT signal is used as a product bypass control signal 215 that triggers bypass product logic 218 of the scheduler 202. The bypass product logic 218 is configured to bypass executing a multiplication operation and pass a precomputed product 216 from a storage bank 206-0 / 1 / 2 / 3 to the accumulator circuit 210. In some implementations, the scheduler 202 can generate individual control signals to independently perform one or more of the preceding functions that are described with reference to HIT signal 211.

[0060] The QPRB 204 can be implemented as an in-order buffer, meaning that it does not reorder received weights and activations, rather the QPRB 204 stores weights and activations in the order in which they are received. The QPRB 204 can have any suitable size. For example, the QPRB 204 can have at least 1,000 entries stored across multiple storage banks. In some implementations, larger QPRB sizes are selectively used to further enhance performance of the QPRB, for example, by increasing the likelihood of finding matching entries of pre-computed products to further reduce the number of redundant computations that are performed, as described below). In some implementations, the size of the QPRB 204 can be adjusted based on design preference, including the hardware on which it is implemented and / or the model architecture with which it is implemented, in order to find an optimal balance of the benefits of different QPRB sizes.

[0061] In the example of Fig. 2, the MACs 115-0 / 1 / 2 / 3 can include or represent the MACs 115-0 / 1 / 17 described above in reference to Fig. 1. Each ofMACs 115-0 / 1 / 2 / 3 performs computations (e.g., multiplications) that produce partial sums or accumulated values for generating outputs for a given neural network layer. For example, the MACs 115-0 / 1 / 2 / 3 can execute Single Instruction, Multiple Data (SIMD) operations for different input or output layers of a neural network. The MACs 115-0 / 1 / 2 / 3 perform computations independently and in parallel. In some implementations, each MAC is configured to perform quantized multiplications with both 4-bit integers and 8-bit integers.

[0062] The scheduler 202 can receive multiple operand pairs 212, where each operand pair 212 can include an activation and a weight. In some implementations, the operand pair can already be stored in a local Static Random- Access Memory (SRAM) and fetched from the SRAM to be received by the scheduler 202. In some implementations, the activation andATTORNEY DOCKET NO. 56113-0834WO1the weight of each operand pair 212 are respective quantized integers having a discrete set of unique integer combinations based on a bit-width used to represent the quantized integer.

[0063] The scheduler 202 can transmit an operand pair to each of the storage banks 206 0 / 1 / 2 / 3. At each storage bank, the operand pair is compared to each entry of the storage bank. In particular, the comparator circuit 207 associated with each entry executes compare operations between individual operands of the operand pair and corresponding individual operands of the entry. For example, the comparator circuit 207 associated with each entry can compare the activation of the entry to the activation included in the operand pair, and can compare the w eight of the entry to the w eight included in the operand pair.

[0064] In examples where the comparator circuit 207 is implemented using one or more XOR gates, a first XOR gate of the comparator circuit 207 can be used to compare the activation of the entry to the activation included in the operand pair, whereas a second XOR gate of the comparator circuit 207 can be used to compare the weight of the entry to the weight included in the operand pair. For each received operand pair, the respective comparator circuits 207 that are each associated with its corresponding storage bank 206 0 / 11213 (or reorder buffer) can perform the compare operations in parallel over a single clock cycle. In other words, multiple compare operations can be performed in parallel (e.g., concurrently), across multiple reorder buffers, over a single clock cycle.

[0065] If, upon execution of the compare operations by the comparator circuit 207 for a given entry, a match is found between the received operand pair and the entry, the comparator circuit 207 generates a compare signal that indicates that the individual operands of the operand pair match the corresponding individual operands of the entry7. The product included in the entry is extracted and forwarded directly to the partial sum adder 210. In some implementations, the product is extracted based on a product control signal generated by the scheduler 202.

[0066] For example, the scheduler 202 can be configured to generate a control signal that is passed to a reorder buffer (e.g., storage bank 206-0) that stores an existing operand entry that includes a corresponding pre-computed product. The control signal causes or triggers the storage bank to output the pre-computed product as a product signal 216. The scheduler 202 generates its control signal, including the product signal 216, based on a product available flag stored in the existing entry' of a reorder buffer 206. In some implementations, a product signal 216 includes a data structure of bits that represents a pre-computed product stored as an entry in a storage bank of the QPRB 204.ATTORNEY DOCKET NO. 56113-0834WO1

[0067] A match can be defined as an instance in which all of the following three conditions are true: i) the activation included in the received operand pair is equal to the activation of the entry, ii) the weight included in the received operand pair is equal to the weight of the entry, and iii) the product_available flag for the entry indicates that the product of the operand pair stored in the entry7has already been computed and stored in the entry. In some implementations, the determination of match is based on a matching activation and weight, without reference to a binary' value of the product available flag.

[0068] When a match is found for a given entry, the scheduler 202 is configured to: i) extract the product stored in that entry7and ii) forward the product directly to the partial sum adder 210 to generate accumulated values. Extracting and providing the product to the partial sum adder 210 for subsequent computations enhances throughput at a compute tile 101 and improves power efficiency of the compute tile 101, as discussed above. For example, the enhanced throughput and power efficiency is achieved by avoiding the need for the MACs 115 to compute the product of the operand pair a second time, as that same product was already computed and stored in the storage bank at an earlier instance of executing a given task or workload.

[0069] In some implementations, when a match is found for a given entry, the scheduler 202 selects a second operand pair 212 and performs related compare operations on that operand pair 212. The scheduler 202 generates a control signal 214m that controls the MUX 214 to route the second operand pair to a corresponding MAC 115-0 / 1 / 2 / 3 for computation of the product of the second operand pair. This can be advantageous if there are a limited number of comparator circuits 207 for each of the storage banks 206-0 / 1 / 2 / 3. For example, if the number of comparator circuits 207 is limited, there may not be a sufficient number of comparators to execute compare operations on the second operand pair. Therefore, the second operand pair can be routed directly to one of the MACs 115-0 / 1 / 2 / 3 for computation of the product of the second operand pair.

[0070] If no match is found upon execution of the compare operations by the comparator circuit 207 associated with some (or all) of the entries across the storage banks 206-0 / 1 / 2 / 3, then the MUX 214 routes the operand pair to one of the MACs 115-0 / 1 / 2 / 3 for computation of the product 220 of the operand pair. For a given entry in a storage bank, the scheduler 202 can determine that no match is found when the three conditions (described above) that define a match are not met.

[0071] In some implementations, if a product of an operand pair is computed by a MAC 115-0 / 1 / 2 / 3 and that product is not a prior computed product that is stored as entry in aATTORNEY DOCKET NO. 56113-0834WO1storage bank 206, then the operand pair and that computed product 220 are routed to a particular storage bank 206 mapped to the MAC 115-0 / 1 / 2 / 3 that computed product. This operand pair and corresponding product 220 are stored as a new entry in that particular storage bank 206. A product available flag is then set for that entry to indicate a prior-computed product is available to be used in subsequent computation as an existing product (or partial sum). When a subsequent operand pair is later routed by the scheduler 202 at the compute tile 101, a comparator 207 associated with this or other entries of a storage bank executes compare operations and performs a comparison between individual operands (e.g., quantized integer values) of this subsequent operand pair and corresponding individual operands of the entry of the storage bank, in the same manner described above.

[0072] If, upon execution of the compare operations by the comparator 207 for a given entry, a match is found between the subsequent operand pair and the entry, control logic of the compute tile 101 can generate a control signal to indicate that a pre-computed product is available. For example, a product_available bit can be set to true and a product (or partial sum) field / signal for a cunent compute operation can be updated to include a product value corresponding to the pre-computed product that is broadcast from the entry. As noted above, a match can occur when: i) the activation included in the received operand pair is equal to the activation of the entry in the storage bank 206 and ii) the w eight included in the received operand pair is equal to the weight of the entry in the storage bank 206. In some implementations, when an operand match is detected by the comparator, product available flag for the entry can be used to indicate that pre-computed product for that operand pair is stored in the entry.

[0073] As noted above, for an operand pair and that did not previously exist in a storage bank 206-0 / 1 / 2 / 3, the product 220 of that operand pair is computed by one of the MACs 115-0 / 1 / 2 / 3. The operand pair and corresponding product 220 are then stored as a new entry in the storage bank 206-0 / 1 / 2 / 3. The control logic of the scheduler 202 is also configured to update the value for the product_available flag to indicate that the product 220 of the operand pair is now available as a pre-computed product and stored in the entry. In some implementations, the product available flag is updated by changing a binary value for the product available flag from “0” to “1,” or from “False” to “True”. Thus, going forward, any subsequent operand pair transmitted by the scheduler 202 that comparator 207 identifies as matching the preceding new entry in the storage bank 206-0 / 1 / 2 / 3 will not require a redundant multiplication operation. Instead, as described above, the product stored in the newly added entry will be transmitted directly to the partial sum adder 210. This advantageous reuse ofATTORNEY DOCKET NO. 56113-0834WO1pre-computed product values reduces the number of multiplications to be performed by the MACs 115-0 / 1 / 2 / 3 of a compute tile 101. This can help to both improve the throughput and enhance the efficiency of the MSQI unit 200.

[0074] The processes described above can be performed until the product of each quantized operand pair 212 received by the scheduler 202 is determined (e.g., pre-computed), stored as a buffer entry, and sent to the partial sum adder 210. In some implementations, the operand pairs 212 can correspond to multiplications to be performed inside an inference accelerator. Thus, the processes described above can be used to improve the efficiency and throughput of multiplications to be performed inside an inference accelerator.

[0075] Fig. 3 shows an example sub-system 300 that includes a quantized product buffer (QPB) entry 302 and related aspects of the quantized product reorder buffer (QPRB) 204 of the MSQI 112. Additionally, the sub-system 300 includes a local controller 103 (e.g., a microcontroller) and weight and activation Static Random- Access Memories (SRAMs) 314. In the example of Fig. 3, the weight and activation SRAMs 314 correspond to the memory 108 and memory 110 described above with reference to Fig. 1. For instance, memory 108 can be an example activation SRAM 314, whereas memory 110 can be an example weight SRAM 314.

[0076] The QPB entry7302 represents an entry7in one of the storage banks 206-0 / 1 / 2 / 3 described above with reference to Fig. 2. The QPB entry 302 can include a weight 304, an activation 306. a product available flag 308, and a product 310. The weight 304 can be a parameter of a neural network layer, and the activation 306 can be an input activation to the neural network layer. In particular, the weight 304 and the activation 306 are respective operands to be multiplied for the purpose of performing neural network inference computations.

[0077] The product 310 can include the product of the multiplication of the weight 304 and the activation 306 to be used for performing neural network inference computations. The product 310 can alternatively be an empty field if the product of the weight 304 and the activation 306 has not yet been computed. The product_available flag 308 can be a binary indication of whether the product of the operand pair stored in the entry has already been computed and stored in the entry7, as described above with reference to Fig. 2.

[0078] The local microcontroller 103 can include or represent the controller 103 described above with reference to Fig. 1. The local microcontroller 103 provides a Single Instruction, Multiple Data(SIMD) instruction 112 to cause weights and activations, such asATTORNEY DOCKET NO. 56113-0834WO1the weight 304 and the activation 306 stored in the QPB entry 302, to be stored at or accessed from the weight and activation SRAMs 314.

[0079] The SIMD instruction 112 can include one or more SIMD instructions. For example, the one or more SIMD instructions can include an instruction that causes weights and activations to be fetched from the weight and activations SRAMs 314 by the MSQI 112 (step 316 in Fig. 3). Upon fetching the weights and activations, the MSQI 112 can determine products of weight-activation pairs by either routing the weight-activation pair to a multiply accumulate cell (e.g., a MAC 115-0 / 1 / 2 / 3) for compute or extracting an available precomputed product of the weight-activation pair from a Quantized Product Reorder Buffer (QPRB) that contains entries such as QPB entry 302. For example, the MSQI 112 can determine products of weight-activation pairs using a process similar to that described above with reference to Fig. 2.

[0080] Fig. 4 shows an example operational framework 400 for executing MSQI computations on a compute tile 101 in the computing system of Fig. 1.

[0081] First, the MSQI 112 generates a control signal 402 that causes an entry including an operand pair to be transmitted ('‘popped”) from a memory location in which it is stored. The operand pair can include a weight-activation pair including an activation (e.g., an input to a neural network layer) and a weight (e.g., a parameter of the neural network layer). The memory location can include the first memory 108 and the second memory 110 described above with reference to Fig. 1.

[0082] Next, control logic of the MSQI 112 determines whether the product of the operand pair included in the popped entry is available in a Quantized Product Reorder Buffer (QPRB) of the MSQI 112 (step 404 of Fig. 4). For example, the MSQI 112 can cause compare operations to be executed that compare the weight-activation pair of the operand pair included in the popped entry to weight-activation pairs stored in the QPRB. The QPRB can include multiple QPB entries such as the QPB entry 302 of Fig. 3. Each of the weightactivation pairs stored in the QPRB can be stored in one of the multiple QPB entries. In particular, each weight-activation pair stored in the QPRB can include a weight such as the weight 304 of Fig. 3 and an activation such as the activation 306 of Fig. 3. The MSQI 112 can also check a product_available flag associated with each weight-activation pair, and stored in the same QPB entry as the weight-activation pair, to determine whether the product of the weight-activation pair has already been computed. For example, each product-available flag associated with a weight-activation pair can be the product available flag 308 stored in the QPB entry 302 of Fig. 3.ATTORNEY DOCKET NO. 56113-0834WO1

[0083] The executed compare operations can result in either: 1) a determination that the product of the operand pair included in the popped entry is available in the QPRB; or 2) a determination that the product of the operand pair included in the popped entry is not available in the QPRB. The MSQI 112 can determine that the product of the operand pair included in the popped entry is available in the QPRB if the weight-activation pair of the operand pair included in the popped entry matches the weight-activation pair of one of the QPB entries of the QPRB. and the product available flag associated with the weightactivation pair of the QPB entry of the QPRB indicates that the product of the weightactivation pair of the QPB entry of the QPRB has already been computed. The weightactivation pair of the operand pair included in the popped entry matches the weight-activation pair of a QPB entry of the QPRB if the weight of the weight-activation pair of the operand pair included in the popped entry is equal to the weight of the weight-activation pair of a QPB entry in the QPRB, and the activation of the weight-activation pair of the operand pair included in the popped entry is equal to the activation of the weight-activation pair of the QPB entry of the QPRB.

[0084] If the MSQI 112 determines that the product of the operand pair included in the popped entry is available in the QPRB, control logic of the MSQI 112 generates a product available control signal 406 that causes a prior computed product to be routed to an example output queue 430 of the compute tile 101. The prior computed product can be stored in the QPRB in such a way that it is associated with the weight-activation pair stored in the QPRB that was determined to match the weight-activation pair of the operand pair included in the popped entry'. In particular, the prior computed product can be included in the QPB entry of the QPRB with the weight-activation pair and associated product_available flag described above. For example, the prior computed product can be the product 310 of the QPB entry 302 of Fig. 3.

[0085] The output queue 430 can be a location on the compute tile 101 where computed products of operand pairs are stored for future use in neural network inference computations. In particular, the output queue 430 can be configured to compute partial sums based on the products of operand pairs determined by the MSQI unit 112. The partial sums computed by the output queue 430 can be used for generating outputs for a given neural network layer. For example, the output queue 430 can include or represent the partial sum adder 210 described above with reference to Fig. 2.

[0086] If the MSQI 112 determines that the product of the operand pair included in the popped entry is not available in the QPRB, control logic of the MSQI 112 generates a productATTORNEY DOCKET NO. 56113-0834WO1unavailable control signal 408 that causes the operand pair included in the popped entry to be routed to one of multiple MACs 410 of the compute tile 101. In some implementations, the operand pair included in the popped entry is routed to a MAC 410 of a compute tile 101 by an example multiplexer, such as the MUX 214 described above with reference to Fig. 2. The MAC 410 to which the operand pair is routed computes the product of the operand pair by multiplying the activation and the weight in the operand pair. The MACs 410 can include or represent the MACs 115-0 / 1 / 17 of Fig. 1 and / or Fig. 2. For example, the MSQI 112 generates control signals to compute the product 412 at a first clock cycle, and then uses the computed product 412 as a pre-computed partial sum to avoid performing a redundant multiplication operation that would have otherwise occurred at a subsequent (or second) clock cycle.

[0087] The product 412 of the operand pair that is computed by one of the MACs 410. along with the weight and activation of the operand pair, is transmitted to a broadcast network 418 of the compute tile 101 (step 416 of Fig. 4). The broadcast network 418 can be a network of the compute tile 101 that is configured to broadcast the product of each operand pair computed by the MACs 410, along with the weight and activation included in the operand pair, to the QPRB of the MSQI 112. Thus, once receiving the transmitted product, weight, and activation of the operand pair, the broadcast network 418 broadcasts the product, weight, and activation to the QPRB of the MSQI 112 (step 420 of Fig. 4).

[0088] The MSQI 112 uses the broadcast product, weight, and activation to update QPB entries of the QPRB (step 422 of Fig. 4). For example, the MSQI 112 can cause compare operations to be executed that compare the weight-activation pair of the operand pair to the weight-activation pair stored in each of the QPB entries of the QPRB. The executed compare operations result in either: 1) a match being found between the weight-activation pair of the operand pair and the weight-activation pair of one of the QPB entries stored in the QPRB (‘’the matched entry”); or 2) no match being found between the weight-activation pair of the operand pair and the w eight-activation pair of any of the QPB entries stored in the QPRB.

[0089] A match betw een the w eight-activation pair of the operand pair and the weightactivation pair of one of the QPB entries stored in the QPRB (“the matched entry”) can be found when the weight of the weight-activation pair of the operand pair is equal to the weight of the weight-activation pair of the matched entry, and the activation of the weight-acti vation pair of the operand pair is equal to the activation of the w eight-activation pair of the matched entry.

[0090] If a match is found, the matched entry is updated (step 424 of Fig. 4). Specifically, the product_available flag of the matched entry is set to indicate that the product of theATTORNEY DOCKET NO. 56113-0834WO1weight-activation pair of the matched entry' has already been computed. Additionally, the matched entry is updated to include the product of the operand pair computed by the MACs 410. Thus, after updating, the matched entry7can have a similar format to the QPB entry 302 of Fig. 3 (e.g., the matched entry' can include a product such as the product 310 in addition to an associated weight, activation, and product_available flag).

[0091] If no match is found (e g., the conditions for a match being found described above are not met), then no update operation is performed to update an operand pair (e.g., an existing operand pair) of the QPRB entries with a pre-computed product (step 426 of Fig. 4).

[0092] Fig. 5 is an example quantization format for operands stored as a quantized product entry in the QPRB.

[0093] In particular, the example QPRB entry 502 can be one of multiple entries included in a QPRB, such as the QPRB 204 described above with reference to Fig. 2. Entries of the QPRB can be used to perform neural network inference computations, as described above.

[0094] The QPRB entry 502 display s a number of fields that can be included in an entry of the QPRB and an example format for each field. The QPRB entry 502 includes a field for a weight that can have the format of a 4-bit integer or an 8-bit integer. The weight of the QPRB entry 502 can be a parameter of a neural network layer. For example, the weight of the QPRB entry' 502 can be the weight 304 described above with reference to Fig. 3.

[0095] Similarly, the QPRB entry 502 includes a field for an activation that can have the format of a 4-bit integer or an 8-bit integer. The activation of the QPRB entry’ 502 can be an input to a neural network layer. For example, the activation of the QPRB entry 502 can be the activation 306 described above with reference to Fig. 3.

[0096] The QPRB entry 502 includes a field for a product available flag that can have the format of a 1 -bit flag. The product available flag of the QPRB entry 502 can be a binary indication of whether the product of the weight and the activation stored in the entry has already been computed and stored in the entry'. For example, the product available flag of the QPRB entry’ 502 can be the product_available flag 308 described above with reference to Fig.3.

[0097] The QPRB entry 502 includes a field for a product that can have the format of an integer w ith a size equal to the sum of the size of the activation, the size of the weight, and one additional bit. For example, if the activation and the weight both have formats of 4-bit integers, the product can have the format of an integer w ith a size equal to twice the size of a 4-bit integer plus one additional bit. If the activation and the weight both have formats of 8-bit integers, the product can have the format of an integer with a size equal to twice the sizeATTORNEY DOCKET NO. 56113-0834WO1of an 8-bit integer plus one additional bit. The product of the QPRB entry' 502 can be a product used in neural network inference computations. For example, the product of the QPRB entry 502 can be the product 310 described above with reference to Fig. 3.

[0098] In some implementations, the storage of the weight and the activation of the QPRB entry' 502 as 4-bit or 8-bit integers can be advantageous because it can reduce the space on the QPRB that is required to store the entries, allowing for more compact and efficient manufacture and handling of the QPRB. However, the representation of the weight and the activation as 4-bit or 8-bit integers can lead to duplicate entries of the QPRB. This is because different weights can have the same quantized 4-bit or 8-bit integer representation. Similarly, different activations can have the same quantized 4-bit or 8-bit integer representation. Thus, it is possible for the weight and activation of a first entry of the QPRB to exactly equal a corresponding weight and activation of a second entry of the QPRB, even though the weight and activation of the first entry are distinct from the weight and activation of the second entry . This can lead to the first entry of the QPRB being identical to the second entry of the QPRB. For example, if the weights and activations of the two entries are equal, then the products and product available flags of the two entries will likely be equal.

[0099] Duplicate entries of the QPRB, such as in the scenario described above, that arise from the quantization of the weights and activations can lead to wasted power consumption by requiring the circuit of which the QPRB is part to unnecessarily perform redundant multiplication operations. However, the technologies disclosed herein provide the advantage of avoiding the need for such redundant operations, thereby reducing power consumption, and enhancing throughput and efficiency of the circuit, as described above.

[0100] The initialization code 504 is an example of a number of lines of computer code that can be used to initialize an entry of a QPRB, such as the QPRB 204 described above with reference to Fig. 2.

[0101] The update code 506 is an example of a number of lines of computer code that can be used to update an entry of a QPRB, such as the QPRB 204 described above with reference to Fig. 2. In particular, the update code 506 can be used to update an entry of a QPRB based on the computed product of an operand pair using the processes described above with reference to Fig. 4. For example, the update code 506 checks if the weight of a given entry of the QPRB is equal to the weight of the operand pair, and if the activation of the given entry' is equal to the activation of the operand pair. If both of these conditions are met, the update code 506 sets the product field of the given entry to be equal to the computed product of theATTORNEY DOCKET NO. 56113-0834WO1operand pair, and sets the product_available flag to indicate that the product of the weight and activation of the given entry has already been computed and stored in the entry.

[0102] Fig. 6 is an example process 600 for removing redundancy in quantized multiplications executed on the compute tile of Fig. 1.

[0103] Process 600 is also implemented or executed at system 100 using at least the compute tile 101(including the multiplication scheduler for quantized inference (MSQI) 112) and / or the partial sum adder 210 described above with reference to Fig. 1 and Fig. 2. Hence, descriptions of process 600 will reference the above-mentioned computing resources of system 100. In some examples, the steps or actions of process 600 are enabled by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non-transitory machine-readable storage device and is executable by one or more of the processors or other resources described in this specification.

[0104] Referring again to process 600, the system 100 receives, at the compute tile, an operand pair comprising an activation and a weight (602). For example, the operand pair can be one of the operand pairs 212 described above with reference to Fig. 2. In some implementations, the operand pair is received by a scheduler of the compute tile from a local Static Random-Access Memory (SRAM).

[0105] The system 100 identifies, in a product buffer of the compute tile, an existing operand pair comprising an existing activation and an existing weight stored in the product buffer (604). For example, the product buffer of the compute tile can be one of the storage banks 206-0 / 1 / 2 / 3 described above with reference to Fig. 2. In particular, the product buffer can include multiple entries, each entry including an existing operand pair, a product, and a binary' flag indicating the availability' of the product.

[0106] In some implementations, the system 100 can use multiple comparator circuits, such as the comparator circuits 207 described above with reference to Fig. 2, to identify the existing operand pair. In particular, there can be a comparator circuit for each entry' of the product buffer that performs compare operations between the entry of the product buffer and the operand pair received by the system. The compare operations performed between the entry of the product buffer and the operand pair received by the system are described in more detail above with reference to Fig. 2.

[0107] The existing operand pair can be identified by performing the compare operations for each entry' of the entries of the product buffer. In particular, the existing operand pair can be identified as the operand pair stored in the entry of the product buffer for which a match isATTORNEY DOCKET NO. 56113-0834WO1found between the operand pair received by the system and the operand pair stored in the entry, as defined above with reference to Fig. 2.

[0108] The system 100 determines, by a comparator circuit of the compute tile, that the operand pair matches the existing operand pair (606). For example, as described above with reference to Fig. 2, the operand pair can be determined to match the existing operand pair if the existing activation is equal to the activation of the operand pair, the existing weight is equal to the weight of the operand pair, and the binary flag of the entry of the existing operand pair indicates that the product for the existing operand pair is available.

[0109] In some implementations, the comparator circuit of the compute tile determines that the operand pair matches the existing operand pair by comparing the activation of the operand pair and the existing activation of the existing operand pair. Based on this comparison, the comparator circuit generates an activation control signal indicating that the activation matches the existing activation. In some implementations, the comparator circuit of the compute tile determines that the operand pair matches the existing operand pair by comparing the weight of the operand pair and the existing weight of the existing operand pair. Based on this comparison, the comparator circuit generates a weight control signal indicating that the weight matches the existing weight.

[0110] The system 100 obtains, at the compute tile, a product of multiplying the existing activation and the existing weight of the existing operand pair (608). For example, upon finding that the operand pair matches the existing operand pair, the product stored in the entry of the product buffer in which the existing operand pair is located can be extracted, as described above with reference to Fig. 2.

[0111] The system 100 computes an output for a layer of the neural network using the obtained product (610). For example, upon extracting the product for the existing operand pair from the entry of the product buffer in which it is located, the system 100 can forward it directly to the partial sum adder 210. The partial sum adder 210 can then use the forwarded product to compute an output for a layer of the neural network.

[0112] In some implementations, the respective steps of process 600 are performed at a hardware integrated circuit as part of a larger compute operation to generate a machinelearning (ML) output, including an output for a neural network layer of a neural network that implements one or more ML models. For example, the output can be a portion of a computation for a ML task or inference workload to generate an image processing, speech processing, or image recognition output. As indicated above, a portion of the integrated circuit can include a special-purpose neural network processor or hardware ML acceleratorATTORNEY DOCKET NO. 56113-0834WO1configured to accelerate computations for generating different types of data processing outputs.

[0113] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus.

[0114] Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0115] The term “computing system” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0116] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0117] A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, subATTORNEY DOCKET NO. 56113-0834WO1programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0118] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as. special purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (General purpose graphics processing unit).

[0119] Computers suitable for the execution of a computer program, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. Some elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0120] Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory¬ devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry-.

[0121] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can beATTORNEY DOCKET NO. 56113-0834WO1received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0122] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

[0123] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0124] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0125] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in theATTORNEY DOCKET NO. 56113-0834WO1embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0126] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

ATTORNEY DOCKET NO. 56113-0834WO1What is claimed is:

1. A method performed using a compute tile of a hardware integrated circuit configured to implement a neural network, the method comprising:receiving, at the compute tile, an operand pair comprising an activation and a weight; identifying, in a product buffer of the compute tile, an existing operand pair comprising an existing activation and an existing weight stored in the product buffer;determining, by a comparator circuit of the compute tile, that the operand pair matches the existing operand pair;obtaining, at the compute tile, a product of multiplying the existing activation and the existing weight of the existing operand pair; andcomputing an output for a layer of the neural network using the obtained product.

2. The method of claim 1 , wherein determining that the operand pair matches the existing operand pair comprises:comparing, by the comparator circuit, the activation of the operand pair and the existing activation of the existing operand pair; andbased on the comparing of the activation and the existing activation, generating, by the comparator circuit, an activation compare signal indicating the activation matches the existing activation.

3. The method of claim 2, wherein determining that the operand pair matches the existing operand pair comprises:comparing, by the comparator circuit, the weight of the operand pair and the existing weight of the existing operand pair; andbased on the comparing of the weight and the existing weight, generating, by the comparator circuit, a weight compare signal indicating the weight matches the existing weight.

4. The method of claim 1, wherein determining that the operand pair matches the existing operand pair comprises:executing, by the comparator circuit, compare operations between individual operands of the operand pair and corresponding individual operands of the existing operand pair; computing a respective result for each of the compare operations; andATTORNEY DOCKET NO. 56113-0834WO1generating, by the comparator circuit, a compare output signal based on the respective result for each of the compare operations.

5. The method of claim 4, wherein the compare output signal indicates the individual operands of the operand pair matches the corresponding individual operands of the existing operand pair.

6. The method of claim 4, wherein the executing compare operations comprises:executing each compare operation in parallel over a single clock cycle using a respective individual comparator of the comparator circuit.

7. The method of claim 1 , wherein obtaining the product comprises:detecting a product flag corresponding to the existing operand pair in response to determining that the operand pair matches the existing operand pair; andobtaining the product corresponding to the existing operand pair based on the detected product flag.

8. The method of claim 7, wherein obtaining the product corresponding to the existing operand pair based on the detected product flag comprises:generating, by a scheduler of the compute tile, a product control signal that causes the product buffer to output a product of multiplying operands of the existing operand pair.

9. The method of claim 1, wherein:the product buffer includes a plurality of entries, andeach entry of the plurality of entries corresponds to a memory location configured to store an activation, a weight, a product flag, and a product of multiplying the activation and the weight.

10. The method of claim 1, wherein the operand pair is a first operand pair among a plurality of operand pairs that are received at the compute tile, and the method further comprises:selecting, by a scheduler of the compute tile, a second operand pair from among the plurality of operand pairs in response to determining that the first operand pair matches the existing operand pair;ATTORNEY DOCKET NO. 56113-0834WO1generating, at the compute tile, a control signal that controls a multiplexer (MUX) of the compute tile; androuting, by the MUX and based on the control signal, the second operand pair to a multiplier-and-accumulator cell (MAC) of the compute tile.

11. The method of claim 10, further comprising:computing a product of multiplying an activation and a weight of the second operand pair using the MAC.

12. The method of claim 10, wherein the compute tile comprises a computing unit that includes a plurality of MACs.

13. The method of claim 1, wherein each operand of the operand pair and each operand of the existing operand pair is a quantized integer having a discrete set of unique integer combinations based on a bit-width used to represent the quantized integer.

14. The method of claim 1 , wherein receiving the operand pair comprises:receiving plurality of operand pairs, each of the plurality of operand pairs comprising a respective activation and a corresponding weight.

15. A hardware integrated circuit configured to implement a neural network, the hardware integrated circuit comprising:a compute tile comprising:a scheduler configured to receive an operand pair comprising an activation and a weight;a product buffer configured to store:i) one or more existing operand pairs, each of the one or more existing operand pairs comprising a respective existing activation and a respective existing weight; andii) for each of the one or more existing operand pairs: a respective product of multiplying the existing activation and the existing weight of the existing operand pair; anda comparator circuit configured to determine whether the operand pair matches an existing operand pair of the one or more existing operand pairs.ATTORNEY DOCKET NO. 56113-0834WO116. The hardware integrated circuit of claim 15, wherein the comparator circuit is configured to:compare the activation of the operand pair and the existing activation of the existing operand pair;determine that the operand pair matches the existing operand pair based on the compare of the activation and the existing activation; andgenerate an activation compare signal indicating the activation matches the existing activation.

17. The hardware integrated circuit of claim 16. wherein the comparator circuit is configured to:compare the weight of the operand pair and the existing weight of the existing operand pair;determine that the operand pair matches the existing operand pair based on the compare of the weight and the existing weight; andgenerate a weight compare signal indicating the weight matches the existing weight.

18. The hardware integrated circuit of claim 15, wherein the comparator circuit is configured to:execute compare operations between individual operands of the operand pair and corresponding individual operands of the existing operand pair;compute a respective result for each of the compare operations; andfor each of the compare operation: generate a compare output signal based on the respective result for the compare operation.

19. The hardware integrated circuit of claim 18, wherein the compare output signal indicates the individual operands of the operand pair matches the corresponding individual operands of the existing operand pair.

20. The hardware integrated circuit of claim 18, wherein:i) the comparator circuit comprises a first comparator and a second comparator; ii) each of the first comparator and the second comparator is configured to execute a respective compare operation; andATTORNEY DOCKET NO. 56113-0834WO1iii) the comparator circuit is configured to execute the respective compare operations in parallel over a single clock cycle using the first comparator and the second comparator.

21. The hardware integrated circuit of claim 16, wherein the product buffer is configured to store, for each of the one or more existing operand pairs, a product flag corresponding to the existing operand pair.

22. The hardware integrated circuit of claim 22, wherein the compute tile is configured to obtain the respective product of the existing operand pair based on the product flag corresponding to the existing operand pair when the received operand pair matches the existing operand pair.

23. The hardware integrated circuit of claim 23, wherein:i) the scheduler is configured to generate a product control signal that causes the product buffer to output the respective product of the existing operand pair; andii) the compute tile obtains the respective product of the existing operand pair based on product flag and the product control signal.

24. The hardware integrated circuit of claim 16, wherein:i) the operand pair is a first operand pair among a plurality of operand pairs that are received by the scheduler;ii) the scheduler is configured to select a second operand pair from among the plurality of operand pairs;iii) the compute tile further comprises a multiplexer (MUX) and is configured to generate a control signal that controls a data routing function of the MUX; andiv) the MUX is configured to route the second operand pair to a multiplier-and-accumulator cell (MAC) of the compute tile based on the control signal.

25. The hardware integrated circuit of claim 25. wherein the MAC is configured to compute a product of multiplying an activation and a weight of the second operand pair.

26. The hardware integrated circuit of claim 16, wherein each of the activation and the weight of the operand pair is a quantized integer having a discrete set of unique integer combinations based on a bit-width used to represent the quantized integer.

Images