Executing floating-point model through integer datapath in neural processing unit

Abstract

A neural processing unit (NPU) may perform computations in integer domains to execute a floating-point model. The NPU may include an input delivery unit (IDU), a processing engine, and a post-processing engine. The IDU may convert floating-point weights to integers, e.g., by normalizing the floating-point weights, mapping the normalized floating-point values to normalized integer values using a look-up table, and scaling up the normalized integer values into integer values. The IDU may load the integer values into the processing engine through an integer datapath in the NPU. The processing engine may compute an output tensor of a neural network operation using the integer values. The post-processing engine may perform per-channel quantization of the output tensor using channel-specific quantization parameters stored in configuration registers of the post-processing engine. The look-up table and configuration registers may be programmed with configuration parameters determined by a compiler.

Description

EXECUTING FLOATING-POINT MODEL THROUGH INTEGER DATAPATH IN NEURAL PROCESSING UNITCross-Reference to Related Application

[0001] This application claims the benefit of U.S. Provisional Patent Application No.63 / 719,508, filed November 12, 2024, and entitled "ENABLING FP4 MODELS IN NEURAL NETWORK SYSTEMS," which is incorporated by reference in its entirety for all purposes.Technical Field

[0002] This disclosure relates generally to neural networks (also referred to as "deep neural networks" or "DNN"), and more specifically, executing floating-point DNNs through integer datapaths in neural processing units (NPUs).Background

[0003] DNNs are used extensively for a variety of artificial intelligence (Al) applications ranging from computer vision to speech recognition and natural language processing due to their ability to achieve high accuracy. However, the high accuracy comes at the expense of significant computation cost. DNNs have extremely high computing demands as there can be a large number of operations as well as a large amount of data to read and write.Therefore, techniques to improve efficiency of DNNs are needed.Brief Description of the Drawings

[0004] Embodiments can be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0005] FIG. 1 illustrates an example DNN, in accordance with various embodiments.

[0006] FIG. 2 illustrates an example convolution, in accordance with various embodiments.

[0007] FIG. 3 is a block diagram of an Al system, in accordance with various embodiments.

[0008] FIG. 4 is a block diagram of a DNN module, in accordance with various embodiments.

[0009] FIG. 5 illustrates mapping floating-point data types to integral data types, in accordance with various embodiments.

[0010] FIG. 6 illustrates per-channel quantization, in accordance with various embodiments.

[0011] FIG. 7 illustrates an input delivery unit (IDU) with a palletization unit, in accordance with various embodiments.

[0012] FIG. 8 illustrates an example processing element (PE), in accordance with various embodiments.

[0013] FIG. 9A illustrates an example sparse cell, in accordance with various embodiments.

[0014] FIG. 9B illustrates an example sparse cell array, in accordance with various embodiments.

[0015] FIG. 10 illustrates an example post-processing engine, in accordance with various embodiments.

[0016] FIG. 11 is a flowchart of a method of executing a DNN, in accordance with various embodiments.

[0017] FIG. 12 is a block diagram of an example computing device, in accordance with various embodiments.Detailed DescriptionOverview

[0018] The last decade has witnessed a rapid rise in Al based data processing, particularly based on DNNs. DNNs are widely used in the domains of computer vision, speech recognition, image, and video processing mainly due to their ability to achieve beyond human-level accuracy. A DNN typically includes a sequence of layers. A DNN layer may include one or more operations, such as convolution, interpolation, layer normalization, batch normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on. These operations are referred to as deep learning operations or neural network operations.

[0019] Neural network operations may be tensor operations. Input or output data of neural network operations may be arranged in data structures called tensors. Taking a convolutional layer for example, the input tensors include an activation tensor (also referred to as "input feature map (IFM)" or "input activation tensor") including one or more activations (also referred to as "input elements") and a weight tensor. The weight tensor may be a kernel (a 2D weight tensor), a filter (a 3D weight tensor), or a group of filters (a 4Dweight tensor). A convolution may be performed on the input activation tensor and weight tensor to compute an output activation tensor in the convolutional layer.

[0020] A tensor is a data structure having multiple elements across one or more dimensions. Examples of tensors include vector (which is one-dimensional (ID) tensor), matrix (which is two-dimensional (2D) tensor), three-dimensional (3D) tensors, fourdimensional (4D) tensors, and even higher dimensional tensors. A dimension of a tensor may correspond to an axis, e.g., an axis in a coordinate system. A dimension may be measured by the number of data points along the axis. The dimensions of a tensor may define the shape of the tensor. A DNN layer may receive one or more input tensors and compute an output tensor from the one or more input tensors. In some embodiments, a 3D tensor may have an X-dimension, a Y-dimension, and Z-dimension. The X-dimension of a tensor may be the horizontal dimension, the length of which may be the width of the tensor; the Y-dimension may be the vertical dimension, the length of which may be the height of the tensor; and the Z-dimension may be the channel dimension, the length of which may be the number of channels. The coordinates of the elements along a dimension may be integers in an inclusive range from 0 to (L — 1), where L is the length of the tensor in the dimension. For instance, the x coordinate of the first element in a row may be 0, the x coordinate of the second element in a row may be 1, and so on. Similarly, the y coordinate of the first element in a column may be 0, the y coordinate of the second element in a column may be 1, and so on. A 4D tensor may have a fourth dimension, which may indicate the number of batches in the operation.

[0021] The evolution of Al is increasingly driven by the demand for higher performance, lower power consumption, and greater efficiency in processing large volumes of data. FP4 precision is a significant innovation, offering the potential to revolutionize neural network computations by drastically reducing the memory footprint and enhancing the speed of operations. This reduction in bit precision can directly translate into lower power usage and faster computations, critical for battery-powered devices and large-scale data centers alike. However, the integration of FP4 precision into currently available NPUs faces substantial obstacles. Currently available neural processing units (NPUs) are typically optimized for higher-precision formats, such as INT9, FP16, and FP8. However, since FP4 parameters are represented in floating-point, this architectural limitation usually requires executing FP4 data through FP8 / FP16 during computation. Such upscaling can be inherently inefficient. Itcan counteract the benefits of using FP4 by increasing the computational load and energy consumption, thus nullifying the potential gains in efficiency that FP4 offers. Moreover, this approach can impose an additional computational overhead that can significantly degrade the throughput of Al applications, from real-time image processing to complex decisionmaking systems in autonomous vehicles. This inefficient conversion process can not only burden the NPU systems at the edge but also lead to increased operational costs and energy usage, which are antithetical to the trends of modern computational sustainability. As energy efficiency and processing power are usually paramount at the edge, there is a critical need for a method that can fully harness the advantages of FP4 precision without the encumbrances of current NPU design limitations.

[0022] Some currently available methods address the computational challenges posed by FP4 precision within NPUs. One method involves computing FP4 through higher precisions such as FP16 or FP32 during processing to maintain data integrity and minimize computational errors. This technique leverages the wider dynamic range of higher-precision formats but at the cost of increased computational overhead, energy usage, and memory demands, which negates the efficiency benefits of FP4 precision. Uniform quantization across all channels is another approach, which applies the same quantization parameters universally irrespective of varying data distributions across channels. Although this method simplifies the implementation, it often results in loss of model accuracy and sensitivity due to its inability to adapt to specific data attributes. Hardware-based solutions are also used. These solutions involve architectural modifications to NPUs to natively support low-bit operations. These modifications require significant changes to the NPU design, increasing both the cost and complexity of production. Such hardware adaptations provided fixed solutions that lack the flexibility needed to accommodate different Al models and applications dynamically. Many currently available NPU architectures handle FP4 computations by converting them to FP16 or, at best, FP8. Similarly, designs that focus on integer computations tend to compute FP4 by converting them to INT16.

[0023] Despite these advancements, existing solutions continued to present trade-offs between computational efficiency, accuracy, operational flexibility, and economic viability. The above-described solutions for computing FP4 precision within NPUs can offer some benefits but are hindered by significant drawbacks. The traditional method of computing FP4 through FP8, FP16, or FP32 datapath, while preserving data integrity, greatly increasedcomputational overhead, leading to slower processing times and higher energy consumption and undermining the efficiency FP4 aims to achieve. Additionally, uniform quantization approaches often cause a loss of model accuracy by applying the same parameters across all channels, failing to account for the unique characteristics of different data distributions, resulting in information loss and degraded performance. Furthermore, hardware-centric solutions requiring architectural modifications to support low-bit operations can introduce high costs and increased complexity, making NPUs less adaptable to evolving Al model requirements.

[0024] Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by executing neural network operations in floatingpoint models through performing higher-precision computations in integer domains. This disclosure addresses a critical bottleneck in NPU performance by introducing a groundbreaking solution for executing FP4 precision computation directly in NPUs by leveraging integer (e.g., INT9, INT5, etc.) datapaths. The approach in this disclosure can enhance performance, efficiency, and adaptability without driving up costs.

[0025] In various embodiments of the present disclosure, an NPU may include an IDU (which may also be referred to as an ingress delivery unit), a processing engine, a postprocessing engine, and an output delivery unit (ODU). Integer datapaths may be present between these components of the NPU. The NPU may execute floating-point DNNs. A floating-point DNN may have weights with floating-point data types. Examples of floatingpoint data types include FP4 and its variants, such as SF4 and NF4. SF stands for student float. NF stands for normalized float. FP4 in various embodiments described hereinbelow includes SF4, NF4, or other variants of 4-bit floating-point data point. The weights may be stored in a memory of the NPU, and the IDU may deliver the weights to the processing engine for performing neural network operations in the DNN. The IDU can facilitate conversion of the floating-point weights to integers. For instance, the IDU normalizes the weights and maps the normalized floating-point values to normalized integer values by using a look-up table. The look-up table may indicate mapping of values within a normalized floating-point range to values within a normalized integer range. The look-up table, after receiving a normalized floating-point value, may output a normalized integer value. The IDU may scale up the normalized integer value into a value of the target integral data type. After the conversion, the IDU may load the integer values into the processing engine for theprocessing engine to perform a neural network operation. The processing engine may compute an output tensor of the neural network operation using the integer values. The output tensor, which may be an activation tensor, may be quantized by the post-processing engine. The quantization may be per-channel quantization. The post-processing engine may include post-processing elements (PPEs) associated with configuration registers. The configuration registers may be programmed with channel-specific quantization parameters. One or more PPEs may use the quantization parameter specific to a channel to quantize the activations in the channel. The quantized activations may be delivered by the ODU to the memory and may be used as input activations for the next neural network operation of the DNN. The activations may be quantized to a different precision from the quantized weights. In an example, the weights may be converted to INT5 values while the acti activations may be converted to INT9 values.

[0026] Unlike currently available methods requiring FP8 or FP16 datapaths, this disclosure introduces a novel approach to FP4 precision computation within NPUs by executing in higher bit precision in the INT domain (e.g., INT9). The approach in this disclosure can utilize integer datapaths (such as INT5 and INT9 datapaths) in NPUs, achieving memory and compute efficiency while supporting ultra-low-precision integer computation (e.g., INT5 computation) for even greater benefits. Unlike the costly FP domain computation, which increases power consumption, this method can optimize power efficiency while maintaining computational precision. By shifting the execution to the INT domain, the approach in this disclosure can reduce memory bandwidth requirements and computational load, allowing NPUs to achieve higher performance with lower power usage.

[0027] A mixed-precision format, such as using INT5 for weights and retaining INT9 for activations, can further optimize memory savings without compromising accuracy, ensuring a more efficient processing pipeline. This advancement can not only leverage the compact 4-bit precision to achieve substantial memory savings but also reduce the computational overhead associated with FP16. The approach can preserve the low-bit precision advantages of FP4 while substantially reducing computational load and enhancing the TOPS / W ratio. As a paradigm shift in current NPU execution, this method can meet the energy efficiency goals of modern computational architectures and integrate seamlessly into existing designs without extensive modification. It can establish a foundation for the next generation Al accelerators, making them more powerful, faster, greener, and economically viable.

[0028] The approach in this disclosure can support more exotic FP4 variants, such as SF4 and NF4. These formats can provide better accuracy compared to conventional FP4 variants (such as E2M1, E1M2, or E3M0), which are typically favored for their hardware efficiency. By leveraging INT9 computation, this method can not only optimize power and performance but also enhance compatibility with these higher-accuracy FP4 formats, achieving a balance between computational precision and hardware efficiency.

[0029] By harnessing the compactness of 4-bit precision, the proposed framework can achieve notable memory savings, while levering the existing compute efficiency in the INT9 domain. The reduced memory footprint and faster multiply-accumulate (MAC) operation allows for faster data processing speeds and lower energy consumption, while furnishing minimal degradation in accuracy (e.g., <1% for many cases compared to a FP32 baseline accuracy), which are crucial for the deployment of Al applications in energy-sensitive environments.

[0030] The shift to INT9 execution from FP4 as opposed to FP16 can lead up to 4X enhancement in the TOPS / W. This improvement represents a significant leap forward in operational efficiency, offering up to fourfold gains compared to traditional methods. This patent further extends the approach by integrating it with the advanced digital compute-in-memory (DCIM) block available in the NPU6 generation. The DCIM block, offering 4X greater power efficiency than the traditional INT9 engine, allows FP4 models to run with significantly improved performance. By executing FP4 models in the DCIM architecture, the approach in this disclosure can deliver up to 8X improvement in TOPS / Watt compared to FP8, and up to 16X compared to FP16, dramatically enhancing power-performance optimization in NPUs.

[0031] Beyond INT9, the approach in this disclosure can enable models originally designed in FP4 precision to execute in a much lower integer precision, such as INT5, for both weights and activations, leveraging the INT5 compute format supported by many NPUs. By utilizing INT5 precision, the approach in this disclosure can significantly enhance throughput-per-watt efficiency (TOPS / W) by almost 4X compared to computations in INT9 precision, with minimal impact on inference classification accuracy. To further optimize both memory savings and accuracy, a mixed-precision strategy is implemented, where weights are maintained at INT5 precision while activations remain at INT9 precision. This approach can maximize the memory savings for weights while achieving substantially higher accuracy thanan a I l-l NT5 configuration. This dual approach— executing weights and activations in INT5 for even greater efficiency, with INT9 available to optimize accuracy in mixed-precision formats— can significantly reduce computational overhead and memory requirements while enhancing the TOPS / W ratio for optimal energy efficiency and processing speed. By incorporating a per-channel quantization strategy, it ensures accuracy can be precisely preserved across diverse data distributions, establishing a new standard for ultra-low-precision computing in Al applications.

[0032] One of the standout features of the approach in this disclosure can is its compatibility with existing NPU designs, which allows for its rapid adoption without any modification to current architecture. This compatibility can ensure that NPUs can be upgraded to this more efficient processing method without disrupting outgoing design, making it a practical solution for immediate implementation. This disclosure is the first to challenge the prevailing assumption that FP4 data need be upscaled to higher bitwidths for effective computation. It demonstrates that INT datapath is not only sufficient for FP4 computations but also achieves this without requiring major architectural changes or sacrificing significant accuracy, all while delivering remarkable performance-per-watt improvements.

[0033] The approach in this disclosure can enable the execution of modern state-of-the-art Al models in low-bit precision (e.g., INT5 and INT9) quantization, achieving substantial reductions in latency and energy consumption without requiring any additional hardware changes or modifications. By directly mapping FP4 models to these efficient precision formats, this approach can facilitate deployment of advanced Al workloads on existing hardware, eliminating the need for specialized modifications. It can deliver up to a 16X improvement in TOPS / Watt compared to currently available methods, making it ideal for power-sensitive applications such as edge Al and mobile devices.

[0034] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it can be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or / and that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0035] Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0036] Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter.However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0037] For the purposes of the present disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0038] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives "first," "second," and "third," etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0039] In the following detailed description, various aspects of the illustrative implementations are described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0040] The terms "substantially," "close," "approximately," "near," and "about," generally refer to being within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between the elements, generally refer to being within + / - 5-20% of a target value as described herein or as known in the art.

[0041] In addition, the terms "comprise," "comprising," "include," "including," "have," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, device, or DNN accelerators. Also, the term "or" refers to an inclusive "or" and not to an exclusive "or."

[0042] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.

[0043] FIG. 1 illustrates an example DNN 100, in accordance with various embodiments. The DNN 100 may be executed by an NPU, e.g., the NPU 302 in FIG. 3. In an example, the DNN 100 may be a convolution-based DNN. In other examples, the DNN 100 may be other types of DNNs. For the purpose of illustration, the DNN 100 receives an input image 105 that includes objects 115, 125, and 135. The DNN 100 includes a sequence of layers comprising a plurality of convolutional layers 110 (individually referred to as "convolutional layer 110"), a plurality of pooling layers 120 (individually referred to as "pooling layer 120"), and a plurality of fully-connected layers 130 (individually referred to as "fully-connected layer 130"). In other embodiments, the DNN 100 may include fewer, more, or different layers. In an execution of the DNN 100, the layers of the DNN 100 execute tensor computation that includes many tensor operations, such as convolutions, interpolations, pooling operations, elementwise operations (e.g., elementwise addition, elementwise multiplication, etc.), other types of tensor operations, or some combination thereof.

[0044] The convolutional layers 110 summarize the presence of features in inputs to the DNN 100, such as features in the input image 105. The convolutional layers 110 function as feature extractors. The first layer of the DNN 100 is a convolutional layer 110. In an example, a convolutional layer 110 performs a convolution on an input tensor 140 (alsoreferred to as IFM 140) and a filter 150. As shown in FIG. 1, the IFM 140 is a 7x7x33D matrix. The IFM 140 includes 3 input channels, each of which is a 7x72D matrix. The 7x72D matrix includes 7 input data elements (also referred to as "input activations") in each row and 7 data elements in each column. The filter 150 is a 3x3x33D matrix. The filter 150 includes 3 kernels, each of which may correspond to a different input channel of the IFM 140. A kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. A kernel can be smaller than the IFM 140. In the embodiments of FIG. 1, each kernel is represented by a 3x32D matrix. The 3x3 kernel includes 3 weights in each row and 3 weights in each column. Weights can be initialized and updated by backpropagation using gradient descent during the training process. The magnitudes of the weights can indicate importance of the filter 150 in extracting features from the IFM 140.

[0045] The convolution includes multiply-accumulate (MAC) operations with the input elements in the IFM 140 and the weights in the filter 150. The convolution may be a standard convolution 163 or a depthwise convolution 183. In the standard convolution 163, the whole filter 150 slides across the IFM 140. All the input channels are combined to produce an output tensor 160 (also referred to as output feature map (OFM) 160). The OFM 160 is represented by a 5x52D matrix. The 5x52D matrix includes 5 output data elements (also referred to as "output activations") in each row and 5 output elements in each column. For the purpose of illustration, the standard convolution includes one filter in the embodiments of FIG. 1. In embodiments where there are multiple filters, the standard convolution may produce multiple OCs in the OFM 160.

[0046] The multiplication applied between a kernel-sized patch of the IFM 140 and a kernel may be a dot product. A dot product is the elementwise multiplication between the kernelsized patch of the IFM 140 and the corresponding kernel, which is then summed, always resulting in a single value. Using a kernel smaller than the IFM 140 may be intentional as it allows the same kernel to be multiplied by the IFM 140 multiple times at different points on the IFM 140. Specifically, the kernel is applied systematically to each overlapping part or kernel-sized patch of the IFM 140, left to right, top to bottom. The result from multiplying the kernel with the IFM 140 one time may be a single value. As the kernel is applied multiple times to the IFM 140, the multiplication result is a 2D matrix of output activations, i.e., the OFM 160).

[0047] In the depthwise convolution 183, the input channels are not combined. Rather, MAC operations are performed on an individual input channel and an individual kernel and produce an OC. As shown in FIG. 1, the depthwise convolution 183 produces a depthwise output tensor 180. The depthwise output tensor 180 is represented by a 5x5x33D matrix. The depthwise output tensor 180 includes 3 OCs, each of which is represented by a 5x52D matrix. The 5x52D matrix includes 5 output elements in each row and 5 output elements in each column. Each OC is a result of MAC operations of an input channel of the IFM 140 and a kernel of the filter 150. For instance, the first OC (patterned with dots) is a result of MAC operations of the first input channel (patterned with dots) and the first kernel (patterned with dots), the second OC (patterned with horizontal strips) is a result of MAC operations of the second input channel (patterned with horizontal strips) and the second kernel (patterned with horizontal strips), and the third OC (patterned with diagonal stripes) is a result of MAC operations of the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes). In such a depthwise convolution, the number of input channels equals the number of OCs, and each OC corresponds to a different input channel. The input channels and output channels are referred to collectively as depthwise channels. After the depthwise convolution, a pointwise convolution 193 is then performed on the depthwise output tensor 180 and a 1x1x3 tensor 190 to produce the OFM 160.

[0048] The OFM 160 is then passed to the next layer in the sequence. In some embodiments, the OFM 160 is passed through an activation function. An example activation function is rectified linear unit (ReLU). ReLU is a calculation that returns the value provided as input directly, or the value zero if the input is zero or less. The convolutional layer 110 may receive several images as input and calculate the convolution of each of them with each of the kernels. This process can be repeated several times. For instance, the OFM 160 is passed to the subsequent convolutional layer 110 (i.e., the convolutional layer 110 following the convolutional layer 110 generating the OFM 160 in the sequence). The subsequent convolutional layer 110 performs a convolution on the OFM 160 with new kernels and generates a new feature map. The new feature map may also be normalized and resized. The new feature map can be kernelled again by a further subsequent convolutional layer 110, and so on.

[0049] In some embodiments, a convolutional layer 110 has four hyperparameters: the number of kernels, the size F kernels (e.g., a kernel is of dimensions FxFxD pixels), the S step with which the window corresponding to the kernel is dragged on the image (e.g., a step of one means moving the window one pixel at a time), and the zero-padding P (e.g., adding a black contour of P pixels thickness to the input image of the convolutional layer 110). The convolutional layers 110 may perform various types of convolutions, such as 2D convolution, dilated or atrous convolution, spatial separable convolution, depthwise separable convolution, transposed convolution, and so on. The DNN 100 includes 16 convolutional layers 110. In other embodiments, the DNN 100 may include a different number of convolutional layers.

[0050] The pooling layers 120 down-sample feature maps generated by the convolutional layers 110, e.g., by summarizing the presence of features in the patches of the feature maps. A pooling layer 120 may be placed between two convolution layers 110: a preceding convolutional layer 110 (the convolution layer 110 preceding the pooling layer 120 in the sequence of layers) and a subsequent convolutional layer 110 (the convolution layer 110 subsequent to the pooling layer 120 in the sequence of layers). In some embodiments, a pooling layer 120 is added after a convolutional layer 110, e.g., after an activation function (e.g., ReLU, etc.) has been applied to the OFM 160.

[0051] A pooling layer 120 receives feature maps generated by the preceding convolution layer 110 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while preserving their important characteristics.Accordingly, the pooling operation improves the efficiency of the DNN and avoids over-learning. The pooling layers 120 may perform the pooling operation through average pooling (calculating the average value for each patch on the feature map), max pooling (calculating the maximum value for each patch of the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature maps. In various embodiments, the pooling operation is 2x2 pixels applied with a stride of two pixels, so that the pooling operation reduces the size of a feature map by a factor of 2, e.g., the number of pixels or values in the feature map is reduced to one quarter the size. In an example, a pooling layer 120 applied to a feature map of 6x6 results in an output pooled feature map of 3x3. The output of the pooling layer 120 is input into the subsequent convolution layer 110 for further feature extraction. In some embodiments, the poolinglayer 120 operates upon each feature map separately to create a new set of the same number of pooled feature maps.

[0052] The fully-connected layers 130 are the last layers of the DNN. The fully-connected layers 130 may be convolutional or not. The fully-connected layers 130 receive an input operand. The input operand defines the output of the convolutional layers 110 and pooling layers 120 and includes the values of the last feature map generated by the last pooling layer 120 in the sequence. The fully-connected layers 130 apply a linear combination and an activation function to the input operand and generate a vector. The vector may contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Each element is therefore between 0 and 1, and the sum of all is worth one. These probabilities are calculated by the last fully-connected layer 130 by using a logistic function (binary classification) or a SoftMax function (multi-class classification) as an activation function.

[0053] In some embodiments, the fully-connected layers 130 classify the input image 105 and return an operand of size N, where N is the number of classes in the image classification problem. In the embodiments of FIG. 1, N equals 3, as there are 3 objects 115, 125, and 135 in the input image. Each element of the operand indicates the probability for the input image 105 to belong to a class. To calculate the probabilities, the fully-connected layers 130 multiply each input element by weight, make the sum, and then apply an activation function (e.g., logistic if N=2, softmax if N>2). This is equivalent to multiplying the input operand by the matrix containing the weights. In an example, the vector includes 3 probabilities: a first probability indicating the object 115 being a tree, a second probability indicating the object 125 being a car, and a third probability indicating the object 135 being a person. In other embodiments where the input image 105 includes different objects or a different number of objects, the individual values can be different.

[0054] FIG. 2 illustrates an example convolution, in accordance with various embodiments. The convolution may be a deep learning operation in a convolutional layer of a DNN, e.g., a convolutional layer 110 in FIG. 1. The convolution can be executed on an activation tensor 210 and filters 220 (individually referred to as "filter 220"). The filters may constitute a weight tensor of the convolution. The result of the convolution is an output tensor 230. In some embodiments, the convolution is performed by a DNN accelerator. An example of theDNN accelerator may be the NPU 302 in FIG. 3. For instance, the convolution may be performed by one or more compute blocks 330 in the NPU 302.

[0055] The activation tensor 210 may be computed in a previous layer of the DNN. In some embodiments (e.g., embodiments where the convolutional layer is the first layer of the DNN), the activation tensor 210 may be an image. In the embodiments of FIG. 2, the activation tensor 210 includes activations (also referred to as "input activations," "elements," or "input elements") arranged in a 3D matrix. The activation tensor 210 may also be referred to as an input tensor of the convolution. An input element is a data point in the activation tensor 210. The activation tensor 210 has a spatial size Hinx Winx Cin, where Hinis the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of activations in a column in the 3D matrix of each input channel), Winis the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of activations in a row in the 2D matrix of each input channel), and Cinis the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of input channels). For the purpose of simplicity and illustration, the activation tensor 210 has a spatial size of 7x7x3, i.e., the activation tensor 210 includes three input channels and each input channel has a 7x72D matrix. Each input element in the activation tensor 210 may be represented by a (X, Y, Z) coordinate. In other embodiments, the height, width, or depth of the activation tensor 210 may be different.

[0056] Each filter 220 includes weights arranged in a 3D matrix. The values of the weights may be determined through training the DNN. A filter 220 has a spatial size Hfx Wfx Cf, where Hfis the height of the filter (i.e., the length along the Y axis, which indicates the number of weights in a column in each kernel), Wfis the width of the filter (i.e., the length along the X axis, which indicates the number of weights in a row in each kernel), and Cfis the depth of the filter (i.e., the length along the Z axis, which indicates the number of channels). In some embodiments, Cfequals Cin. For purpose of simplicity and illustration, each filter 220 in FIG. 2 has a spatial size of 2x3x3, i.e., the filter 220 includes 2 convolutional kernels with a spatial size of 2x3. In other embodiments, the height, width, or depth of the filter 220 may be different. The spatial size of the convolutional kernels is smaller than the spatial size of the 2D matrix of each input channel in the activation tensor 210.

[0057] An activation or weight may take one or more bytes in a memory. The number of bytes for an activation or weight may depend on the data format. For example, when the activation or weight has an INT8 format, the activation takes one byte. When the activation or weight has a FP16 format, the activation or weight takes two bytes. Other data formats may be used for activations or weights.

[0058] In the convolution, each filter 220 slides across the activation tensor 210 and generates a 2D matrix for an output channel in the output tensor 230. In the embodiments of FIG. 2, the 2D matrix has a spatial size of 5x5. The output tensor 230 includes activations (also referred to as "output activations," "elements," or "output element") arranged in a 3D matrix. An output activation is a data point in the output tensor 230. The output tensor 230 has a spatial size Houtx Woutx Cout, where Houtis the height of the 3D matrix (i.e., the length along the Y axis, which indicates the number of output activations in a column in the 2D matrix of each output channel), Woutis the width of the 3D matrix (i.e., the length along the X axis, which indicates the number of output activations in a row in the 2D matrix of each output channel), and Coutis the depth of the 3D matrix (i.e., the length along the Z axis, which indicates the number of output channels). Coutmay equal the number of filters 220 in the convolution. Houtand Woutmay depend on the heights and weights of the activation tensor 210 and each filter 220. In an example where the kernel size is lxl, Houtand Woutmay equal to Hinand Win, respectively.

[0059] As a part of the convolution, MAC operations can be performed on a 2x3x3 subtensor 215 (which is highlighted with a dotted pattern in FIG. 2) in the activation tensor 210 and each filter 220. The result of the MAC operations on the subtensor 215 and one filter 220 is an output activation. In some embodiments (e.g., embodiments where the convolution is an integral convolution), an output activation may include 8 bits, e.g., one byte. In other embodiments (e.g., embodiments where the convolution is a floating-point convolution), an output activation may include more than one byte. For instance, an output element may include two bytes.

[0060] After the MAC operations on the subtensor 215 and all the filters 220 are finished, a vector 235 is produced. The vector 235 is highlighted with a dotted pattern in FIG. 2. The vector 235 includes a sequence of output activations, which are arranged along the Z axis. The output activations in the vector 235 have the same (x, y) coordinate, but the output activations correspond to different output channels and have different Z coordinates. Thedimension of the vector 235 along the Z axis may equal the total number of output channels in the output tensor 230. After the vector 235 is produced, further MAC operations are performed to produce additional vectors till the output tensor 230 is produced.

[0061] In some embodiments, the MAC operations on a 3x3x3 subtensor (e.g., the subtensor 215) and a filter 220 may be performed by a plurality of MAC units. One or more MAC units may receive an input operand (e.g., an activation operand 217 shown in FIG. 2) and a weight operand (e.g., the weight operand 227 shown in FIG. 2). The activation operand 217 includes a sequence of activations having the same (x, y) coordinate but different z coordinates. The activation operand 217 includes an activation from each of the input channels in the activation tensor 210. The weight operand 227 includes a sequence of weights having the same (x, y) coordinate but different z coordinates. The weight operand 227 includes a weight from each of the channels in the filter 220. Activations in the activation operand 217 and weights in the weight operand 227 may be sequentially fed into a MAC unit. The MAC unit may receive an activation and a weight ("an activation-weight pair") at a time and multiple the activation and the weight. The position of the activation in the activation operand 217 may match the position of the weight in the weight operand 227. The activation and weight may correspond to the same channel.

[0062] Activations or weights may be floating-point numbers. Floating-point numbers may have various data formats, such as FP32, FP16, BF16, and so on. A floating-point number may be a positive or negative number with a decimal point. A floating-point number may be represented by a sequence of bits that includes one or more bits representing the sign of the floating-point number (e.g., positive or negative), bits representing an exponent of the floating-point number, and bits representing a mantissa of the floating-point number. The mantissa is the part of a floating-point number that represents the significant digits of that number. The mantissa is multiplied by the base raised to the exponent to give the actual value of the floating-point number. In some embodiments, floating-point activations or floating-point weights may be quantized before the MAC operations are performed. The quantization may increase efficiency of the MAC operations.

[0063] In some embodiments, the output activations in the output tensor 230 may be further processed based on one or more activation functions before they are written into the memory or inputted into the next layer of the DNN. The processing based on the one or more activation functions may be at least part of the post processing of the convolution. Insome embodiments, the post processing may include one or more other computations, such as offset computation, bias computation, and so on. The results of the post processing may be stored in a local memory of the compute block and be used as input to the next DNN layer. In some embodiments, the input activations in the activation tensor 210 may be results of post processing of the previous DNN layer.

[0064] FIG. 3 is a block diagram of an Al system 300, in accordance with various embodiments. The whole Al system 300 or a part of the Al system 300 may be implemented in one or more computing devices, such as the computing device 1200 in FIG. 12. The Al system 300 can generate and execute DNNs. As shown in FIG. 3, the Al system 300 includes a DNN module 301 and an NPU 302. In other embodiments, alternative configurations, different or additional components may be included in the Al system 300. For instance, the Al system 300 may include multiple DNN modules or multiple NPUs. Further, functionality attributed to a component of the Al system 300 may be accomplished by a different component included in the Al system 300 or a different system. In some embodiments, the Al system 300 may include different types of processing units. In an example, the DNN module 301 may be implemented by one or more central processing units (CPUs). The NPU 302 may also be referred to as an Al processor or accelerator. The DNN module 301 and NPU 302 may be implemented in the same chip or separate chips. Even though not shown in FIG. 3, the Al system 300 may include one or more graphics processing units.

[0065] The DNN module 301 facilitates generation and deployment of DNNs. In some embodiments, the DNN module 301 may generate and train DNNs. For instance, the DNN module 301 can define the layered architecture of a DNN. The DNN module 301 can also determine the internal parameters of the DNN through a DNN training process. The DNN module 301 may also determine one or more hyperparameters that define how the DNN is trained. An example hyperparameter is a sparsity ratio that defines the sparsity level of one or more deep learning tensors for the DNN. The DNN module 301 may also compress DNNs, e.g., during or after training. In some embodiments, the DNN module 301 may prune weights in one or more layers of a DNN by changing nonzero weight to zeros. The DNN module 301 may prune weights based on a target weight sparsity ratio. A weight sparsity ratio may be the ratio of the number of zero weights to the total number of weights. In an example where the DNN module 301 prunes weight during DNN training, the DNN module 301 may prune weight of a layer to achieve a target sparsity ratio after one or more epochs.The DNN module 301 may prevent the pruned weights from changing values during the rest of the training process. Alternatively, the DNN module 301 may allow the pruned weights to change values so that a pruned, zero weight may have a nonzero value after further training. The DNN module 301 may prune weights of the layer again after one or more additional epochs.

[0066] The DNN module 301 may deploy trained, compressed, or validated DNNs for use in neural network applications. In some embodiments, the DNN module 301 may distribute trained, compressed, or validated DNNs to devices or systems which may use the DNNs to perform tasks (e.g., image classification, motion planning, etc.) for which the DNNs were trained. In other embodiments, the DNN module 301 may facilitate deployment of the DNNs using the NPU 302. For instance, the DNN module 301 may receive data from a device or system coupled with the Al system 300 and input the received data (or data generated by the DNN module 301, e.g., based on the received data) into a DNN. The DNN module 301 may generate instructions (e.g., computer program instructions) that can be executed by the NPU 302 for DNN execution. The DNN module 301 may receive an output of the DNN from the NPU 302. The DNN module 301 may transmit the output of the DNN (or a result of processing the output of the DNN by the DNN module 301) to the device or system. In some embodiments, the DNN module 301 may control execution processes of trained, compressed, or validated DNNs. The DNN module 301 may function as a compiler for DNNs executed by the NPU 302. The DNN module 301 may perform compilation of DNNs and generate compilation descriptors, based on which the DNNs may be executed.

[0067] The DNN module 301 may facilitate executions of floating-point models by using integer datapaths within the NPU 302. For instance, the DNN module 301 may facilitate executions of FP4 models by using INT5 or INT9 datapaths in the NPU 302. In some embodiments, the DNN module 301 may determine which integral data type(s) activations or weights are to be quantized to. The DNN module 301 may determine the same data type for activations and weights. Alternatively, the DNN module 301 may determine two different types for activations and weights, respectively. In some embodiments, the DNN module 301 may generate configuration parameters to be used by the NPU 302 to quantize weights or activations in floating-point models. The configuration parameters may include parameters for programming one or more look-up tables or configuration registers in the NPU 302. The DNN module 301 may provide the configuration parameters to the NPU 302for the NPU 302 to execute floating-point DNNs by performing computations in integer domains. Certain aspects of the DNN module 301 are provided below in conjunction with FIG. 4.

[0068] The NPU 302 executes DNNs, including DNNs provided by the DNN module 301. For instance, the NPU 302 can execute a DNN by carrying out neural network operations in the DNN. The process of carrying out a neural network operation is also referred to as a process of executing the neural network operation or performing the neural network operation. The execution of the DNN may be for training the DNN or for using the DNN to perform Al tasks. The NPU 302 may execute neural network operations in floating-point DNNs by performing computations in integer domains, such as INT5 or INT9 domain. The NPU 302 may quantize activations or weights of a neural network operation to integer values and perform the neural network operation on the integer values.

[0069] As shown in FIG. 3, the NPU 302 includes a memory 310, a direct memory access (DMA) engine 320, and compute blocks 330 (individually referred to as "compute block 330"). In other embodiments, alternative configurations, different or additional components may be included in the NPU 302. For example, the NPU 302 may include more than one memory 310 or DMA engine 320. As another example, the NPU 302 may include a single compute block 330. Further, functionality attributed to a component of the NPU 302 may be accomplished by a different component included in the NPU 302 or by a different system. A component of the NPU 302 may be implemented in hardware, software, firmware, or some combination thereof.

[0070] The memory 310 stores data associated with neural network operations performed by the NPU 302. In some embodiments, the memory 310 may store data to be used by the compute blocks 330 for executing neural network operations. The memory 310 may store inputs to DNNs and outputs of DNNs. The memory 310 may also store activations (such as input activations and output activations of neural network operations) and weights (such as weights determined by training DNNs) in DNNs. In some embodiments, the memory 310 may store activations and weights with floating-point precisions, such as FP4, SF4, NF4, and so on. The memory 310 may also quantized activations or weights. In some embodiments, the memory 310 may store configuration parameters generated by the DNN module 301. The memory 310 includes one or more dynamic random-access memories (DRAMs).

[0071] The DMA engine 320 facilitates data transfer between the memory 310 and the compute blocks 330. For example, the DMA engine 320 can read data from the memory 310 and write data into a local memory of a compute block 330. As another example, the DMA engine 320 can read data from a local memory of a compute block 330 and write data into the memory 310. For instance, the DMA engine 320 may read input activations and weights of convolution from the memory 310 and load the input activations and weights to one or more compute blocks 330. The DMA engine 320 may also write output activations of convolutions computed by one or more compute blocks 330 to the memory 310. The DMA engine 320 provides a DMA feature that allows the compute block 330 to initiate data transfer between the memory 310 and the local memories of the compute blocks 330 and to perform other operations while the data transfer is being conducted. In some embodiments, the DMA engine 320 may read tensors from the memory 310, modify the tensors in a way that is optimized for the compute block 330 before it writes the tensors into the local memories of the compute blocks 330.

[0072] The compute blocks 330 perform neural network operations in DNNs. For instance, a compute block 330 may execute a DNN layer by running one or more deep learning operations in the DNN layer. A compute block 330 may execute a layer, or a portion of a layer, at a time. In some embodiments, the operations of the DNN layers may be run by multiple compute blocks 330 in parallel. For instance, multiple compute blocks 330 may each perform a portion of a workload for a neural network operation. Data may be shared between the compute blocks 330. A compute block 330 may also be referred to as a compute tile, compute core, or data processing unit. The compute blocks 330 may be capable of running various types of neural network operations, such as convolution, matrix multiplication, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on. Neural network operations performed by the compute blocks 330 include tensor operations, i.e., operations whose inputs are tensors or operations whose outputs are tensors. In an example, the compute block 330 receives an input tensor and one or more convolutional kernels and performs a convolution with the input tensor and convolutional kernels. The result of the convolution may be an output tensor, which can be further computed, e.g., by the compute block 330 or another compute block 330.

[0073] In the embodiments of FIG. 3, each compute block 330 includes a local memory 340, a IDU 360, a processing engine 370, a post-processing engine 380, and an ODU 390. The arrows between the components in the compute block 330 indicate data flow between these components. For instance, the IDU 360 may read data from the memory 340 and send the data to the processing engine 370. The DIU 360 may also send data to the postprocessing engine 380. The processing engine 370 may sent data that it has computed to the post-processing engine 380. The ODU 390 may drain data from the post-processing engine 380 into the local memory 340. In other embodiments, data flow within the compute block 330 may have different paths. Some or all the components of the compute block 330 can be implemented on the same chip. In other embodiments, alternative configurations, different or additional components may be included in the compute block 330. Further, functionality attributed to a component of the compute block 330 may be accomplished by a different component included in the compute block 330, a different compute block 330, another component of the NPU 302, or a different system. A component of the compute block 330 may be implemented in hardware, software, firmware, or some combination thereof.

[0074] The local memory 340 is local to the corresponding compute block 330. In the embodiments of FIG. 3, the local memory 340 is inside the compute block 330. In other embodiments, the local memory 340 may be outside the compute block 330. Data in the local memory 340 may be transferred to or from the memory 310, e.g., through the DMA engine 320. In some embodiments, data in the local memory 340 may be transferred to or from the memory 310 or from the local memory of another compute block 330. The local memory 340 may store data received, used, or generated by the IDU 360, the processing engine 370, the post-processing engine 380, or the ODU 390. Examples of the data may include input activations, weights, output activations, configuration parameters, and so on.

[0075] In some embodiments, the local memory 340 includes one or more static randomaccess memories (SRAMs). The local memory 340 may be byte-addressable, and each memory address identifies a single byte (eight bits) of storage. In some embodiments, the local memory 340 may include memory banks. The number of data banks in the local memory 340 may be 16, 64, 128, 356, 512, 1024, 2048, or other numbers. A memory bank may include a plurality of storage units. In an example, a data bank may include 8, 16, 64, or a different number of storage units. A memory bank or a storage unit in a memory bankmay have a memory address. In an example, a storage unit may store a single byte, and data larger than a single byte may be stored in storage units with consecutive memory addresses, i.e., adjacent storage units. For instance, a storage unit can store an integer number in the INT8 format, versus two storage units may be needed to store a number in the FP16 or BF16 format, which has 16 bits. In some embodiments, 16 bits can be transferred from the local memory 340 in a single read cycle. In other embodiments, 16 bits can be transferred from the local memory 340 in multiple read cycles, such as two cycles.

[0076] The IDU 360 loads data from the local memory 340 to the processing engine 370 or to the post-processing engine 380. The IDU 360 may read tensors from the local memory 340. The tensors may include activation tensors, weights tensor, and so on. In some embodiments, the IDU 360 may read data from the local memory 340 and write the data into storage units in the processing engine 370. For instance, the IDU 360 may load activations into activation register files in the processing engine 370 and load weights into weight register files in the processing engine 370. In some embodiments, the IDU 360 may read configuration parameters from the local memory 340 and load the configuration parameters into configuration registers in the post-processing engine 380.

[0077] In some embodiments, the IDU 360 converts values of a floating-point data type to values of an integral data type. For instance, the IDU 360 converts values in a weight tensor of a floating-point data type (e.g., FP4, SF4, NF4, etc.) to values of an integral data type (e.g., INT5, INT9, etc.). The IDU 360 may include a palletization unit to facilitate the quantization. The palletization unit may include a look-up table that maps floating-point values to integer values. The mapping may be normalized mapping. For instance, the look-up table may store normalized floating-point values and normalized integer values. The palletization unit may compute the normalized floating-point values from weights read from the local memory 340. After the quantization, the IDU 360 may load the integer values to the processing engine 370 for performing computations in an integer domain. A datapath between the local memory 340 and the IDU 360 may be a floating-point datapath that can support transmission of floating-point values. A datapath between the IDU 360 and the processing engine 370 may be an integer datapath that can support transmission of integer values. Certain aspects of IDU are described below in conjunction with FIG. 7.

[0078] The processing engine 370 performs operations in DNNs. The processing engine 370 may include one or more processing cells. In some embodiments, the processing cells maybe arranged in one or more rows and one or more columns in the processing engine 370. Each processing cell may include PEs that may be arranged in an array that includes rows and columns. All the PEs in the processing engine 370 may constitute a bigger array that includes more rows and columns. An example PE may be or may include one or more MAC units that can perform MAC operations. In some embodiments (e.g., embodiments where the compute block 330 executes a convolutional layer), a computation in an MAC unit may be an MAC operation on an activation operand and a weight operand. The activation operand may be an activation tensor that may include one or more activations in the input tensor of the convolution. Different activations may be in different input channels. The weight operand may be a weight tensor that may include one or more weights in the filter of the convolution. The values of the weights are determined through training the DNN or compressing the neural network operation after training. The weights in the weight operand may be in different input channels. In some embodiments, the activation operand or weight operand is a vector along the IC dimension.

[0079] In some embodiments, an MAC unit includes one or more multipliers for performing multiplications. An MAC unit may also include one or more accumulators ("adders") for performing accumulations. An MAC unit may also include one or more shifters to facilitate mixed-precision computations. A column of MAC units is referred to as an MAC column. An MAC column may be associated with one or more MAC lanes. A MAC lane is a path for loading data e.g., by the IDU 360, into an MAC column. A MAC lane may be also referred to as a data transmission lane or data loading lane. An MAC column may have multiple MAC lanes. The loading bandwidth of the MAC column is an aggregation of the loading bandwidths of all the MAC lanes associated with the MAC column. With a certain number of MAC lanes, data can be fed into the same number of independent MAC units simultaneously. In some embodiments where an MAC column has four MAC lanes for feeding activations or weights into the MAC column and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes can have a total loading bandwidth of 64 bytes. Certain aspects of processing engine and processing element are described below in conjunction with FIGS. 8-10.

[0080] The post-processing engine 380 processes outputs of the processing engine 370. The post-processing engine 380 may include one or more PPEs. In some embodiments, the PPEs in the post-processing engine 380 may be arranged in an array that has rows and columns.The post-processing engine 380 may quantize activations computed by the processing engine 370. The post-processing engine 380 may receive an output tensor of a neural network operation from the processing engine 370. Values in the output tensor may be of a floating-point data point. The post-processing engine 380 may quantize the floating-point values using quantization parameters. A quantization parameter may include a scale. In some embodiments, a quantization parameter may further include a zero point. The postprocessing engine 380 may include configuration registers programmed with quantization parameters. In some embodiments, different configuration registers may store quantization parameters for different channels of the output tensor so that PPEs in the post-processing engine 380 can perform per-channel quantization. In an example, an activation operand having activations in the same channel of the output tensor may be provided to a PPE. The PPE may receive a quantization parameter from the channel from a configuration register and quantize the activations in the activation operand using the quantization parameter.

[0081] In some embodiments, the post-processing engine 380 computes activation functions. The post-processing engine 380 may receive outputs of the processing engine 370 as inputs to the activation functions. In addition or alternative to activation functions, the post-processing engine 380 may perform other types of post processing on outputs of the processing engine 370. For instance, the post-processing engine 380 may apply a bias on an output of the processing engine 370. In some embodiments, the post-processing engine 380 may be bypassed for certain neural network operations. Certain aspects of post-processing engine are described below in conjunction with FIG. 11.

[0082] The ODU 390 drains data from the processing engine 370 or from the postprocessing engine 380, e.g., from register files in the processing engine 370 or from the post-processing engine 380. The drain module may write the data to the local memory 340. The drained data may be tensors, such as output tensors of neural network operations. In some embodiments, the ODU 390 may drain data on a cell level. For each processing cell, the ODU 390 may drain outputs of PEs in the processing cell based on a row index or column index of each PE. For instance, the ODU 390 may use a sequence of cycles to drain data from a processing cell. The ODU 390 may drain the output of some of the PEs in each cycle. The sequence of the cycles may be configured based on a configuration parameter indicating the operation mode of the IDU 360.

[0083] In some embodiments, the ODU 390 includes sparsity encoding logic that can convert outputs of the processing engine 370 from a dense format to a sparse format. For instance, the ODU 390 may be implemented with one or more sparsity encoders. A sparsity encoder converts dense data to compressed data based on sparsity in the dense data. For instance, the sparsity encoder may remove zeros from data computed by the processing engine 370. The sparsity encoder may also generate sparsity maps that represents sparsity in the dense data.

[0084] In some embodiments, the data drained from the processing engine 370 may be at least part of an output tensor (e.g., the output tensor 230 in FIG. 2) of a neural network operation. The sparsity encoder may generate a compressed version of the output tensor. The sparsity encoder may identify every zero activation in the output tensor and remove these activations from the output tensor to generate a compressed activation tensor (aka "sparse activation tensor"). The sparsity encoder may also generate one or more sparsity maps for the output tensor. A sparsity map may indicate sparsity in at least part of the output tensor. The sparsity map may include sparsity elements (e.g., bits), each of which corresponds to a different activation in the vector and indicates whether the corresponding activation is zeroed or not.

[0085] The ODU 390 may write the compressed activation tensor and the one or more sparsity maps into the local memory 340. The sparse activation tensor and the one or more sparsity maps may be further loaded to the memory 310, e.g., through the DMA engine 320. Additionally or alternatively, the sparse activation tensor and the one or more sparsity maps may be loaded by the IDU 360 to the processing engine 370 for further computation, e.g., for performing a deep learning operation in the next layer.

[0086] FIG. 4 is a block diagram of a DNN module 400, in accordance with various embodiments. The DNN module 400 may be an example of the DNN module 301 in FIG. 3. As shown in FIG. 4, the DNN module 400 includes an interface module 410, a training module 420, a compiler 430, and a datastore 440. In other embodiments, alternative configurations, different or additional components may be included in the DNN module 400. Further, functionality attributed to a component of the DNN module 400 may be accomplished by a different component included in the DNN module 400 or a different module or system.

[0087] The interface module 410 facilitates communications of the DNN module 400 with other modules or systems. For example, the interface module 410 establishes communications between the DNN module 400 with an external database to receive data that can be used to train DNNs or input into DNNs to perform tasks. As another example, the interface module 410 may distribute trained DNNs to other systems, e.g., computing devices configured to apply DNNs to perform tasks.

[0088] The training module 420 trains DNNs by using a training dataset. The training module 420 forms the training dataset. In an example where the training module 420 trains an DNN to recognize objects in images, the training dataset includes training images and training labels. The training labels describe ground-truth classifications of objects in the training images. In some embodiments, each label in the training dataset corresponds to an object in a training image. In some embodiments, a part of the training dataset may be used to initially train the DNN, and the rest of the training dataset may be held back as a validation subset used by the training module 420 to validate performance of a trained DNN. The data portion of the training dataset not including the tuning subset and the validation subset may be used to train the DNN.

[0089] The training module 420 also determines hyperparameters for training the DNN. Hyperparameters are variables specifying the DNN training process. Hyperparameters are different from parameters inside the DNN (e.g., weights of filters). In some embodiments, hyperparameters include variables determining the architecture of the DNN, such as number of hidden layers, etc. Hyperparameters also include variables which determine how the DNN is trained, such as batch size, number of epochs, etc. A batch size defines the number of training samples to work through before updating the parameters of the DNN. The batch size is the same as or smaller than the number of samples in the training dataset. The training dataset can be divided into one or more batches. The number of epochs defines how many times the entire training dataset is passed forward and backwards through the entire network. The number of epochs defines the number of times that the deep learning algorithm works through the entire training dataset. One epoch means that each training sample in the training dataset has had an opportunity to update the parameters inside the DNN. An epoch may include one or more batches. The number of epochs may be 1, 5, 10, 50, 100, 500, 1000, or even larger.

[0090] The training module 420 defines the architecture of the DNN, e.g., based on some of the hyperparameters. The architecture of the DNN includes an input layer, an output layer, and a plurality of hidden layers. The input layer of an DNN may include tensors (e.g., a multidimensional array) specifying attributes of the input image, such as the height of the input image, the width of the input image, and the depth of the input image (e.g., the number of bits specifying the color of a pixel in the input image). The output layer includes labels of objects in the input layer. The hidden layers are layers between the input layer and output layer. The hidden layers include one or more convolutional layers and one or more other types of layers, such as pooling layers, fully-connected layers, normalization layers, SoftMax or logistic layers, and so on. The convolutional layers of the DNN abstract the input image to a feature map that is represented by a tensor specifying the feature map height, the feature map width, and the feature map channels (e.g., red, green, blue images include 3 channels). A pooling layer is used to reduce the spatial volume of input image after convolution. It is used between two convolution layers. A fully-connected layer involves weights, biases, and neurons. It connects neurons in one layer to neurons in another layer. It is used to classify images between different categories by training.

[0091] In the process of defining the architecture of the DNN, the training module 420 also adds an activation function to a hidden layer or the output layer. An activation function of a layer transforms the weighted sum of the input of the layer to an output of the layer. The activation function may be, for example, a ReLU activation function, a tangent activation function, or other types of activation functions.

[0092] After the training module 420 defines the architecture of the DNN, the training module 420 inputs a training dataset into the DNN. The training dataset includes a plurality of training samples. An example of a training sample includes an object in an image and a ground-truth label of the object. The training module 420 modifies the parameters inside the DNN ("internal parameters of the DNN") to minimize the error between labels of the training objects that are generated by the DNN and the ground-truth labels of the objects. The internal parameters include weights of filters in the convolutional layers of the DNN. In some embodiments, the training module 420 uses a cost function to minimize the error.

[0093] The training module 420 may train the DNN for a predetermined number of epochs. The number of epochs is a hyperparameter that defines the number of times that the deep learning algorithm works through the entire training dataset. One epoch means that eachsample in the training dataset has had an opportunity to update internal parameters of the DNN. After the training module 420 finishes the predetermined number of epochs, the training module 420 may stop updating the parameters in the DNN. The DNN having the updated parameters is referred to as a trained DNN.

[0094] The training module 420 may also verify accuracy of DNNs after training. In some embodiments, the training module 420 inputs samples in a validation dataset into a trained DNN and uses the outputs of the DNN to determine the model accuracy. In some embodiments, a validation dataset may be formed of some or all the samples in the training dataset. Additionally or alternatively, the validation dataset includes additional samples, other than those in the training sets. In some embodiments, the training module 420 may determine an accuracy score measuring the precision, recall, or a combination of precision and recall of the DNN. The training module 420 may use the following metrics to determine the accuracy score: Precision = TP / (TP + FP) and Recall = TP / (TP + FN), where precision may be how many the DNN correctly predicted (TP or true positives) out of the total it predicted (TP + FP or false positives), and recall may be how many the DNN correctly predicted (TP) out of the total number of objects that did have the property in question (TP + FN or false negatives). The F-score (F-score = 2 * PR / (P + R)) unifies precision and recall into a single measure.

[0095] The training module 420 may compare the accuracy score with a threshold score. In an example where the training module 420 determines that the accuracy score of the DNN is less than the threshold score, the training module 420 instructs the training module 420 to re-train the DNN. In one embodiment, the training module 420 may iteratively re-train the DNN until the occurrence of a stopping condition, such as the accuracy measurement indication that the DNN may be sufficiently accurate, or a number of training rounds having taken place.

[0096] The compiler 430 compiles information of DNNs to executable instructions that can be executed, e.g., by the NPU 302, to carry out neural network operations in DNNs. In some embodiments, the compiler 405 may generate a graph representing a DNN. The graph may include nodes and edges. A node may represent a specific neural network operation in the DNN. An edge may connect two nodes and represent a connection between the two corresponding neural network operations. In an example, an edge may encode a tensor that flows from one of the neural network operations to the other neural network operation.The tensor may be an output tensor of the first neural network operation and an input tensor of the second neural network operation. The edge may encode one or more attributes of the tensor, such as size, shape, storage format, and so on. The compiler 430 may use the graph to generate executable DNNs. For instance, the compiler may generate computer program instructions that are executable by the NPU 302. The computer program instructions may cause the NPU 302 to perform neural network operations in the DNNs.

[0097] The compiler 430 may generate configuration parameters that configure components of NPUs (e.g., the NPU 302) for DNN executions. The configuration parameters for a DNN may be stored in one or more configuration registers in the NPU executing the NPU. In some embodiments, the compiler 430 generates configuration parameters for floating-point DNNs so that the NPU can execute the DNNs by performing computations in integer domains. The compiler 430 may generate configuration parameters that facilitate quantizing values of a floating-point data type to values of an integral data type through direct mapping. The configuration parameters may configure a palletization unit in the NPU. The palletization unit may reside in an IDU of the NPU (e.g., the IDU 360). For instance, the compiler 430 may obtain (e.g., retrieve or determine) a normalization factor for a tensor of the floating-point data type. The tensor may be an input tensor of a neural network operation, such as a weight tensor or activation tensor. The compiler 430 may determine the normalization factor by normalizing the maximum absolute value of the tensor. For instance, the compiler 430 may use the maximum absolute value of the tensor as the value of the normalization factor. The compiler 430 may provide the normalization factor to the palletization unit, and the palletization unit may normalize floating-point values using the normalization factor. For instance, the palletization unit may normalize the floating-point values by dividing the floating-point values by the normalization factor so that the normalized floating-point values fall within the [-1.0, 1.0] range.

[0098] The compiler 430 may also generate one or more configuration parameters that indicate mapping of a normalized range of the floating-point data type (e.g., a normalized FP4 range) to a normalized range of the integral data type (e.g., a normalized INT9 range). The configuration parameter(s) may configure a look-up table in the palletization unit, which can be used to map the normalized floating-point values to normalized integer values. In some embodiments, the compiler 430 ensures that the two ranges are the same. For instance, both are [-1.0, 1.0], The compiler 430 may map values in the normalized floating-point range to values in the normalized integer range. The compiler 430 may use midpoints between consecutive values in the normalized floating-point range as boundaries to determine where each normalized floating-point value should land within the normalized integer range. For each value in the normalized floating-point range, the compiler 430 may compare it against these midpoints to decide its nearest corresponding value in the normalized integer range. The normalized floating-point values, which lie between the predefined floating-point set points, may be mapped proportionally into the normalized integer range. After a value in the normalized floating-point range is mapped to a value in the normalized integer range, the compiler 430 may pair the two values and program an entry in the look-up table with the two values. The compiler 430 may program a plurality of entries in the look-up table in this way. When the programmed look-up table receives a normalized floating-point value, it can output the corresponding normalized integer value.

[0099] The palletization unit may further use the normalization factor to convert the normalized integer values output from the look-up table to integer values of the target integral data type. The compiler 430 may further generate configuration parameters that cause the IDU to load the integer values into a processing engine (e.g., the processing engine 370) or post-processing engine (e.g., the post-processing engine 380). The compiler 430 may further generate configuration parameters for configuring the processing engine or post-processing engine to perform the neural network operation in the integer domain using the integer values. The processing engine may produce an output tensor of the neural network operation.

[0100] The compiler 430 may also determine one or more quantization parameters for quantizing the output tensor of a neural network operation. A quantization parameter may be a scale, zeropoint, or combination of both. In some cases, different channels may contain activations that vary significantly in magnitude, which can present a challenge for uniform quantization. For instance, one channel might have activations between [-0.5, 0.5], while another might span a much larger range, such as [-2.5, 2.5], Using a global scale would lead to underutilization of the available INT9 precision for channels with smaller dynamic ranges or clipping for channels with large activations. In some embodiments, the compiler 430 facilitates per-channel quantization and determines channel-specific quantization parameters for the output tensor.

[0101] The compiler 430 may perform the following steps for each channel. First, for the channel Ctin the output tensor, the compiler 430 may determine the maximum activation value ximnrand the minimum activation value ximiri, where i denotes the index of the channel in the output tensor. The range of the activations in the channel falls in the range [xi,min>Xi,max]- This range may be dynamic as different neural network operations may produce different output tensors. The compiler 430 may compute a scale Stfor the channel. In an example for quantizing FP4 values to INT9 values, the compiler 430 may determine that St= 127 / xi max. This scale can facilitate quantization with minimal information loss and ensure that the maximum FP4 value within the channel, ximnr, is mapped to 127, which is the maximum representable value in INT9. The per-channel scale Sthelps in mapping the FP4 values optimally, preventing clipping of large values or a loss of resolution for smaller ones. The compiler 430 may configure a post-processing engine in the NPU (e.g., the post-processing engine 380) with the channel-specific scales. For instance, the scales may be stored in configuration registers in the post-processing engine. The post-processing engine may scale and quantize each FP4 value x£j- within channel C£to its corresponding INT9 representation using the channel-specific scale:represents the quantized INT9 value for activation x£j-. This ensures that the FP4 values are accurately mapped into the INT9 range of [-128, 127] while maintaining relative magnitudes across the data.

[0102] The datastore 440 stores data received, generated, used, or otherwise associated with the DNN module 400. For example, the datastore 440 stores the datasets used by the training module 420. The datastore 440 may also store data generated by the training module 420, such as the hyperparameters for training DNNs, internal parameters of trained DNNs (e.g., weights, etc.), and so on. The datastore 440 may store configuration parameters, computer program instructions, or other data generated by the compiler 430. The datastore 440 may include one or more memories. In the embodiment of FIG. 4, the datastore 440 is a component of the DNN module 400. In other embodiments, the datastore 440 may be external to the DNN module 400 and communicate with the DNN module 400 through a network.

[0103] FIG. 5 illustrates mapping floating-point data types to integral data types, in accordance with various embodiments. The floating-point data types may be data types ofactivations or weights in DNNs, which can be quantized to values of the integral data types. For the purpose of illustration, FIG. 5 shows two floating-point formats, SF4 and NF4, and two integral formats, INT5 and INT9. In other embodiments, other floating-point formats or integral formats may be used.

[0104] FP4 models may be DNNs having weights or activations of the FP4 datatype, which includes variants of FP4. SF4 and NF4 are two variants of FP4. A FP4 value may be represented by 4 bits, including one bit for sign, two bits for exponent, and one bit for mantissa. The 4-bit precision can significantly scale down computational requirements while maintaining robust accuracy and operational efficiency, making it a key enabler in modern deep learning systems. Among FP4 variants, SF4 may be a tailored solution specifically developed to align with the unique distribution patterns of data in neural networks. The strategic design of SF4 can incorporate advanced quantization techniques that cater specifically to the non-normal distribution of weights and activations within DNNs. By employing a distribution-aware quantization strategy, SF4 can closely align with the statistical properties of the data, reducing quantization errors and enhancing computational efficiency. This quantization approach can minimize data loss and preserves model fidelity more effectively than currently available quantization methods, which often degrade information significantly when compressing data to lower bitwidths. NF4, another effective variant of FP4, can be compatible with diverse data distributions. Both SF4 and NF4 variants can match or even surpass higher-precision formats for certain Al tasks, such as Al tasks using vision models. SF4 and NF4 can support optimization of computational resources while ensuring accuracy for many Al applications. Even though not shown in FIG. 5, datatypes of weights can be other variants of FP4 or other floating-point data types.

[0105] FP4 values in DNNs may be quantized to INT5 or INT9 values. Executing DNNs in the INT9 domain can remove the need for specialized FP4 hardware, allowing seamless compatibility across FP4 variants without added area or power overhead. Moreover, incorporating INT5 as another precision option further extends flexibility, enabling INT5 execution for both weights and activations or in a mixed-precision configuration with INT9 for even greater efficiency. For instance, the weights may be INT5 values while the activations may be INT values, or vice versa. This versatility can allow the system (e.g., the Al system 300) to maintain iso-power performance relative to FP4, ensuring optimized accuracy and efficiency in low-precision scenarios regardless of the FP4 format chosen bythe Al model. FP4 values can be quantized to INT5 or INT9 values through a direct mapping strategy that maintains normalized value ranges between -1.0 and 1.0, as shown in FIG. 5. The direct mapping approach can be used for multiple formats and precision levels. For the purpose of illustration, FIG. 5 shows SF4 mapped to INT5, SF4 mapped to INT9, and NF4 mapped to INT9. There can be direct mapping from other floating-point formats to other integral formats.

[0106] In some embodiments, FP values in a tensor may be first normalized. Then the normalized FP4 values are mapped directly to corresponding INT5 or INT9 values, e.g., by using a look-up table. Compared with INT9, INT5 can provide an additional lower precision option for even greater efficiency. In some embodiment, the normalized FP4 values may be mapped into the INT9 or INT5 domain using a normalization factor. The normalization factor may be determined by normalizing the maximum absolute value of the tensor, allowing the values to effectively fit within the INT9 or INT5 range without substantial information loss. The normalized FP4 values may be mapped proportionally to either INT9 or INT5, maintaining relative distances between data points for each format.

[0107] In an example of mapping FP4 values into the INT9 domain, a mapping function may effectively map the FP4 values to an INT9 range by using the normalization factor. This may be done by mapping the normalized FP4 values, such as values that are already confined within a small range, to the normalized INT9 range. The small range may be [-1.0, 1.0], The normalized INT9 range may be the same as the normalized FP4 range. The mapping function may use midpoints between consecutive FP4 values to further guide the conversion. These midpoints may act as boundaries, helping determine where each normalized FP4 value should land within the normalized INT9 range. For each FP4 value, the mapping function may compare it against these midpoints to decide its nearest corresponding INT9-mapped value. In other words, the normalized FP4 values, which lie between the predefined FP4 set points, are mapped proportionally into the normalized INT9 range, as shown in FIG. 5. After the mapping, the values may be scaled up based on the calculated normalization factor to occupy the appropriate positions within the INT9 range, maintaining the relative distances between the original FP4 values. The scaled values may then be cast into INT9 format, resulting in a compact and computationally efficient INT9 representation of the original FP4 data.

[0108] This mapping approach may be applied to INT5, although with a reduced range, making INT5 suitable for situations where additional memory and power savings are required without a significant impact on accuracy. The normalization factor may stretch the discrete FP4 values into the INT9 or INT5 domains, ensuring full utilization of each range while preserving the structure and fidelity of the original data. This direct mapping, combined with per-format midpoints, can minimize quantization errors and allows efficient computations by eliminating complex floating-point arithmetic typical of higher-precision formats, such as FP16 and FP32. The alignment of INT9 and INT5 mappings with SF4 and NF4, which is shown in FIG. 5, can ensure compatibility across various FP4 variants.

[0109] The computational efficiency gained from this quantization can be profound. By eliminating unnecessary steps in the execution process, the system can reduce both latency and power consumption, a critical advantage in real-time Al systems, edge devices, and mobile platforms where computational resources are limited. This direct mapping mechanism can not only optimize the computational pipeline but also ensure that the computation doesn't introduce inaccuracies or inefficiencies. The consistent, reliable transformation of FP4 data into INT can allow NPUs to execute a broader range of Al tasks with higher throughput, while also consuming less energy. This characteristic can be particularly beneficial in environments where maintaining both high performance and energy efficiency is paramount, such as autonomous systems, large-scale Al infrastructures, power-sensitive devices, and so on. Moreover, by embedding this quantization process seamlessly within the NPU's operational flow, the Al system can adapt to various workloads without the need for extensive reconfiguration or specialized hardware. This flexibility can facilitate a more general application of the technique across different Al models and architectures, making it a highly versatile solution in various Al accelerator designs.

[0110] FIG. 6 illustrates per-channel quantization, in accordance with various embodiments. For the purpose of simplicity and illustration, FIG. 6 shows an image 600 that has three channels 610, 620, and 630. The image 600 may be an activation tensor with a spatial size H x W x C, where H denotes the height of the tensor, W denotes the width of the tensor, and C denotes the depth of the tensor. A pixel in the image 600 may be an activation. In this example, C = 3, as there are three channels in the image 600. The channels 610, 620, and 630 may correspond to red, green, and blue, respectively. In embodiments of per-channel quantization, the channels 610, 620, and 630 are quantized separately using differentquantization parameters. A quantization parameter includes a scale S and a zero point Z. The quantization parameter for the channel 610 is shown as SR, ZR, the quantization parameter for the channel 620 is shown as SG, ZG, and the quantization parameter for the channel 630 is shown as SB,ZB. The quantization parameter for a channel may be used to quantize all the activations in the channel.

[0111] The per-channel quantization strategy can optimize accuracy in low-bit computations by adjusting quantization parameters uniquely for each channel in the incoming activations. In some embodiments, per-channel quantization is used for the conversion from FP4 to INT9. Unlike traditional uniform quantization, which applies a single quantization scale globally across all channels, the per-channel quantization method can dynamically compute individual scales or zero points for each channel based on the statistical distribution of the activations within that channel. This strategy may be applied during calibration using a representative dataset and can ensure that each channel is quantized optimally, thereby reducing quantization error and preserving accuracy, particularly in scenarios with varying data ranges or outliers. For example, a feature map may have several channels, where each channel might represent different features of the input data. In a conventional setup, applying a single quantization scale could result in significant information loss, particularly in cases where one channel contains high-magnitude values while another contains low-magnitude values. Per-channel quantization addresses this problem by analyzing the data distribution within each channel independently, allowing each to be quantized more optimally. This can ensure that the range of values in each channel is preserved, minimizing quantization errors.

[0112] In some embodiments, the process of determining these channel-specific quantization parameters involves calculating the maximum absolute value within each channel and deriving an optimal scale. The scale may adjust based on the data's dynamic range, ensuring that values are spread evenly across the available precision levels within the INT9 format. By normalizing each channel individually, the approach can prevent clipping of high-magnitude values or underutilization of the available range for lower-magnitude values.

[0113] One of the major advantages of per-channel quantization is its ability to handle outliers— values that deviate significantly from the typical range of activations in a given channel. Outliers can cause clipping if a single global scale is used, as they would skew thescale towards high-magnitude values, leading to loss of precision for the rest of the channel. By calculating the scale Si individually for each channel, the method avoids this issue, ensuring that channels with outliers are scaled appropriately without impacting the quantization precision of other channels. For example, consider a channel where most activations are between -0.5 and 0.5, but there are occasional spikes up to 1.5. In traditional quantization, this would result in either clipping of the high values or reduced precision for the lower values. With per-channel quantization, the scale is adjusted for each channel's data distribution, allowing high-magnitude outliers to be quantized without distorting the majority of the activations. This technique is invaluable in ensuring that the computational efficiency gained by using low-precision formats such as FP4 does not come at the cost of significant accuracy degradation, particularly in large and complex Al models.

[0114] The per-channel quantization strategy can mitigate the accuracy degradation typically associated with FP4 quantization. By finely adjusting quantization parameters for each channel according to its unique data distribution, this approach can effectively preserve data integrity and model performance. This strategy can not only maximize the computational benefits of 4-bit precision but also ensure robustness against the precision losses common in low-bit computing.

[0115] FIG. 7 illustrates an IDU 700 with a palletization unit 730, in accordance with various embodiments. The IDU 700 may be an example of the IDU 360 in FIG. 3. As shown in FIG. 7, the IDU 700 also includes a datapath 710 and a datapath 720. The datapath 710 may be a floating-point datapath that facilitates transmission of floating-point values, such as FP4 values. The datapath 710 may be used to transmit data from a memory (e.g., the local memory 340) to the palletization unit 730. The datapath 720 may be an integer datapath that facilitates transmission of integer values, such as INT9 values or INT5 values. The datapath 710 may be used to transmit data from the palletization unit 730 to a processing engine (e.g., the processing engine 370).

[0116] The palletization unit 730 has a normalizing module 740, a look-up table 750, and a scaling module 760. The normalizing module 740 may receive a normalization factor, e.g., from the compiler 430. The normalizing module 740 may use the normalization factor to normalize floating-point values received through the datapath 710. For instance, the normalizing module 740 may divide each floating-point value by the normalization factor to produce the normalized floating-point value. The normalized floating-point values may fallwithin a predetermined range. An example of the range is [-1.0, 1.0], The normalizing module 740 may provide the normalized floating-point values to the look-up table 750.

[0117] The look-up table 750 may have been programmed with values within the normalized floating-point range and values within the normalized integer range. For instance, the look-up table 750 may include a plurality of entries, each entry may be a row in the look-up table 750. The entry may include a value in the normalized floating-point range and a corresponding value in the normalized integer range. When the look-up table 750 receives a normalized floating-point value from the normalizing module 740, the lookup table 750 may locate the entry having the normalized floating-point value and output the normalized integer value in the entry.

[0118] The scaling module 760 may receive normalized integer values output by the look-up table 750. The scaling module 760 may scale up the normalized integer values and produce integer values of the target integral data type. For instance, the scaling module 760 may multiple the normalized integer value with the normalization factor to produce the corresponding integer value. These integer values may maintain the relative distances between the original floating-point values received through the datapath 710. The IDU 700 loads the integer values to the processing engine through the datapath 720 for the processing engine to perform integral computations.

[0119] In an example, the data type of the original floating-point values is FP4, and the target integral data type is INT 9. By using the palletization unit 730, the process of computing FP4 datatype through INT9 datapath in the NPU is seamlessly integrated into the hardware. This process can allow for the efficient conversion and handling of low-precision FP4 values by mapping them directly to INT9 values, minimizing the computational overhead associated with higher bit-width formats, while enabling compute support and associated memory savings for low-precision formats. This can be crucial for efficiently managing memory, especially in large-scale Al models. The process begins by mapping the FP4 values to INT9, where they undergo palletization via a look-up table. The quantization involves mapping the FP4 values onto a higher-precision INT9 space using normalized mapping. The quantized INT9 parameters may then be prepared for further processing within the neural network pipeline.

[0120] The palletization unit 730 can play a key role by converting the incoming FP4 values into corresponding INT9 values, maintaining their normalized range and ensuring minimalloss of precision. In the embodiments of FIG. 7, as the conversion from FP4 to INT9 is handled in the IDU 700, the FP4 weights are mapped to INT9 before entering the processing engine. This allows real-time data mapping from FP4 to INT9 with minimal latency, which is vital for maintaining performance in high-throughput applications like convolutional layers or matrix multiplications. Moreover, the process can ensure that the INT9 values are passed seamlessly into the processing engine, where they are utilized in the NPU's computational workflows, such as matrix multiplications or convolutions, without the need for any intermediate conversions back to FP16. This can preserve computational efficiency while minimizing memory bandwidth requirements, a critical factor in Al accelerators, where managing data efficiently can be paramount.

[0121] A key innovation in this approach is the ability to perform these conversions directly within the IDU. By relocating the palletization and mapping process to the IDU, as opposed to managing in the processing engine, this architecture can reduce processing overhead and improve computational accuracy. This method can not only ensure the execution FP4 via INT9 is efficient but also guarantee that the data integrity and precision are retained through hardware optimization, making it suitable for handling complex Al models with reduced power and memory demands.

[0122] FIG. 8 illustrates an example PE 800, in accordance with various embodiments. The PE may be an example of PEs in the processing engine 370. In FIG. 8, the PE 800 includes a MAC unit 810, an input storage unit 820, a weight storage unit 830, an accumulator 880, and an output storage unit 890. The MAC unit 810 includes multipliers 850A-D (collectively referred to as "multipliers 850" or "multiplier 850") and an adder tree 840. The adder tree 840 includes adders 860A and 860B and an adder 865. In other embodiments, alternative configurations, different or additional components may be included in the PE 800. For example, the PE 800 may include more than one MAC unit. The MAC unit 810 may include a different number of multipliers. The adder tree 840 may include a different number of adders. Further, functionality attributed to a component of the PE 800 may be accomplished by a different component included in the PE 800, a different component included in a PE array where the PE 800 is placed, or by a different system. The positions of the components of the PE 800 in FIG. 8 are for the purpose of illustration only. Even though the positions of the components may reflect the direction of data flow in the PE 800, the positions of thecomponents in FIG. 8 do not necessarily represent physical positions of the components in the PE 800.

[0123] The PE 800 may perform sequential cycles of MAC operations. In a cycle of MAC operations, the PE 800 may process multiple input operands and multiple weight operands, e.g., given the presence of multiple multipliers 850 in the MAC unit 810. Activations may be provided to the input storage unit 820 and stored in the input storage unit 820. In some embodiments, the input storage unit 820 may store activations of up to four input operands in the cycle of MAC operations. Weights may be provided to the weight storage unit 830 and stored in the weight storage unit 830. The weight storage unit 830 may store weights of up to four weight operands in the cycle of MAC operations. The multipliers 850 may fetch activations and weights from the input storage unit 820 and weight storage unit 830 and compute products. In an example round, each multiplier 850 receives an activation and a corresponding weight and outputs the product of the activation and the weight. In other cycles, the activations and weights may be reused by different multipliers 850. The activations and weights from the input storage unit 820 and weight storage unit 830 may be reused more than once. In some embodiments, the activations or weights may be integer values. The multipliers 850 may perform integer multiplications.

[0124] The adder tree 840 receives dot products computed by the multiplier 850 and accumulates the dot products. The adder 860A receives products computed by the multipliers 850A and 850B and computes a first sum. The adder 860B receives products computed by the multiplier 850C and 850D and computes a second sum. The adder 865 receives the first sum and the second sum from the pipeline registers 870A and 870B and accumulates the sums to generate an output of the MAC unit 810.

[0125] The output of the MAC unit 810 is further provided to the accumulator 880. The accumulator 880 may accumulate the output of the MAC unit 810 with a value stored in the output storage unit 890. The value may be an output of another PE 800, which has been sent to the PE 800 and stored in the output storage unit 890. The output of the accumulator 880 can be stored in the output storage unit 890. In some embodiments, the output of the accumulator 880 may be a floating-point value.

[0126] FIG. 9A illustrates an example sparse cell 900, in accordance with various embodiments. The sparse cell 900 may be a processing cell in a processing engine, e.g., the processing engine 370 in FIG. 3. The sparse cell 900 includes 16 MAC units 910 (individuallyreferred to as "MAC unit 910"), which constitutes a MAC array having four rows and four columns. The MAC array has a spatial shape of 4x4, meaning the height of the MAC array is four and the width of the MAC array is also 9. The sparse cell 900 also includes 16 weight register files 920 (individually referred to as "weight register file 920"), 16 activation register files 930 (individually referred to as "activation register file 930"), four row buffers 940 (individually referred to as "row buffer 940"), and acceleration modules 960 (individually referred to as "acceleration module 960"). In other embodiments, the sparse cell 900 may include fewer, more, or different components. For example, the sparse cell 900 may include a different number of MAC units 910, weight register files 920, activation register files 930, row buffers 940, or acceleration modules 960. As another example, the sparse cell 900 may include column buffers in lieu of or in addition to the row buffers 940. Also, the shape (e.g., the height or width) of the MAC array may be different.

[0127] The MAC units 910 are configured to perform MAC operations. Each MAC unit 910 may include one or more multipliers and one or more adders. A multiplier may multiply an activation with a weight at a time to compute a product. In some embodiments (e.g., embodiments where the MAC unit 910 includes multiple multipliers), the multipliers may operate simultaneously to process multiple activation-weight pairs and compute multiple products in one cycle. An adder may accumulate products computed by the multipliers. Even though not shown in FIG. 9A, the sparse cell may include an adder tree including a plurality of adder tiers. The first tier may receive outputs of a plurality of MAC units 910. The number of adders in the first tier may be half of the number of the MAC units 910, and each adder may accumulate the outputs of two MAC units 910. The second tier may receive outputs of adders in the first tier. The number of adders in the second tier may be half of the number of adders in the first tier, and each adder in the second tier may accumulate the outputs of two adders in the first tier. The adder tree may include one or more other tiers. The last tier may include a single adder that accumulates outputs of adders in the second last tier to compute a partial sum of the sparse cell 900.

[0128] The weight register files 920 store weights to be processed in MAC operations. In the embodiments of FIG. 9A, four weight register files 920 are grouped into a storage set that stores data to be used by a column of MAC units 910. There are four storage sets corresponding to the four columns of MAC units 910. In some embodiments, a weight register file 920 may correspond to a MAC unit 910 and store data to be processed by theMAC unit. In some embodiments, all the 16 weight register files 920 constitute a weight storage unit.

[0129] The activation register files 930 stores activations to be processed in MAC operations. In the embodiments of FIG. 9A, four activation register files 930 are grouped into a storage set that stores data to be used by a row of MAC units 910. There are four storage sets corresponding to the four rows of MAC units 910. In some embodiments, an activation register file 930 may correspond to a MAC unit 910 and store data to be processed by the MAC unit. In some embodiments, all the 16 activation register files 930 constitute an activation storage unit. The row buffers 940 store outputs of the MAC units 910. Each row buffer 940 may drain outputs of a single row of MAC units 910.

[0130] The acceleration module 960 facilitates acceleration of computations in the sparse cell 900 based on mixed formats of weights. In the embodiments of FIG. 9A, each acceleration module 960 may control acceleration of computations in a different MAC unit 910. The number of acceleration modules 960 in the sparse cell 900 is the same as the number of MAC units 910 in the sparse cell 900. In other embodiments, an acceleration module 960 may control acceleration in multiple MAC units 910. As shown FIG. 9A, each acceleration module 960 includes a storage unit 965 and a control logic 967. The storage unit 965 stores sparsity maps. The control logic 967 may control distributions of activations and weights stored from the weight register files 920 and the activation register files 930 to the MAC units 910 based on sparsity maps. In some embodiments, the control logic 967 may distribute a weight operand and a corresponding activation operation to a MAC unit 910 for an MAC operation. In some embodiments, a weight operand may include one or more nonzero weight and one or more zero weights. In some embodiments, the control logic 967 may select nonzero weights stored in the weight register files 920 based on the sparsity map and distribute these nonzero weights to the MAC unit 910 for computation. The control logic 967 may also distribute activations, which correspond to the nonzero weights, to the MAC unit 910 from in the activation register files 930. The control logic 967 may ignore zero weights and activations corresponding the zero weights so that these weights and activations can be skipped from computation.

[0131] As shown in FIG. 9A, the sparse cell 900 is associated with multiplexers (MUXs) 903, 904, 905, and 906. In other embodiments, the sparse cell 900 may be associated with a different number of MUXs or other devices. The MUX 903 facilitates loading weights, e.g.,from the local memory 340, into the weight register files 920. The MUX 904 facilitates loading activations, e.g., from the local memory 340, into the activation register files 930. The MUX 905 facilitates loading mixed-format maps into the storage unit 965. The MUX 906 may be a drain MUX that can facilitate draining outputs of the MAC units 910, e.g., to the local memory 340.

[0132] FIG. 9B illustrates a sparse cell array 970, in accordance with various embodiments. The sparse cell array 970 may be an example of the processing engine 370 in FIG. 3. In FIG.9B, the sparse cell array 970 includes sparse cells 980 (individually referred to as "sparse cell 980") arranged in four columns and four rows, an activation memory 990, and a weight memory 995. In other embodiments, the sparse cell array 970 may include fewer, more, or different components. For instance, the sparse cell array 970 may include a different number of columns, rows, or sparse cells 980.

[0133] Each sparse cell 980 may perform accelerated MAC operations. MAC operations in the sparse cells 980 may be accelerated based on mixed formats of weights. An embodiment of a sparse cell 980 may be the sparse cell 900 in FIG. 9A. The activation memory 990 stores activations, such as activations in input tensors of neural network operations. Activations may be loaded from the activation memory 990 to sparse cells 980, e.g., to activation register files. The weight memory 995 stores weights, such as weights in filters of neural network operations. Weights may be loaded from the weight memory 995 to sparse cells 980, e.g., to weight register files. The activation memory 990 or weight memory 995 may be a buffer.

[0134] FIG. 10 illustrates an example post-processing engine 1000, in accordance with various embodiments. The post-processing engine 1000 may be an example of the postprocessing engine 380 in FIG. 3. In the embodiments of FIG. 10, the post-processing engine 1000 includes PPEs 1010 (individually referred to as "PPE 1010") and configuration registers 1020 (individually referred to as "configuration register 1020"). Each corresponding PPE 1010 is paired with a corresponding configuration register 1020. For the purpose of simplicity and illustration, the post-processing engine 1000 includes 64 PPEs and 64 configuration registers 1020 arranged in an array. In other embodiments, the postprocessing engine 1000 may include fewer or more PPEs or configuration registers. Further, a PPE 1010 may be associated with multiple configuration registers 1020, or multiple PPEs 1010 may share a single configuration register 1020.

[0135] The post-processing engine 1000 may perform quantization of data (e.g., an output tensor) computed by a processing engine, such as the sparse cell array 970 in FIG. 9B. The processing engine may send the data to the post-processing engine 1000. For instance, sparse cells in the processing engine (e.g., sparse cells 980 in FIG. 9B) may correspond to different PPEs 1010. Each sparse cell may drain its output to the corresponding PPE 1010. The PPE 1010 may use a quantization parameter stored in the corresponding configuration register 1020 to quantize the output of the sparse cell. The quantization parameter may be a scale, zeropoint, or combination of both.

[0136] In some embodiments, the quantization may be per-channel quantization. Each PPE may apply a unique scale and zero point for each output channel. For instance, each PPE 1010 may process its own unique quantization parameter at every cycle, thus enabling per-channel quantization. A critical part of this process is the ability of the hardware to handle different quantization parameters (e.g., different scales or different zero points) across multiple cycles (e.g., t=0, t=T, t=2T, or t=3T, etc.), as the output from each sparse cell is drained into the corresponding PPE 1010. In an example, a PPE 1010 may retrieve a scale Stfor a channel C£from the corresponding configuration register 1020. The PPE 1010 may then quantize activations in the channel C£by computing qtj = round(Xij X St), where x£- denotes activation andrepresents the quantized value for activation x£j-, which is an integer value.

[0137] In some embodiments, the configuration registers 1020 are configured or programmed in advance, ensuring each PPE 1010 operates with the predefined scale and zero point, thus maximizing quantization precision at the channel level. The configuration registers 1020 may be a notable aspect of the PPE design. The configuration registers 1020 may remain static throughout the processing workload. For instance, the configuration registers 1020 may be programmed before the sparse cells perform computations to produce the output tensor. The configuration registers 1020, once configured, can ensure that the appropriate scales and zero points are applied to the corresponding channels as the data flows through the system. The configuration registers 1020 may be updated when PPE 1010, or even including the IDU and ODU, is in an idle state, preventing disruptions in ongoing computations. This setup can guarantee that the PPEs 1010 can perform precise per-channel quantization throughout the workload, leading to significant improvements in both accuracy and computational efficiency.

[0138] The per-channel quantization feature may be an inherent part of the system's design, utilizing existing scale tables and configuration registers, thus adding minimal or even zero additional hardware overhead. By leveraging the pre-existing data structures and operational flow of the NPU, this functionality integrates seamlessly without requiring any extra hardware resources or increasing complexity.

[0139] FIG. 11 is a flowchart of a method 1100 of executing a DNN, in accordance with various embodiments. The method 1100 may be performed by the NPU 302 in FIG. 3.Although the method 1100 is described with reference to the flowchart illustrated in FIG. 11, many other methods for executing DNNs may alternatively be used. For example, the order of execution of the steps in FIG. 11 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

[0140] The NPU 302 computes 1110 normalized floating-point values from floating-point values in an input tensor of a neural network operation. In some embodiments, a data type of the floating-point values is a 4-bit floating-point data type. In some embodiments, the input tensor is a weight tensor, and the floating-point values are determined by training a neural network comprising the neural network operation.

[0141] The NPU 302 maps 1120 the normalized floating-point values to normalized integer values by using a look-up table. The look-up table has a range of normalized integer values and a range of normalized floating-point values. In some embodiments, the range of normalized integer values and the range of normalized floating-point values are the same. In some embodiments, the range of normalized integer values or the range of normalized floating-point values is from -1 to +1. In some embodiments, a data type of the normalized integer values is a 5-bit integral data type or a 9-bit integral data type.

[0142] The NPU 302 converts 1130 the normalized integer values to integer values. In some embodiments, the NPU 302 converts the normalized integer values to the integer values based on a normalization factor that the NPU 302 uses to compute the normalized floatingpoint values from the floating-point values. In some embodiments, the normalization factor is determined based on a maximum absolute value of the input tensor.

[0143] The NPU 302 executes 1140 the neural network operation using the integer values. In some embodiments, the neural network operation includes one or more MAC operations. In some embodiments, the NPU 302 generates an output tensor of the neural network operation. In some embodiments, the NPU 302 quantizes, using a plurality of quantizationparameters stored in a plurality of configuration registers, an output tensor computed from executing the neural network operation. In some embodiments, different ones of the plurality of configuration registers are configured with different ones of the plurality of quantization parameters before the neural network operation is executed. In some embodiments, different ones of the plurality of quantization parameters are used for quantizing different channels of the output tensor. In some embodiments, a configuration parameter includes a scale that is determined based on a maximum value and a minimum value of a channel of the output tensor.

[0144] FIG. 12 is a block diagram of an example computing device 1200, in accordance with various embodiments. In some embodiments, the computing device 1200 can be used as at least part of the Al system 300. A number of components are illustrated in FIG. 12 as included in the computing device 1200, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 1200 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 1200 may not include one or more of the components illustrated in FIG. 12, but the computing device 1200 may include interface circuitry for coupling to the one or more components. For example, the computing device 1200 may not include a display device 1206, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 1206 may be coupled. In another set of examples, the computing device 1200 may not include an audio input device 1218 or an audio output device 1208 but may include audio input or output device interface circuitry to which an audio input device 1218 or audio output device 1208 may be coupled.

[0145] The computing device 1200 may include a processing device 1202 (e.g., one or more processing devices). The processing device 1202 processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. The computing device 1200 may include a memory 1204, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid state memory, and / or a hard drive. In some embodiments, the memory 1204 may include memory that shares a die with the processing device 1202. Insome embodiments, the memory 1204 includes one or more non-transitory computer-readable media storing instructions executable to perform operations for executing DNNs (e.g., the method 1100 described in conjunction with FIG. 11) or some operations performed by one or more components of the Al system 300. The instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 1202.

[0146] In some embodiments, the computing device 1200 may include a communication chip 1212 (e.g., one or more communication chips). For example, the communication chip 1212 may be configured for managing wireless communications for the transfer of data to and from the computing device 1200. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

[0147] The communication chip 1212 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 1212 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. The communication chip 1212 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 1212 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, aswell as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 1212 may operate in accordance with other wireless protocols in other embodiments. The computing device 1200 may include an antenna 1222 to facilitate wireless communications and / or to receive other wireless communications (such as AM or FM radio transmissions).

[0148] In some embodiments, the communication chip 1212 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, the communication chip 1212 may include multiple communication chips. For instance, a first communication chip 1212 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 1212 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication chip 1212 may be dedicated to wireless communications, and a second communication chip 1212 may be dedicated to wired communications.

[0149] The computing device 1200 may include battery / power circuitry 1214. The battery / power circuitry 1214 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1200 to an energy source separate from the computing device 1200 (e.g., AC line power).

[0150] The computing device 1200 may include a display device 1206 (or corresponding interface circuitry, as discussed above). The display device 1206 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.

[0151] The computing device 1200 may include an audio output device 1208 (or corresponding interface circuitry, as discussed above). The audio output device 1208 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0152] The computing device 1200 may include an audio input device 1218 (or corresponding interface circuitry, as discussed above). The audio input device 1218 may include any device that generates a signal representative of a sound, such as microphones,microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0153] The computing device 1200 may include a GPS device 1216 (or corresponding interface circuitry, as discussed above). The GPS device 1216 may be in communication with a satellite-based system and may receive a location of the computing device 1200, as known in the art.

[0154] The computing device 1200 may include another output device 1210 (or corresponding interface circuitry, as discussed above). Examples of the other output device 1210 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0155] The computing device 1200 may include another input device 1220 (or corresponding interface circuitry, as discussed above). Examples of the other input device 1220 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0156] The computing device 1200 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultramobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 1200 may be any other electronic device that processes data.

[0157] The following paragraphs provide various examples of the embodiments disclosed herein.

[0158] Example 1 provides an apparatus, including a processing engine including one or more processing elements for performing one or more multiply-accumulate operations for an execution of a neural network operation; a post-processing engine including a plurality of configuration registers, different ones of the plurality of configuration registers to be configured with different quantization parameters before the execution of the neuralnetwork operation; and an input delivery unit to: convert floating-point values in an input tensor of the neural network operation to integer values based on the different quantization parameters, and deliver the integer values to the processing engine, in which the processing element is to perform the one or more multiply-accumulate operations using the integer values.

[0159] Example 2 provides the apparatus of example 1, in which the post-processing engine further includes a plurality of post-processing elements, a configuration register storing a quantization parameter, a post-processing element to quantize a data element using the quantization parameter, in which the data element is computed by the processing engine by performing the one or more multiply-accumulate operations.

[0160] Example 3 provides the apparatus of example 2, in which the configuration parameter includes a scale that is determined based on a maximum value and a minimum value of a channel of the output tensor, in which the data element is in the channel.

[0161] Example 4 provides the apparatus of any one of examples 1-3, in which the plurality of post-processing elements are to quantize different channels of an output tensor of the neural network operation using the different quantization parameters.

[0162] Example 5 provides the apparatus of any one of examples 1-4, in which a data type of the integer values is different from a data type of the quantized data element.

[0163] Example 6 provides the apparatus of any one of examples 1-5, in which the input delivery unit includes a look-up table, the look-up table having normalized integer values, in which the input delivery unit is further to convert the floating-point values to the integer values by: computing normalized floating-point values from the floating-point values; and mapping the normalized floating-point values to a plurality of normalized integer values in the look-up table; and converting the plurality of normalized integer values to the integer values.

[0164] Example 7 provides the apparatus of example 6, in which the normalized floatingpoint values have a same range as the normalized integer values.

[0165] Example 8 provides the apparatus of any one of examples 1-7, in which a data type of the floating-point values is a 4-bit floating-point data type.

[0166] Example 9 provides the apparatus of any one of examples 1-8, in which a data type of the integer values is a 5-bit integral data type or a 9-bit integral data type.

[0167] Example 10 provides the apparatus of any one of examples 1-9, in which the input tensor includes at least one of a weight tensor and an activation tensor, in which the input delivery unit is to: convert floating-point values in the weight tensor to 5-bit integer values; and convert floating-point values in the activation tensor to 9-bit integer values.

[0168] Example 11 provides a method, including computing normalized floating-point values from floating-point values in an input tensor of a neural network operation; mapping the normalized floating-point values to normalized integer values by using a look-up table, the look-up table having a range of normalized integer values and a range of normalized floating-point values; converting the normalized integer values to integer values; and executing the neural network operation using the integer values.

[0169] Example 12 provides the method of example 11, further including quantizing, using a plurality of quantization parameters stored in a plurality of configuration registers, an output tensor computed from executing the neural network operation.

[0170] Example 13 provides the method of example 12, in which different ones of the plurality of configuration registers are configured with different ones of the plurality of quantization parameters before the neural network operation is executed, in which different ones of the plurality of quantization parameters are used for quantizing different channels of the output tensor.

[0171] Example 14 provides the method of example 12 or 13, in which a configuration parameter includes a scale that is determined based on a maximum value and a minimum value of a channel of the output tensor.

[0172] Example 15 provides the method of any one of examples 12-14, in which a data type of the integer values is different from a data type of the quantized output tensor.

[0173] Example 16 provides the method of any one of examples 11-15, in which the range of normalized integer values and the range of normalized floating-point values are the same.

[0174] Example 17 provides the method of any one of examples 11-16, in which the range of normalized integer values or the range of normalized floating-point values is from -1 to +1.

[0175] Example 18 provides the method of any one of examples 11-17, in which a data type of the floating-point values is a 4-bit floating-point data type.

[0176] Example 19 provides the method of any one of examples 11-18, in which a data type of the normalized integer values is a 5-bit integral data type or a 9-bit integral data type.

[0177] Example 20 provides the method of any one of examples 11-19, in which the input tensor is a weight tensor, and the floating-point values are determined by training a neural network including the neural network operation.

[0178] Example 21 provides an apparatus, including a memory to store an input tensor of a neural network operation, the input tensor including floating-point values; a processing engine including one or more processing elements for performing one or more multiply-accumulate operations for an execution of the neural network operation; a post-processing engine including a plurality of configuration registers, different ones of the plurality of configuration registers to be configured with different quantization parameters before the execution of the neural network operation; and an input delivery unit to: convert the floating-point values in the input tensor of the neural network operation to integer values based on the different quantization parameters, and deliver the integer values to the processing engine, in which the processing element is to perform the one or more multiply-accumulate operations using the integer values.

[0179] Example 22 provides the apparatus of example 21, in which the post-processing engine further includes a plurality of post-processing elements, a configuration register storing a quantization parameter, a post-processing element to quantize a data element using the quantization parameter, in which the data element is computed by the processing engine by performing the one or more multiply-accumulate operations.

[0180] Example 23 provides the apparatus of example 22, in which the configuration parameter includes a scale that is determined based on a maximum value and a minimum value of a channel of an output tensor computed by the processing engine in the execution of the neural network operation, in which the data element is in the channel of the output tensor.

[0181] Example 24 provides the apparatus of example 22 or 23, further including an output delivery unit to write the quantized data element into the memory.

[0182] Example 25 provides the apparatus of any one of examples 21-24, in which a data type of the floating-point values is a 4-bit floating-point data type, in which a data type of the integer values is a 5-bit integral data type or a 9-bit integral data type.

[0183] The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art can recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

Claims1. An apparatus, comprising:a processing engine comprising one or more processing elements for performing one or more multiply-accumulate operations for an execution of a neural network operation; a post-processing engine comprising a plurality of configuration registers, different ones of the plurality of configuration registers to be configured with different quantization parameters before the execution of the neural network operation; andan input delivery unit to:convert floating-point values in an input tensor of the neural network operation to integer values based on the different quantization parameters, and deliver the integer values to the processing engine,wherein the processing element is to perform the one or more multiply-accumulate operations using the integer values.

2. The apparatus of claim 1, wherein the post-processing engine further comprises a plurality of post-processing elements, a configuration register storing a quantization parameter, a post-processing element to quantize a data element using the quantization parameter, wherein the data element is computed by the processing engine by performing the one or more multiply-accumulate operations.

3. The apparatus of claim 2, wherein the configuration parameter includes a scale that is determined based on a maximum value and a minimum value of a channel of the output tensor, wherein the data element is in the channel.

4. The apparatus of any one of claims 1-3, wherein the plurality of post-processing elements are to quantize different channels of an output tensor of the neural network operation using the different quantization parameters.

5. The apparatus of any one of claims 1-4, wherein a data type of the integer values is different from a data type of the quantized data element.

6. The apparatus of any one of claims 1-5, wherein the input delivery unit comprises a lookup table, the look-up table having normalized integer values, wherein the input delivery unit is further to convert the floating-point values to the integer values by:computing normalized floating-point values from the floating-point values; and mapping the normalized floating-point values to a plurality of normalized integer values in the look-up table; andconverting the plurality of normalized integer values to the integer values.

7. The apparatus of claim 6, wherein the normalized floating-point values have a same range as the normalized integer values.

8. The apparatus of any one of claims 1-7, wherein a data type of the floating-point values is a 4-bit floating-point data type.

9. The apparatus of any one of claims 1-8, wherein a data type of the integer values is a 5-bit integral data type or a 9-bit integral data type.

10. The apparatus of any one of claims 1-9, wherein the input tensor comprises at least one of a weight tensor and an activation tensor, wherein the input delivery unit is to:convert floating-point values in the weight tensor to 5-bit integer values; and convert floating-point values in the activation tensor to 9-bit integer values.

11. A method, comprising:computing normalized floating-point values from floating-point values in an input tensor of a neural network operation;mapping the normalized floating-point values to normalized integer values by using a look-up table, the look-up table having a range of normalized integer values and a range of normalized floating-point values;converting the normalized integer values to integer values; andexecuting the neural network operation using the integer values.

12. The method of claim 11, further comprising:quantizing, using a plurality of quantization parameters stored in a plurality of configuration registers, an output tensor computed from executing the neural network operation.

13. The method of claim 12, wherein different ones of the plurality of configuration registers are configured with different ones of the plurality of quantization parameters before the neural network operation is executed, wherein different ones of the plurality of quantization parameters are used for quantizing different channels of the output tensor.

14. The method of claim 12 or 13, wherein a configuration parameter includes a scale that is determined based on a maximum value and a minimum value of a channel of the output tensor.

15. The method of any one of claims 12-14, wherein a data type of the integer values is different from a data type of the quantized output tensor.

16. The method of any one of claims 11-15, wherein the range of normalized integer values and the range of normalized floating-point values are the same.

17. The method of any one of claims 11-16, wherein the range of normalized integer values or the range of normalized floating-point values is from -1 to +1.

18. The method of any one of claims 11-17, wherein a data type of the floating-point values is a 4-bit floating-point data type.

19. The method of any one of claims 11-18, wherein a data type of the normalized integer values is a 5-bit integral data type or a 9-bit integral data type.

20. The method of any one of claims 11-19, wherein the input tensor is a weight tensor, and the floating-point values are determined by training a neural network comprising the neural network operation.

21. An apparatus, comprising:a memory to store an input tensor of a neural network operation, the input tensor comprising floating-point values;a processing engine comprising one or more processing elements for performing one or more multiply-accumulate operations for an execution of the neural network operation;a post-processing engine comprising a plurality of configuration registers, different ones of the plurality of configuration registers to be configured with different quantization parameters before the execution of the neural network operation; andan input delivery unit to:convert the floating-point values in the input tensor of the neural network operation to integer values based on the different quantization parameters, anddeliver the integer values to the processing engine,wherein the processing element is to perform the one or more multiply-accumulate operations using the integer values.

22. The apparatus of claim 21, wherein the post-processing engine further comprises a plurality of post-processing elements, a configuration register storing a quantization parameter, a post-processing element to quantize a data element using the quantization parameter, wherein the data element is computed by the processing engine by performing the one or more multiply-accumulate operations.

23. The apparatus of claim 22, wherein the configuration parameter includes a scale that is determined based on a maximum value and a minimum value of a channel of an output tensor computed by the processing engine in the execution of the neural network operation, wherein the data element is in the channel of the output tensor.

24. The apparatus of claim 22 or 23, further comprising:an output delivery unit to write the quantized data element into the memory.

25. The apparatus of any one of claims 21-24, wherein a data type of the floating-point values is a 4-bit floating-point data type, wherein a data type of the integer values is a 5-bit integral data type or a 9-bit integral data type.

Images