Integrated circuit package with composable network architecture

Abstract

An integrated circuit package may include chiplets. Each chiplet may include at least one compute core, memory, microcontroller, and accelerator. Each chiplet may also include a network providing inter-chiplet and intra-chiplet connectivity. The network may have switches arranged in a hierarchical tree structure. The switches in an edge level may each be coupled to at least one address module. An address module may determine a physical address referring to a memory region of a shared memory for a job to be performed by the chiplet, generate a packet for the job, and determine a path through which the packet may be transferred. The address module may handle memory failures by remapping predetermined paths to alternative paths. The networks may facilitate connectivity between the package and one or more other packages. These packages may be stacked in a rack. The switch networks may facilitate connectivity between the rack with another rack.

Description

INTEGRATED CIRCUIT PACKAGE WITH COMPOSABLE NETWORK ARCHITECTURECross-Reference to Related Application

[0001] This application claims the benefit of U. S. Provisional Patent Application No.63 / 713,960, filed October 30, 2024, and titled "COMPOSABLE NETWORK ARCHITECTURE WITH INBUILT RESILIENCE FOR SHARED MEMORY," which is incorporated herein by reference in its entirety for all purposes.Technical Field

[0002] This disclosure relates generally to artificial intelligence (Al), and more specifically, integrated circuit (IC) packages with composable network architectures.Background

[0003] Neural networks (also referred to as "deep neural networks" or "DNNs") are used extensively for a variety of Al applications ranging from natural language processing to computer vision, speech recognition, and image processing due to their ability to achieve high accuracy. However, the high accuracy comes at the expense of significant computation cost. DNNs have extremely high computing demands as there can be a large number of operations as well as a large amount of data to read and write. Therefore, techniques to improve efficiency of DNN workloads are needed.Brief Description of the Drawings

[0004] Embodiments can be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0005] FIG. 1A illustrates a package with a composable network architecture, in accordance with various embodiments.

[0006] FIG. 1B illustrates a package, in accordance with various embodiments.

[0007] FIG. 2 illustrates an architecture of a chiplet, in accordance with variousembodiments.

[0008] FIG. 3 illustrates a perspective view of a chiplet in a package, in accordance with various embodiments.

[0009] FIG. 4 illustrates a switch network having a three-level FatTree topology, in accordance with various embodiments

[0010] FIG. 5 illustrates a switch network having a two-level FatTree topology, in accordance with various embodiments.

[0011] FIG. 6 illustrates address translation in a package, in accordance with various embodiments.

[0012] FIG. 7 illustrates an example address module, in accordance with various embodiments.

[0013] FIG. 8 illustrates fault tolerance for memory failure, in accordance with various embodiments.

[0014] FIG. 9 illustrates a multi-chiplet system, in accordance with various embodiments.

[0015] FIG. 10 illustrates a multi-chiplet system, in accordance with various embodiments

[0016] FIG. 11 illustrates an example switch, in accordance with various embodiments.

[0017] FIG. 12 illustrates system connectivity in a currently available multi-hierarchy rack system, in accordance with various embodiments.

[0018] FIG. 13 illustrates system connectivity in a multi-hierarchy rack system 1300 with BiFrost packages, in accordance with various embodiments.

[0019] FIG. 14 is a block diagram of a neural processing unit (NPU), in accordance with various embodiments.

[0020] FIG. 15 illustrates an example sparse cell, in accordance with various embodiments.

[0021] FIG. 16 illustrates an example sparse cell array, in accordance with various embodiments.

[0022] FIG. 17 illustrates an example processing element (PE), in accordance with various embodiments.

[0023] FIG. 18 illustrates an example transformer model, in accordance with various embodiments.

[0024] FIG. 19 illustrates an example CNN, in accordance with various embodiments.

[0025] FIG. 20 is a block diagram of an example computing device, in accordance with various embodiments.Detailed Description

[0026] The last decade has witnessed a rapid rise in Al based data processing, particularly based on DNNs. DNNs are widely used in the domains of computer vision, speech recognition, image, and video processing mainly due to their ability to achieve beyond human-level accuracy. A DNN typically includes a sequence of layers. A DNN layer may include one or more operations, such as matrix multiplication, convolution, interpolation, layer normalization, batch normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on. These operations are referred to as deep learning operations or neural network operations.

[0027] With the advent of advanced packaging and fabrication technologies, modern high-performance Al accelerators running the prevailing machine learning workloads (such as transformer models of various modalities) usually exhibit extremely high compute density with tremendous demand for high memory bandwidth, such as memory bandwidth greater than 4TB / s. A conventional interconnect design with typical mesh topology or similar connectivity typically cannot such high data bandwidth within a smaller silicon geometry to meet the demands of modern Al accelerators.

[0028] In particular, per so-called scaling law of large language model (LLM), for data center Al training and inference usages, as the LLM model parameter sizes increase, the amount of compute required scale proportionally, leading to need for a scale-up and scale-out system level solution built out of the accelerator silicon as chiplets. It is highly desirable to ensure the interconnect supporting high bandwidth in the chiplet silicon also supports inter-chiplet interconnect for scale-out and scale-up requirements at the package level.

[0029] To harness the raw compute power endowed by the scale-out and scale-up multi-chiplet designs, it is needed to reuse the existing programming system and operation system (OS) environment so that various legacy software runtime and workloads can work out of the box on such systems. It would be highly desirable to have a unified shared address space for all the chiplets as it can deliver the legacy Portable Operating System Interface (POSIX) programming legacy compatibility to the programmers in addition to the scaled performance benefits. Currently available solutions to build scale-out system, such as solutions using mesh topology, fail to provide shared address space amongst chiplets, thus imposing tremendous challenge to balance performance scalability and workload enabling.

[0030] Advanced packaging and silicon fabrication can provide significant benefits, but at the same time multi-chiplet packages face reduced yield due to complex processes. A highly resilient system can be required to recover from the failures or should offer a mechanism to work through the faulty parts or chiplets. It can be highly desirable to handle resilience features including graceful degradation in ways resembling machine check architectural mechanism that are part of foundational RAS features of modern system architecture for which modern OSes support.

[0031] For the System on a Chip (SoC) connectivity a mesh, point-to-point or similar network is used which provides a fixed configuration and requires custom software techniques to manage disjoint address space. For the static and dynamic failures, software techniques are applied to remap the failing memory / logic. Such "custom" solutions cannot be supported by the mainstream OSes or runtime systems (e.g. POSIX, OneAPI, etc.) and workloads.

[0032] Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by providing BiFrost, a composable network architecture that enables shared memory systems across multiple chiplets or packages. BiFrost provides a package architecture with scalable, configurable high-bandwidth connectivity and fault tolerance. For example, the package architecture can leverage routers as fundamental building blocks, allowing chiplets to connect modularly to enable seamless scalability. This approach can maintain critical performance metrics including bandwidth, latency, and throughput. BiFrost may also implement an address-to-endpoint mapping technique using a configurable translation logic, named BF-TAC (BiFrost-Token Address Compute). BF-TAC may be a combination of look-up table (LUT) and computational logic. BF-TAC may have robust fault tolerance features that can enhance system reliability by maintaining operations during hardware failures, supporting sustained uptime for critical Al and high performance computing (HPC) applications.

[0033] In various embodiments of the present disclosure, a package, which may be an IC package may include chiplets. Each chiplet may be referred to as an IC device. Each chiplet may include one or more compute cores and its local dedicated physical memory. The physical memory may be a stacked High Bandwidth Memory (HBM). Each chiplet may also include other types of data storage (such as scratchpad, buffer, cache, etc.) and control modules (such as direct memory access (DMA) engine, microcontroller, etc.). Each chipletmay further have a network that provides inter-chiplet and intra-chiplet connectivity. The network may be a network of switches and may also be referred to as a switch network. A switch may be a router. The switch network may have a hierarchical tree structure that includes one or more edge levels and a spine level. In an example, the spine level may be arranged between two edge levels. Each level may include a series of switches. A switch may be communicatively connected to one or more switches in a neighboring level. Each switch in an edge level may also be coupled to at least one address module.

[0034] The address module may implement BF-TAC described above. The address module may have address compute logic that maps a physical address to a network endpoint. The network endpoint may be a component in the chiplet or a scale-out network. The scale-out network may be implemented using an input / output (I / O) chiplet in the package. The address module may also have route compute logic that determines a data transfer path, e.g., by using a configurable table. The route compute logic may also provide security check functions. The address module may further have packet generate unit that generates a packet. The packet may include the data to be transferred and metadata. The metadata may be in a header of the packet. The packet may be sent along the path. The address module can handle memory failures by remapping predetermined paths to alternative paths. The configurable routing logic along with BF-TAC provides capability to implement multiple scale-up and scale-out network topologies, such as HyperX, Dragonfly, and so on. This can be achieved by mapping compute, memory and control logic to a high bandwidth, multi-flit packet network providing a scalable solution. The network and address modules in each chiplet may facilitate connectivity between packages or between racks. A rack may be a stack of multiple packages. With the chiplet network and address modules, switches outside chiplets may not be needed.

[0035] BiFrost can provide high-performance interconnect between compute cores and the local memory to allow the compute cores to maximally take advantage of the high bandwidth offered by the local memory. It also provides point-to-point logical topological connection between any two chiplets in the system (e.g. a package) such that any compute core in one chiplet can access a physical address on local memory stacked on the other chiplet. It can maximize bandwidth utilization from the local memory and can enlist the full compacity of the sum of all memories across all chiplets, thus capable of supporting DNNs (e.g., LLMs) and compatible with POSIX shared virtual memory programming models likeOpenMP, OneAPI, and all legacy Al and HPC workloads built on top of these standard runtime environments.

[0036] BiFrost can architecturally expose the configuration management of the composability as system architectural features that OS can directly use in ways akin to feature management (e.g., CPUID, etc.) by all the mainstream modern OSes. The observability and controllability offered through such system architectural interfaces can be consistent with the legacy system architectural support to graceful degradation upon machine fault handling and quality-of-service management in adaptive resource provisioning in virtual channel resource managements to adapt to bandwidth requirement for performance or survivability. By articulating these features as part of the scalable interconnect design, the legacy mainstream OSes, for all variations of the composable configurations described herein can naturally support these features, which can be critical for mission-critical tasks like large scale training that can take months of execution across a large network of distributed computing nodes using building blocks like those BiFrost is designed to support optimally across multiple scales (in SoC, across chiplets in package, across packages, etc.).

[0037] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it can be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or / and that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0038] Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0039] Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter.However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order fromthe described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0040] For the purposes of the present disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0041] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives "first," "second," and "third," etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0042] In the following detailed description, various aspects of the illustrative implementations are described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0043] The terms "substantially," "close," "approximately," "near," and "about," generally refer to being within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between the elements, generally refer to being within + / - 5-20% of a target value as described herein or as known in the art.

[0044] In addition, the terms "comprise," "comprising," "include," "including," "have," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or DNN accelerator that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, device, or DNN accelerators. Also, the term "or" refers to an inclusive "or" and not to an exclusive "or."

[0045] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.

[0046] FIG. 1A illustrates a package 100 with a composable network architecture, in accordance with various embodiments. The package 100 may also be referred to as an IC package or semiconductor package. The composable network architecture may also be referred to as a BiFrost architecture. As shown in FIG. 1A, the package 100 includes chiplets 110 (individually referred to as "chiplet 110"), two I / O chiplets 120 (individually referred to as "I / O chiplet 120"), and a substrate 130. In other embodiments, alternative configurations, different or additional components may be included in the package 100. For example, the package 100 may include a different number of I / O chiplets or substrates. Further, functionality attributed to a component of the package 100 may be accomplished by a different component included in the package 100 or a different system. For instance, functionality attributed to the I / O chiplets 120 may be accomplished by one or more chiplets 110.

[0047] The chiplets 110 include compute, control, or memory functions. The chiplets 110 may be individual functional blocks designed to perform specific tasks within the package 100. The tasks may include Al tasks performed using DNNs. For instance, one or more chiplets 110 may be used to train or deploy a DNN, such as transformer-based model, convolutional neural network (CNN), and so on. A chiplet 110 may include one or more layers. Each layer may be a die or wafer. In the embodiments of FIG. 1A, a chiplet 110 includes two layers: a memory layer 140 and a compute layer 150. The memory layer 140 may include a physical memory that is local to the chiplets 110. The memory may be a dynamic random-access memory (DRAM), a stacked HBM, and so on. In some embodiments, the memory may offer significantly higher capacity and bandwidth than traditional local static random-access memory (SRAM). For example, for many currently available Al accelerator designs, the largest local SRAM per chiplet is at 256MB in capacity and 3-4 TB / s in bandwidth. In comparison, stacked HBM (e.g. HBM4) can offer 32 to 64 GB in capacity with 4-12 TB / s in bandwidth, per chiplet.

[0048] The compute layer 150 may include one or more compute devices, such as compute cores in CPU, GPU, NPU, or other types of processing units. The compute layer 150 mayinclude a compute core array. In some embodiments, the compute layer 150 may implement a particular type of processing unit. In other embodiments, the compute layer 150 may implement multiple types of processing unit. The memory layer 140 or compute layer 150 may also include other components that can be combined with the memory and compute cores to create a powerful system-on-chip (SoC). Examples of these components include scratchpad, buffer, cache, microcontroller, DMA engine, and so on. The compute layer 150 may also be referred to as a logic layer with complementary metal-oxide-semiconductor (CMOS) devices. Even though each chiplet 110 in FIG. 1 has two layers, a chiplet 110 may include more layers in other embodiments.

[0049] Each chiplet 110 may be equipped with mechanisms for address map management and network switching capabilities that enable flexible system configurations supporting both scale-up and scale-out deployments. In some embodiments, each chiplet 110 may include a network (not shown in FIG. 1A) that facilitates intra-chiplet connectivity (e.g., connectivity among components of the chiplet 110) and inter-chiplet connectivity (e.g., connectivity between the chiplet and another chiplet, which may or may not be in the package 100). The network may be referred to as a BiFrost network or switch network. In some embodiments, the network may be a network of switches. The switches may be arranged in a hierarchical architecture. By implementing such networks on the chiplets 110, no switches or routers may be needed outside the chiplets 110. That way, chiplet-scale implementation can be achieved. The package 100 is scalable. This topology can enable high-bandwidth, fault tolerant design. Certain aspects regarding BiFrost network are described below in conjunction with FIGS. 4 and 5.

[0050] In some embodiments, the chiplets 110 may be individual semiconductor dies that are integrated together to form a larger IC package, i.e., the package 100. They can offer advantages such as improved flexibility, scalability, cost efficiency, and performance optimization in the design and manufacturing of semiconductor devices. The chiplets 110 may be manufactured separately and then integrated together on the substrate 130 to create a more complex and powerful integrated circuit. The package 100 may be built on holistic system design principles that can enable seamless connectivity across chiplets and support both scale-up and scale-out configuration. Certain aspects regarding chiplets are described below in conjunction with FIGS. 2 and 3.

[0051] The I / O chiplets 120 provide scale-up or scale-out connectivity for the package 100. In some embodiments, the I / O chiplets 120 may perform protocol conversation to enable UCIe. UCIe may serve as the chiplet interface and enable connectivity via unified access layer (UAL) and ultra ethernet consortium (UEC) through protocol conversion in the I / O chiplets 120. The BiFrost architecture can maintain interoperability with industry standard. In some embodiments, the I / O chiplets 120 may provide point-to-point connectivity. To enhance scale-out link-level resiliency, error control mechanisms (such as Cyclic Redundancy Check and Forward Error Correction) may be appended to scale-out packets before transmission. The I / O chiplets 120 may translate the BiFrost protocol to various industry standards, including Ethernet, Infiniband, or Ultra Ethernet. Physical implementations may leverage silicon photonics, or adapt to conventional fiber optic connectors.

[0052] In some embodiments, scale-up networks are designed for ultra-high interconnect bandwidth coupled with low latency, often extending beyond a single rack and requiring advanced interconnect solutions like in-package optical I / O. In a scale-up data storage architecture, storage drives may be added to increase storage capacity and performance. The drives may be managed by two controllers. When storage capacity is run out, another shelf of drives can be added to the architecture. A scale-out network may use software-defined storage (SDS) to separate the storage hardware from the storage software. The software may act as the controllers. Scale-out storage may be network attached storage (NAS). Scale-out NAS systems may involve clusters of software nodes that work together. Nodes can be added or removed, allowing things like bandwidth, compute, and throughput to increase or decrease as needed. New clusters can be created to upgrade a scale-out system.

[0053] The substrate 130 may be a package substrate. The substrate 130 may be integrated with the chiplets 110 and I / O chiplets 120. In some embodiments, the substrate 130 may be formed of one or more dielectric materials, such as ceramic, glass, combination of organic and inorganic materials, buildup film, epoxy film having filler particles therein, other types of dielectric materials, or some combination thereof. The substrate 130 may have embedded portions made of different materials. In some embodiments, the substrate 130 may have conductive pathways extending through the dielectric material between its top surface and bottom surface.

[0054] FIG. 1B illustrates a package 105, in accordance with various embodiments. The package 105 may be an example of the package 100 in FIG. 1A. In this example, the package 105 has 24 chiplets 115 and two I / O chiplets 125 arranged on a substrate 135. There are connections 145 among the chiplets 115. The connections 145 may be connected to BiFrost networks within the chiplets 115 or I / O chiplets 125. A connection may be a fabric that can transfer data. As described above, each chiplet 110 may serves as a fundamental building block, equipped with mechanisms for address map management and network switching capabilities that enable flexible system configurations supporting both scale-up and scale-out deployments.

[0055] The BiFrost package architecture illustrated in FIGS. 1A and IB can provide a unified and holistic solution to the system network architecture which spans within SoC and across chiplets, package and system levels using a distributed network implementation. Such an architecture can enable shared physical memory on a SoC and memory having across chiplets within a package or between multiple packages. It can provide high bandwidth data communication for the SoC components and flexible to scale-up and scale-out using multiple network topologies. Further, the package can provide tolerance for memory faults, such as static or dynamic memory faults. In some embodiments, the BiFrost architecture can provide foundational capabilities for a Distributed Global Address Space (DGAS), incorporating hardware optimizations to enhance data transfer efficiency in both bandwidth and latency.

[0056] The system architectural interfaces can provide observability and controllability regarding all aspects of interconnect resource management, including machine check architecture to monitor, record, report, isolate, survive through graceful degradation various potential point of failures across the interconnect. These architectural features may be congruent to the legacy machine check architecture interface and mechanism for the mainstream OSes to handle and manage. The system architecture interface for QoS management may include adaptive provisioning of critical resources like virtual channel allocation to perform just-in-time bandwidth support for hotspots for the most critical aspect (including phases) of workload execution through entire lifetime of workload execution across the full span of the systems. All of these may be congruent with the legacy prevailing runtime systems such as OneAPI, including POSIX compliant shared virtualmemory, multithreading models like OpenMP, and multi-cluster multiprocessing runtime like Modin based on MPI, etc.

[0057] By construction, BiFrost can architecturally expose the configuration management of the composability as system architectural features that OS can directly use in ways akin to feature management (such as CPUID, etc.) by various OSes. In particular, the full observability and controllability offered through such system architectural interfaces on the BiFrost interconnect resource provisioning and potential faults occurrence and location, can be consistent with the legacy system architectural support to graceful degradation upon machine fault handling and quality-of-service management in adaptive resource provisioning in virtual channel resource managements to adapt to bandwidth requirement for performance or survivability. By articulating these features as part of the scalable interconnect design, many Oses (such as the legacy mainstream Oses) for all variations of the composable configurations shown above can naturally support these features, which can be critical for mission-critical tasks like large scale training that can take months of execution across a large network of distributed computing nodes using building blocks like those BiFrost is designed to support optimally across multiple scales (in SoC, across chiplets in package, across packages). The BiFrost network architecture can provides fundamental capability to compose various OS-visible configurations out of chiplets. Certain aspects regarding OS-visible configurations are described below in conjunction with FIGS. 8-10.

[0058] FIG. 2 illustrates an architecture of a chiplet 200, in accordance with various embodiments. The chiplet 200 may be an IC device with compute, control, and memory functions. The chiplet may be an example of the chiplets 110 in FIG. 1A and chiplets 115 in FIG. 1B. As shown in FIG. 2, the chiplet 200 includes a compute core array 210, network 220, memory 230, accelerator 240, microcontroller 250, scratchpad 260, cache 270, and buffer 280. In other embodiments, the chiplet 200 may include fewer, more, or different components.

[0059] The compute core array 210 includes compute cores that may be arranged in rows and columns. For the purpose of illustration, the compute core array 210 in FIG. 2 has 16 compute cores arranged in four rows and four columns. In other examples, the compute core array 210 may include fewer or more compute cores. The shape of the compute core array 210 may be different. Each compute core is represented by a box in FIG. 2. In some embodiments, the compute core array 210 may be a processor, such as a CPU, GPU, NPU,and so on. Every compute core may be an individual processing unit in the processor,. The compute cores may handle tasks simultaneously. Each compute core can execute instructions for control or calculation purpose. These instructions may range from basic arithmetic to complex tasks like language processing, image rendering, and so on. The implementation of multiple cores in the chiplet 200 can significantly enhance computational speed and efficiency of the chiplet 200, enabling more complex and demanding operations.

[0060] The network 220 facilitates inter-chiplet and intra-chiplet connectivity. The network 220 includes one or more switches. In some embodiments, switches in the network 220 includes are arranged in a hierarchical tree structure. The hierarchical tree structure may be referred to as FatTree. The hierarchical tree structure may include a plurality of levels, such as edge level and spine level. The architecture of the chiplet 200 can provide fundamental interconnect capabilities. Certain aspects regarding FatTree are described below in conjunction with FIGS. 4 and 5.

[0061] The memory 230 may be a high bandwidth memory. The memory 230 may be used as the main memory of the chiplet 200. The accelerator 240 and microcontroller 250 may facilitate management of the memory 230. In some embodiments, the accelerator 240 may be a DMA engine. The scratchpad 260, cache 270, and buffer 280 may provide extra data storage in addition to the memory 230. In some embodiments, the scratchpad 260, cache 270, or buffer 280 may provide temporary data storage, while the memory 230 may provide relatively permanent data storage. The scratchpad 260 may also be referred to as a scratchpad memory. It may be a random-access memory (RAM). The scratchpad 260 may be an internal memory that is used for temporary storage of calculations, data, and other work in progress. In some embodiments, the scratchpad 260 is a special high-speed memory used to hold small items of data for rapid retrieval for the compute core array 210. The scratchpad 260 may be used as a bump storage of the main memory, e.g., the memory 230. The cache 270 may be a data storage used to store frequently accessed information for faster response time. In some embodiments, the cache 270 may function as a buffer between the compute core array 210 and the memory 230. The buffer 280 may hold data while the data is being transferred between different parts of the chiplet 200, such as data being transferred between the compute core array 210 and the accelerator 240, data being transferred between the compute core array 210 and the microcontroller 250, and so on.

[0062] In some embodiments, the memory 230 may be implemented on a different die from the compute core array 210. For instance, the compute core array 210 may be on a compute die, while the memory 230 may be on a memory die. In some embodiments, the accelerator 240 (e.g., a DMA engine), microcontroller 250, scratchpad 260, cache 270, and buffer 280 may also be on the memory die. In some embodiments, at least part of the network 220 may be on the same die as the compute core array 210 or the memory 230.

[0063] FIG. 3 illustrates a perspective view of a chiplet 300 in a package, in accordance with various embodiments. The chiplet 300 may be an example of the memory base 310 in FIG.1A, the chiplets 115 in FIG. 1B, or the chiplet 200 in FIG. 2. The chiplet 300 includes four memory bases 310 (individually referred to as "memory base 310"), a logic base 320, and a plurality of through-silicon vias (TSVs) 330 (individually referred to as "TSV 330"). In other embodiments, the chiplet 300 may include fewer, more, or different components.

[0064] Each memory base 310 may be a memory layer, such as a memory die or memory wafer. In some embodiments, each memory base 310 includes one or more memory arrays. A memory array may include memory cells arranged in rows and columns, one or more bit lines, and one or more word lines. An example memory cell may include a memory element and one or more access transistors. A row memory cells may be coupled to a common word line. A column of memory cells may be coupled to a common bit line. A memory cell may be activated or accessed (e.g., for data read or write operations) using the corresponding word line and bit line. In some embodiments, each memory base 310 may be a DRAM device that includes DRAM arrays.

[0065] The logic base 320 may be a logic layer, such as a logic die or logic wafer, which may also be referred to as a compute die / wafer or compute logic die / wafer. The logic base 320 may include one or more processing units that can execute instructions. The processor(s) may perform various tasks, including Al tasks. In some embodiments, the logic base 320 may control the memory bases 310. The logic base 320 may include one or more logic circuits that control operations of the memory bases 310. A logic circuit may be a peripheral circuit. A logic circuit may include transistors fabricated through a CMOS process. For instance, the transistors can be used to form sense amplifiers, row decoders, column decoders, word line drivers, timers, multipliers, CMOS logic, SRAM cells, power delivery network, signal delivery network, or other types of devices or circuits in the logic base 320.

[0066] As shown in FIG. 1A, the memory bases 310 are at the frontside of the logic base 320. Additionally or alternatively, the chiplet 300 may include one or more memory bases at the backside of the logic base 320 in other embodiments. In some embodiments, the memory bases 310 may be fabricated through a different process from the logic base 320. For instance, the logic base 320 may be fabricated using one or more CMOS processes, in which MOSFETs may be formed. The CMOS process(es) may be part of a FEOL process. The MOSFETs may be used in logic circuits in the logic base 320 for controlling operations of the memory cells in memory base 310. In contrast, transistors of memory cells in the memory bases 310 may be fabricated through a BEOL process, which does not require CMOS fabrication. A base, either a memory base 310 or the logic base 320, may include a support structure on which transistors are implemented. The transistors may be transistors in memory cells or transistors in logic cells. The support structure may be a semiconductor structure, such as a silicon structure.

[0067] The TSVs 330 extend through the four memory bases 310 and logic base 320. A TSV 330 may include an electrically conductive material, such as copper, tungsten, other types of electrically conductive materials, or some combination thereof. A TSV 330 may provide an electrical pathway etched through at least one base, enabling the stacking of multiple bases on top of each other. The TSVs 330 may facilitate signal transmission between the memory bases 310 and logic base 320. For example, signals may be transferred between a memory base 310 and the logic base 320 through one or more TSVs 330. As another example, signals may be transferred between a memory base 310 and another memory base 310 through one or more TSVs 330. A TSV 330 may extend from the top surface of a semiconductor structure of a base to the bottom surface of the semiconductor structure. Even though the TSVs 330 in FIG. 3 form electrical pathways through all the bases, the chiplet 300 may include electrical pathways through some (but not all) bases in other embodiments.

[0068] The interposer 340 may provide an intervening substrate used to bridge the circuit board 2302 and the IC package 2320. Generally, the interposer 340 may spread a connection to a loose pitch or reroute a connection to a different connection. For example, the interposer 340 may couple the IC package 2320 (e.g., a die) to a BGA of the coupling components 2316 for coupling to the circuit board 2302. In the embodiment illustrated in FIG. 12, the IC package 2320 and the circuit board 2302 are attached to opposing sides of the interposer 340; in other embodiments, the IC package 2320 and the circuit board 2302may be attached to a same side of the interposer 340. In some embodiments, three or more components may be interconnected by way of the interposer 340.

[0069] The chiplet 300 is implemented on an interposer 340 of the package. In some embodiments, the interposer 340 may be formed of an epoxy resin, a fiberglass-reinforced epoxy resin, a ceramic material, or a polymer material such as polyimide. In some implementations, the interposer 340 may be formed of alternate rigid or flexible materials that may include the same materials described above for use in a semiconductor substrate, such as silicon, germanium, and other group lll-V and group IV materials. The interposer 340 may include metal interconnects or vias, including but not limited to TSVs. The interposer 340 may further include embedded devices, including both passive and active devices. Such devices may include, but are not limited to, capacitors, decoupling capacitors, resistors, inductors, fuses, diodes, transformers, sensors, ESD protection devices, and memory devices. More complex devices such as further radio frequency (RF) devices, power amplifiers, power management devices, antennas, arrays, sensors, and microelectromechanical systems (MEMS) devices may also be formed on the interposer 340. The package-on-interposer structure 2336 may take the form of any of the package-on-interposer structures known in the art.

[0070] The interposer 340 is implemented on a substrate 350 of the package. The interposer 340 is a layer over the substrate 350 and may be attached to the substrate 350 through one or more adhesive materials. The substrate 350 may provide physical support to the chiplet 300. One or more other chiplets may be implemented on the substrate 350 through the interposer 340. The substrate 350 may be an example of the substrate 130 in FIG. 1A or the substrate 135 in FIG. 1B

[0071] FIG. 4 illustrates a switch network 400 having a three-level FatTree topology, in accordance with various embodiments. The switch network 400 may be implemented within a chiplet, which may be an example of the chiplets 110 in FIG. 1A, chiplets 115 in FIG. 1B, chiplet 200 in FIG. 2, or chiplet 300 in FIG. 3. The switch network 400 may be an example of the network 220 in FIG. 2. The switch network 400 can provide both intra-chiplet connectivity and inter-chiplet connectivity. As shown in FIG. 4, the switch network 400 includes switches 410A-410D (collectively referred to as "switches 410" or "switch 410"), switches 420A-420H (collectively referred to as "switches 420" or "switch 420"), andswitches 430A-430H (collectively referred to as "switches 430" or "switch 430"). In other embodiments, the switch network 400 may include fewer, more, or different components.

[0072] The switches 410 are in the first level of the switch network 400. The switches 420 are in the second level of the switch network 400. The switches 430 are in the third level of the switch network 400. In some embodiments, the first level is referred to as a spine level, the second level is referred to as LI, and the third level is referred to as L0. The switches 410 are also referred to as spine switches. The switches 430 are also referred to as edge switches. In some embodiments, switches in a level may communicate with switches in an immediately adjacent level through connections between the two levels. The connections, which are illustrated as double headed arrows in FIG. 4, may be fabrics.

[0073] A switch in the switch network 400 (e.g., a switch 410, switch 420 or switch 430) may receive data from or send data to another switch or an address modules. As shown in FIG. 4, address modules 401 (individually referred to as "address module 401") and address modules 402 (individually referred to as "address module 402") are placed between the switch network 400 and other components of the chiplet. The address modules 401 facilitate connection between the switch network 400 and other components of the chiplet, such as compute, memory, or other types of components. Each address modules 401 may have various functions, such as packet formation, routing logic, address translation, security checks, other functions, or some combination thereof. As shown in FIG. 4, each of the switches 430A-430F is communicatively connected to two address modules 401. Each address module 401 may include reconfigurable logic that can provide address translation or other functionalities. The endpoints (e.g., hosts) may be mapped to compute, memory and other components providing high bandwidth network connectivity. In the example shown in FIG. 4, the endpoints are mapped to compute cores, HBM access, scratchpad, and LI memory (shown as "Ll$" in FIG. 4) in the chiplet. The address modules 401 can support multiple network topologies, such as HyperX, Dragonfly, and so on.

[0074] The switch 430G and switch 430H are each coupled to two address modules 402. An address module 402 may have the functionality of an address module 401. Additionally, each address module 402 may integrate with Universal Chiplet Interconnect Express (UCIe) protocol. For instance, each address modules 402 may include logic for UCIe connectivity. UCIe is a chip-to-chip interconnect technology that allows for high-speed data transfer between different chiplets within a package. The switch 430G and switch 430H plus theaddress modules 402 may facilitate scale-up connectivity. Scale-out, also referred to as the front-end network, is the network architecture that interconnects separate compute clusters in a data center. It typically uses standard networking technologies like InfiniBand or Ethernet. Scale-out networks are well-suited for serving datacenter-wide communications, providing connectivity between distinct compute clusters. These networks often feature tapered bandwidth and are designed to handle communications across larger distances within the data center infrastructure.

[0075] In the example shown in FIG. 4, the switch 430A communicates with the switches 420. The switch 420A communicates with the switches 410 as well as scale-up network switches 403. The scale-up network may be a network architecture that interconnects Al compute clusters together, typically using standard networking topologies. The scale-up network switches 403 may provide multi-chiplet connectivity. For example, the scale-up network switches 403 may be communicatively connected to scale-up network switches of another chiplet.

[0076] In some embodiments, each switch may be a router. A switch can filter and forward data to a specific device within the switch network 400, the chiplet, or the package, e.g., through packet switching. In some embodiments, a switch can provide dedicated bandwidth to each port, allowing devices to communicate simultaneously without interfering with each other. The switches may provide security features and create separate collision domains for each port, preventing data collisions and unauthorized access to network traffic. Certain aspects regarding BiFrost switch are described below in conjunction with FIG. 11.

[0077] In some embodiments, a switch network (e.g., the switch network 400) may be characterized by two parameters: K and L, where K refers to the number of ports on each switch (also referred to as radix), and L refers to the number of levels in the network hierarchy. The switch network 400 may be organized into L layers of switches, such that the number of routers N and the number of links between them can grow in a predictable manner based on K and L. In some embodiments, the total number of switches Nrouter= (2L — 1)(-)L-1, and the number of hosts Nhost= 2(-)L. In an example, each switch 410, switch 420, or switch 430 may be a radix-16 switch, such as a 16x16 switch. The radix of each address modules 401 may be 8. The switch network 400 may provide 1024 endpoints, which can be arranged in 16 groups of 64 endpoints. Each group may be mapped to achiplet, and it can instantiate some of the spine switches too. The switch network 400 may provide fault tolerance. In some embodiments, the number of paths between two switches Ppaths = KL~ and the number of alternative paths in a case of link failure

[0078] FIG. 5 illustrates a switch network 500 having a two-level FatTree topology, in accordance with various embodiments. The switch network 500 may be implemented within a chiplet, which may be an example of the chiplets 110 in FIG. 1A, chiplets 115 in FIG. 1B, chiplet 200 in FIG. 2, or chiplet 300 in FIG. 3. The switch network 500 may be an example of the network 220 in FIG. 2. The switch network 500 can provide both intra-chiplet connectivity and inter-chiplet connectivity. As shown in FIG. 5, the switch network 500 includes switches 510A-510H (collectively referred to as "switches 510" or "switch 510"), switches 520A-520H (collectively referred to as "switches 520" or "switch 520"), and switches 530A-530H (collectively referred to as "switches 530" or "switch 530"). In other embodiments, the switch network 500 may include fewer, more, or different components.

[0079] The switches 510 and switches 520 are in the first level of the switch network 500. The switch 530 are in the second level of the switch network 500. The second level may be referred to as spine. The switches 510 may be in a level referred to as "LI north" as they are placed at the north side of the spine, while the switches 520 may be in a level referred to as "LI south" as they are placed at the south side of the spine. The switches 510 and switches 520 may be referred to as edge switches. The switches 530 may be referred to as spine switches. In total, the switch network 500 shown in FIG. 5 has 24 switches. In some embodiments, the radix of each switch may be 16. In some embodiments, switches in different levels may communicate through connections. The connections, which are illustrated as double headed arrows in FIG. 5, may be fabrics. In the example shown in FIG.5, the switch 530A communicates with the switches 510, and the switch 520A communicates with the switches 530.

[0080] Address modules 501 (individually referred to as "address module 501") and address modules 502 (individually referred to as "address module 502") are placed between the switch network 500 and other components of the chiplet. As shown in FIG. 5, each edge switch is coupled to two address modules. For instance, each of the switches 510A-510F is coupled to two address modules 501 at the north side, and each of the switches 530A-530F is coupled to two address modules 501 at the south side. Also, the switch 510G, switch510H, switch 530G, and switch 530H are each coupled to two address modules 502. Each address module 501 may include reconfigurable logic that can provide address translation or other functionalities. The endpoints (e.g., hosts) may be mapped to compute, memory and other components providing high bandwidth network connectivity. In the example shown in FIG. 5, the endpoints are mapped to compute cores, scale-out networks, HBM access, scratchpad, and LI memory (shown as "Ll$" in FIG. 5) in the chiplet. In some embodiments, an address modules 501 may be the same as an address module 401 in FIG.4. An address module 502 may be the same as an address module 402 in FIG. 4. As shown in FIG. 5, the switch 510G, switch 510H, switch 530G, and switch 530H can each communicate with a scale-out network in the chiplet.

[0081] A switch in the switch network 500 (e.g., a switch 510, switch 520 or switch 530) may receive data from or send data to another switch within the switch network 500 or an address modules 501. In some embodiments, each switch may be a router. A switch can filter and forward data to a specific device within the switch network 500, the chiplet, or the package, e.g., through packet switching. In some embodiments, a switch can provide dedicated bandwidth to each port, allowing devices to communicate simultaneously without interfering with each other. The switches may provide security features and create separate collision domains for each port, preventing data collisions and unauthorized access to network traffic. In some embodiments, a switch 510, switch 520, or switch 530 may be a 16x16 switch.

[0082] While the same switching capability could be achieved using a configuration with radix-128 switching using 80 radix-8 routers in 3-levels, the configuration of the switch network 500 can optimize the area-latency tradeoff. There may be an alternative configuration with a three-level network capable of providing 328 TB / s per package by distributing 320 routers across 16 chiplets. While this configuration can reduce overall router requirements, it can increase inter-chiplet connectivity demands. Other similar configurations may either require substantial number of routers per chiplet or offer lower than required 10's of TB / s bandwidth. Compared with these alternative configuration, the configuration of the switch network 500 can be more advantages for both chiplet and package level.

[0083] At the package level, chiplets may connect via this two-level FatTree topology where each chiplet functions as a 16x16 switch. The inter-chiplet connectivity follows the patternshown in FIG. 5, with each package-level connection carrying two transactions. This architecture can logically combine two endpoints per chiplet for scale-up connectivity, representing a 32x32 switch as a 16x16 switch to facilitate 24 chiplet organization in the example shown in FIG. 5. The switch network 500 may provide 128x128 switching capabilities. In some embodiments, a chiplet in the package (e.g., the chiplet on which the switch network 500 is implemented) may have 128 network endpoints. Out of the 128 network endpoints, 32 network endpoints may be dedicated to scale-up connectivity while 96 network endpoints may serve memory, compute, accelerators, and control components. In some embodiments, the switch network 500 may operate at 2 GHz. The switch network 500 may achieve 160 GB / s uni-directional raw bandwidth per connection.

[0084] In some embodiments, the network protocol may optimize for data transfers of 256-Bytes and beyond, implementing an 80-Byte network packet flit width. A 256-Byte data payload requires four flits, accounting for 20% overhead to accommodate network inefficiencies. This overhead encompasses network saturation (e.g., 85-90%) and packet, flit header requirements.

[0085] The seamless integration between chiplet-level and package-level networks is novel and advantageous. While both levels implement 2-level FatTree topologies, they can operate at different scales and serve complementary purposes. The chiplet-level network, with its 128x128 switching capability, can optimize local compute and memory access patterns. The package-level network can extend this architecture across chiplets, logically representing each chiplet as a 16x16 switch in a larger FatTree. This hierarchical approach can maintain consistent routing protocols and latency characteristics across both levels while enabling flexible scaling. When traffic moves between chiplets, the address modules can handle protocol translation and maintains addressing consistency, creating a unified network that appears homogeneous to software despite its 2-level physical implementation.

[0086] FIG. 6 illustrates address translation in a package, in accordance with various embodiments. For the purpose of illustration and simplicity, FIG. 6 shows two chiplets: chiplet 610 and chiplet 620 in the package. The package may have more chiplets. FIG. 6 also shows a shared memory 630 that is accessible to the chiplet 610 and chiplet 620. In some embodiments, the shared memory 630 may be a memory in the chiplet 610 or chiplet 620. The chiplet 610 and chiplet 620 may be arranged on the same package or separatepackages. The system memory address space shown in FIG. 6 can span across multiple chiplets or multiple packages and can be shared across multiple jobs.

[0087] The chiplet 610 has a virtual address table 613 and a physical address table 615. The virtual address table 613 lists virtual addresses for workloads assigned to the chiplet 610. A virtual address is an address in a virtual memory. The physical address table 615 lists physical addresses for the workloads. A physical address is an address in a physical memory. For the purpose of illustration, the virtual address table 613 lists virtual addresses for three workloads: virtual address 612A, virtual address 612B, and virtual address 612C. Each virtual address is translated to two physical addresses listed in the physical address table 615 by a memory management unit (MMU) 614. The MMU 614 may use a translation lookaside buffer (TLB) to speed up address translation. For instance, when a virtual address in the virtual address table 613 is accessed, the MMU 614 may check whether the translation to the physical address(es) is cached in the TLB. In embodiments where the translation is in the TLB (i.e., a TLB hit occurs), the MMU 614 may retrieve the physical address(es) from the TLB and bypass the slower page table walk. In embodiments where the TLB does not have the translation (i.e., a TLB miss occurs), the MMU 614 may walk the page tables in a memory (e.g., the main memory of the chiplet) to find the physical address(es). The MMU 614 may use the virtual address to find the corresponding entry in the page tables. The new physical address(es) may then be stored in the TLB for future use. The TLB update may be done by the MMU 614 caching the newly found translation in the TLB. The MMU 614 may use the TLB as a high-speed cache.

[0088] In the example illustrated by FIG. 6, the MMU 614 translates the virtual address 612A into a physical address 616A and a physical address 618A. The MMU 614 also translates the virtual address 612B into a physical address 616B and a physical address 618B. The MMU 614 further translates the virtual address 612C into a physical address 616C and a physical address 618C. The physical address 616A, physical address 616B, and physical address 616C may be physical addresses used by the operating system (OS), which may run on a CPU in the chiplet 610. The physical address 618A, physical address 618B, and physical address 618C may be physical addresses used by applications to perform the corresponding workloads. In some embodiments, the physical address 618A, physical address 618B, and physical address 618C may be used by a GPU or NPU for performing DNN inference workloads.

[0089] The chiplet 620 has a virtual address table 623 and a physical address table 625. The virtual address table 623 lists virtual addresses for workloads assigned to the chiplet 620. A virtual address is an address in a virtual memory of the chiplet 620. The physical address table 625 lists physical addresses for the workloads. A physical address is an address in a physical memory of the chiplet 620. For the purpose of illustration, the virtual address table 623 lists virtual addresses for two workloads: virtual address 622A and virtual address 622B. Each virtual address is translated to two physical addresses listed in the physical address table 625 by a MMU 624. The MMU 624 may be the same as or similar to the MMU 614. In the example illustrated by FIG. 6, the MMU 624 translates the virtual address 622A into a physical address 626A and a physical address 628A. The MMU 624 also translates the virtual address 622B into a physical address 626B and a physical address 628B. The physical address 626A and physical address 626B may be physical addresses used by the operating system (OS), which may run on a CPU in the chiplet 620. The physical address 628A and physical address 628B may be physical addresses used by applications to perform the corresponding workloads. In some embodiments, the physical address 628A and physical address 628B may be used by a GPU or NPU for performing DNN inference workloads.

[0090] Physical addresses in the physical address table 615 and physical address table 625 are translated to system physical addresses through two address modules 640 (individually referred to as "address module 640"), respectively. The system physical addresses are addresses in the shared memory 630. In some embodiments, each system physical address may point to a memory region in the 630. For the purpose of illustration, FIG. 6 shows (n + 1) memory regions: Mothrough Mn. The memory regions may be Odin memory regions. The address module 640 may be used for token address compute. Each address module 640 may be an example of the address modules 401 in FIG. 4, the address modules 501 in FIG. 5, or the address modules 502 in FIG. 5. In some embodiments, each address module 640 may use an address translation table to convert chiplet physical addresses (e.g., the physical addresses in the physical address table 615 and physical address table 625) to system physical addresses. In some embodiments, the address module 640 may also translate system physical addresses to network endpoints, e.g., by using a configurable translation logic. The address modules 640 may perform translation between different transaction formats, such as between chiplet transaction formats to UCIe transaction formats, enabling DGAS hardware implementation beyond chiplet and package boundaries. This integrationcan allow jobs to access memory uniformly across the full system within a single address range. Certain aspects regarding address mapping are described below in conjunction with FIGS. 6 and 7.

[0091] FIG. 7 illustrates an example address module 700, in accordance with various embodiments. The address module 700 may be an example of the address modules 401 in FIG. 4, the address modules 501 in FIG. 5, the address modules 502 in FIG. 5, or the address modules 640 in FIG. 6. As shown in FIG. 7, the address module 700 includes an address translation unit 710, a route compute unit 720, a packet generation unit 730, a secure channel 740, and a configuration route table 750. In other embodiments, the address module 700 may include fewer, more, or different components.

[0092] FIG. 7 shows a dataflow within the address module 700. The address module 700 receives a configuration transaction 701 and an input transaction 702. The input transaction 702 is transmitted to the address translation unit 710. The address translation unit 710 may translate a physical address in the configuration transaction 701 to a system physical address. As an example, the address translation unit 710 may be configured to translate chiplet physical addresses (such as the physical addresses in the physical address table 615 or physical address table 615 in FIG. 6) to system physical addresses (such as addresses of the memory regions in the shared memory 630 in FIG. 6). System may refer to a package or multiple packages.

[0093] In some embodiments, the address translation unit 710 and route compute unit 720 may leverage a combined algorithmic and LUT approach. In some embodiments, the address translation unit 710 may implement up to 32 rules, each comprising several fields, including pivot and shift values, compare high and compare low values, an empty, fixed, and drop mask, and a base address. Each rule may define an address range using "Compare Low and Compare High" fields, along with a "Control" field to enable or disable the rule and enforce permission checks. The address translation unit 710 may also incorporate mechanisms for handling special address bits and hierarchical mapping. The address translation unit 710 may use an Empty Mask to mask out special address bits (bits in the address map that specify a memory space, such as chiplet ID or a scratchpad within a chiplet) and unpopulated address space regions. Conversely, the address translation unit 710 may use a Fixed Mask to insert predefined bits for address remap into the translated address. For flexible address mapping across hierarchies, Pivot and Shift registers may be provided. In anexample, there may be six Pivot and Shift registers. Each register may include a 6-bit pivot and a 6-bit signed shift value, enabling controlled manipulation of address ranges with up to 32-bit shifts (left or right). During address translation, the address translation unit 710 may use a Drop Mask to select relevant bits from the original address. The address translation unit 710 may use a Base Address as the starting point for the translated address. This mechanism can enable efficient virtual-to-physical address mapping, critical for inter-chiplet communication within the shared address space of the multi-chiplet package

[0094] The configuration transaction 701 is transmitted of the configuration route table 750 through the secure channel 740. The configuration route table 750 may provide information to the route compute unit 720. The route compute unit 720 may determine a data transfer path based on the system physical address determined by the address translation unit 710. The path may be start from a source address (e.g., the source from which the input transaction 702 is received) to a destination address (e.g., the system physical address). The route compute unit 720 provides routing logic which may adapt the FatTree hierarchical routing mechanism, where packets travel up towards spine switches and then down to leaf routers towards their destination, leveraging redundant paths for fault tolerance. In some embodiments, the configuration route table 750 may map a Job ID in the configuration transaction 701 to a memory range. This mapping may be consulted after address translation to perform security checks. As shown in FIG. 7, the physical address translated by the address translation unit 710 is provided to the route compute unit 720. The route compute unit 720 may perform a security check using the physical address from the address translation unit 710 and the memory range from the configuration route table 750. The configuration route table 750 may be a LUT-based data structure. In some embodiments, the secure channel 740 is a configuration update channel. Security can be enhanced by configuring the configuration route table 750 through the secure channel 740.

[0095] Transactions that fail the security checks may be dropped. A NACK (Negative acknowledgment) may be sent back to the originator. Transactions that pass the security checks are forwarded to the packet generation unit 730 for packet generation. As shown in FIG. 730, the packet generation unit 730 outputs a packet 703. The packet 703 may also be referred to as a network packet. The packet generation unit 730 may incorporate buffers capable of accommodating data equivalent to the latency-bandwidth product of the underlying network. The packet generation unit 730 may also facilitate the formation ofnetwork packets and flits. The packet 703 may include data to be written into a system memory, such as the shared memory 630 in FIG. 6. The data may be used for performing a workload, such as a DNN workload. The packet 703 may also include a header including metadata for the data. In some embodiments, a network packet header includes destination address, source address, packet type, transaction ID, and transaction type fields. A network package header may also have a process ID field referencing a configurable table mapping processes to memory regions (e.g., the configuration route table 750). The process ID field may enable Virtual Extension to Global Address Space (VEGAS) capabilities. Each flit may carry flit destination address, Head, Tail, Valid information, virtual channel (VC) number, and parity bits for error detection.

[0096] In some embodiments, the address module 700 may orchestrate both local and package-level routing. Inter-switch connectivity, such as the inter-switch connectivity shown in FIG. 4 or FIG. 5, may require careful management, particularly in the middle layers where wire density peaks due to east-west crossings. To maintain a desirable operating frequency (such as a 2 GHz operating frequency), the design may incorporate pipeline stages for long routes between extreme positions.

[0097] In some embodiments, the absolute routing area may depend on specific process technology parameters, including metal layer count, pitch, or frequency requirements. For certain implementation, wiring and interconnect may dominate the overall switching area on silicon. At the package level, routing can maintain consistency with the chi plet-level FatTree topology while enabling efficient scaling. This unified approach to routing and addressing can create a seamless transition between intra-chiplet and inter-chiplet communication, supporting both the flexible system configurations and the fault tolerance mechanisms. Certain aspects regarding fault tolerance are described below in conjunction with FIG. 8.

[0098] FIG. 8 illustrates fault tolerance for memory failure, in accordance with various embodiments. For the purpose of illustration, FIG. 8 shows a multi-chiplet system 800 that includes 24 chiplets. The 24 chiplets may serve as a discrete device to an OS running on a host 805. The host 805 may be a host computer, such as a host personal computer (PC). In other embodiments, the multi-chiplet system 800 may include fewer, more, or different components.

[0099] In the example shown in FIG. 8, a chiplet 810A has an address module 815. The address module 815 determines a default path 817 from the chiplet 810A to another chiplet 810B. For instance, the address module 815 may receive a transaction from a switch in a network implemented on the chiplet 810A and identifies an endpoint pointing to the chiplet 810B. The address module 815 may determine a system physical address referring to a memory region of a memory in the chiplet 810B. The address module 815 may also generate a packet from the transaction and send the packet to the chiplet 810B for writing data in the packet into the memory region. However, a memory failure occurs. For instance, the data in the packet cannot be written into the memory region due to an error or unexpected instance.

[0100] A notification of the memory failure may be sent to the address module 815. In response to receiving the notification, the address module 815 may determine a modified path 819 for the transaction, which is a path from the chiplet 810A to a chiplet 810C. The determination of the modified path 819 may be done by a configuration update. In an example where the address module 815 is the address module 700 in FIG. 7, the configuration transaction 701 may indicate the memory failure and may be provided to the configuration route table 750 through the secure channel 740, and the configuration route table 750 may be updated based on the configuration transaction 701. Configurable rules implemented on the configuration route table 750 may enable the remapping of the faulty memory region to other memory areas within or across the chiplet or package. For instance, the faulty memory region may be remapped to a healthy chiplet. In the example shown in FIG. 8, the faulty memory region is remapped to the chiplet 810C in the multi-chiplet system 800. The route compute unit 720 may be dynamically updated to direct packets to the remapped chiplets, allowing the system to continue operation despite faulty chiplets.

[0101] As described above, the multi-chiplet system 800 is a 24-chiplet package that serves as a discrete device to the OS on the host 805. This configuration may be akin to a host PC system with the multi-chiplet package as a discrete accelerator, such as an Al accelerator. The Al accelerator may be a GPU or NPU. In some embodiments, the multi-chiplet system 800 and host 805 may be on separate packages. The multi-chiplet system 800 and host 805 form an OS-visible configuration, which may preserve full compatibility with legacy software stacks. The OS on the host 805 may manage the multi-chiplet system 800 through one or more device drivers, including device drivers that can support DNN optimization ordeployment. The device drivers may enable immediate compatibility with existing Al frameworks like OpenVINO without requiring modification to the software stack.

[0102] The architecture of the multi-chiplet system 800 may explicitly expose resource management on all aspects of the BiFrost interconnect, including composable topology configuration, observability, controllability, flight recorder of critical events (including performance counter of utilization) including failures, and various self-healing survivability measures (including health routing map in presence of failed nodes). These features can be congruent and natural extension to the existing machine check architectures that are fully utilized by the modern mainstream OSes to manage RAS and QoS. In some embodiments, the architecture may define a set of addressable control registers that can be exposed at PCIe compatible ranges thus allowing all mainstream OS to enumerate the underlying control configurations. With these control registers, the architecture of the multi-chiplet system 800 can be composed to support various legacy compliant (e.g., PC AT compatible) configurations. Other system architectures, such as the ones shown in FIGS. 9 and 10, can also be composed to support various legacy compliant configurations.

[0103] FIG. 9 illustrates a multi-chiplet system 900, in accordance with various embodiments. The multi-chiplet system 900 may be a single package, which may be an example of the package 100 in FIG. 1A or the package 105 in FIG. 1B. For the purpose of illustration, the multi-chiplet system 900 has 24 chiplets. One of the 24 chiplets serves as an OS-visible controller. The other 23 chiplets may serve as sequestered accelerator subsystem. This architecture may constitute an integrated graphics model where a GPU or NPU is integrated with the CPU via chipset and both sharing the same physical address space. With the BiFrost package architecture, this model is extended significantly by supporting up to 23 sequestered integrated accelerator chiplets in a 24-chiplet package, all sharing a unified physical address space while maintaining traditional OS memory management paradigms.

[0104] FIG. 10 illustrates a multi-chiplet system 1000, in accordance with various embodiments. The multi-chiplet system 1000 may be a single package, which may be an example of the package 100 in FIG. 1A or the package 105 in FIG. 1B. The multi-chiplet system 1000 is partitioned into four multi-chiplet subsystems 1010, individually referred to as "multi-chiplet subsystem 1010." Each multi-chiplet subsystem 1010 has six chiplets and may be referred to as a six-chiplet cluster. Within each multi-chiplet subsystem 1010, one of the six chiplets serves as an OS-visible controller, and the other five chiplets may serve assequestered accelerator subsystem. In an example, each multi-chiplet subsystem 1010 may be system in which a GPU or NPU is integrated with a CPU with all the processing units sharing the same physical address space. This partitioned approach can allow each cluster to run a separate OS instance while preserving the shared physical memory benefits within each partition. The partitioning can be dynamically adjusted based on workload requirements, with BiFrost's unified address space ensuring that standard OS memory management and process isolation remain fully functional across all configurations, like Sequencer Virtualization.

[0105] In some embodiments, all the multi-chiplet subsystems 1010 in the multi-chiplet system 1000 may share the same physical address space. In some embodiments, the four hosts in the four multi-chiplet subsystem 1010 may each run a single-system image OS, or in general, a distinct copy of an OS. The multi-chiplet system 1000 may be equivalent to a distributed clustered architecture with four nodes. For such a clustered design, an intercluster programming environment may work more efficiently since the shared memory runtime can be implemented on top of shared physical memory space.

[0106] FIGS. 8-10 show three different configurations, which can be used to establish various additional combinations of globally distributed and locally shared memory subsystem clusters across chiplets inside one package or across multiple packages, all built with the proposed BiFrost interconnect technology. BiFrost's support for legacy OS compatibility can extend beyond basic memory management to encompass the entire software ecosystem. Inter-cluster communication in partitioned configurations can leverage parallel programming frameworks like MPI and OneAPI, but with substantially improved efficiency due to the underlying shared physical memory space. This capability can eliminate the traditional overhead of data copying between separate memory domains while maintaining full compatibility with existing parallel programming models. The architecture's unified memory space and flexible partitioning capabilities can enable legacy applications to benefit from increased bandwidth and reduced latency without requiring code modifications or special programming models.

[0107] Through these comprehensive OS support mechanisms, BiFrost can achieve a critical balance between innovation and backward compatibility. The architecture can enable advanced features like dynamic resource partitioning, fault tolerance, and high-bandwidth inter-chiplet communication while ensuring that existing OSs, device drivers, andapplications continue to function without modification. This approach can significantly reduce adoption barriers while providing a clear path for incremental optimization of legacy software to take full advantage of BiFrost's advanced capabilities.

[0108] FIG. 11 illustrates an example switch 1100, in accordance with various embodiments. The switch 1100 may be a network router. The switch 1100 may be an example of the switches 410, switches 420, switch 430, switches 510, switches 520, or switch 530 described above. The switch 1100 may be a 16x16 router. The switch 1100 may receive 16 inputs (lnput-0 through Input-15 in FIG. 11) and provide 16 outputs (Output-O through Output-15 in FIG. 11) in a cycle. In some embodiments, the switch 1100 may incorporate eight Virtual Channels (VC), each with a 16-entry deep buffer. The switch 1100 also includes two route compute units and a VC arbiter for each input. The VC arbiters are coupled to a flow controller. Also, each VC arbiter is coupled to its own output arbiter and multiplexer (MUX). FIG. 11 shows BiFrost's switch microarchitecture at an abstraction level for the purpose of illustration. In other embodiments, the switch 1100 may include fewer, more or different components.

[0109] In some embodiments, the VC buffers in the switch 1100 may be implemented on one or more memories (e.g., one or more Static Random-Access Memories (SRAMs)) in the chiplet. In an example, 30% area may be allocated for logic, 10% area may be allocated for spacing between macros, and 10% may be allocated for area projections. The chiplet may include a plurality of switches arranged in a tree structure, such as the FatTree structure shown in FIG. 4 or FIG. 5. In some embodiments, each switch can drive over 5 TB / s of bidirectional bandwidth.

[0110] FIG. 12 illustrates system connectivity in a currently available multi-hierarchy rack system 1200, in accordance with various embodiments. For the purpose of illustration, the multi-hierarchy rack system 1200 includes six racks 1210, individually referred to as "rack 1210." Each rack 1210 may include a number of currently available packages stacked over each other. A rack 1210 may include a top switch 1211, a GPU server 1212, and a switch server 1213. The switch server 1213 may have switching blades for scale-up connectivity. The racks 1210 are connected through switches 1220, individually referred to as "switch 1220." As shown in FIG. 12, the five switches 1220 are arranged outside all the racks 1210. Such switches are referred to as Top-of-Rack (TOR) switches. In other embodiments, themulti-hierarchy rack system 1200 may include a different number of racks or a different number of switches.

[0111] Currently available rack-scale systems typically rely on TOR switches for inter-server communication, as shown in FIG. 12. This approach has several limitations. For instance, servers are required to connect through the TOR for routing, which can create a latency bottleneck. Also, dedicated switch hardware adds cost and management complexity.Further, the fixed network topology hinders adaptability to diverse workloads. The TOR can become a performance chokepoint, limiting overall system scalability.

[0112] FIG. 13 illustrates system connectivity in a multi-hierarchy rack system 1300 with BiFrost packages, in accordance with various embodiments. For the purpose of illustration, the multi-hierarchy rack system 1300 includes six racks 1310, individually referred to as "rack 1310." Each rank 1310 may include a number of BiFrost packages stacked over each other. Each BiFrost package includes one or more chiplets. A switch network is implemented on each chiplet. The racks 1310 are connected through the switch networks. Different from the multi-hierarchy rack system 1200, the multi-hierarchy rack system 1300 does not include any TOR switches.

[0113] The BiFrost architecture can eliminate the limitations of TOR switches described above by removing the TOR switch entirely. It can distribute switching functionality across chiplets within each server's multi-chiplet package. Dedicated I / O chiplets on each package can provide point-to-point connectivity. To enhance scale-out link-level resiliency, error control mechanisms such as Cyclic Redundancy Check and Forward Error Correction can be appended to the BiFrost scale-out packet before transmission. These I / O chiplets can translate the BiFrost protocol to various industry standards including Ethernet, InfiniBand, Ultra Ethernet, and so on. Physical implementations may leverage silicon photonics or adapt to conventional fiber optic connectors.

[0114] Address modules associated with switch networks can incorporate fundamental capabilities for defining routing across multiple scale-out network topologies, including DragonFly, HyperX, Slim Fly, PolarFly. This flexibility may require deeper buffers at the router interface to accommodate latency-bandwidth requirements. The BiFrost architecture can enable reduced latency through direct inter-server communication, while simultaneously lowering system cost by eliminating dedicated switches. Additionally, the design can improve scalability as switching capacity inherently grows with server count. The inherentcomposability, observability, and controllability of BiFrost can enable dynamic network resource management, OS-level topology configuration, and built-in fault tolerance. This approach represents a paradigm shift toward unified, software-defined rack-scale systems, offering enhanced flexibility and efficiency for diverse applications while paving the way for seamlessly integrated compute and network resources - a true realization of holistic system design.

[0115] FIG. 14 is a block diagram of a NPU 1400, in accordance with various embodiments. The NPU 1400 may be implemented on a chiplet, which may be an example of the chiplets 110 in FIG. 1A, the chiplets 115 in FIG. 1B, the chiplet 200 in FIG. 2, or the chiplet 300 in FIG.3. The NPU 1400 can execute DNNs. For instance, the NPU 1400 can execute layers in a DNN by carrying out neural network operations in the layers. The layers may be arranged in a sequence, and the NPU 1400 may execute the layers in the sequence. The execution of the DNN may be for training the DNN or for using the DNN to perform Al tasks. The NPU 1400 may also perform computations in backward passes for training DNNs. As shown in FIG. 14, the NPU 1400 includes a memory 1410, a DMA engine 1420, and compute cores 1430 (individually referred to as "compute core 1430"). In other embodiments, alternative configurations, different or additional components may be included in the NPU 1400. For example, the NPU 1400 may include more than one memory 1410 or DMA engine 1420. As another example, the NPU 1400 may include a single compute core 1430. Further, functionality attributed to a component of the NPU 1400 may be accomplished by a different component included in the NPU 1400 or by a different system. A component of the NPU 1400 may be implemented in hardware, software, firmware, or some combination thereof.

[0116] The memory 1410 may be a local physical memory of the NPU 1400. In some embodiments, the memory 1410 is implemented on a memory die of a chiplet. The memory 1410 stores data associated with neural network operations performed by the NPU 1400. In some embodiments, the memory 1410 may store data to be used by the compute cores 1430 for executing neural network operations. The memory 1410 may store inputs to DNNs and outputs of DNNs. The memory 1410 may also store activations (such as input activations and output activations of neural network operations) and weights (such as weights determined by training DNNs) in DNNs. In some embodiments, the memory 1410 may store activations and weights with floating-point precisions, such as FP4, SF4, NF4,FP16, BP16, FP32 and so on. The memory 1410 may also quantized activations or weights. In some embodiments, the memory 1410 includes one or more dynamic random-access memories (DRAMs). In some embodiments, the memory 1410 includes a stacked HBM.

[0117] The DMA engine 1420 facilitates data transfer between the memory 1410 and the compute cores 1430. For example, the DMA engine 1420 can read data from the memory 1410 and write data into a local memory of a compute core 1430. As another example, the DMA engine 1420 can read data from a local memory of a compute core 1430 and write data into the memory 1410. For instance, the DMA engine 1420 may read input activations and weights of convolution from the memory 1410 and load the input activations and weights to one or more compute cores 1430. The DMA engine 1420 may also write output activations of convolutions computed by one or more compute cores 1430 to the memory 1410. The DMA engine 1420 provides a DMA feature that allows the compute core 1430 to initiate data transfer between the memory 1410 and the local memories of the compute cores 1430 and to perform other operations while the data transfer is being conducted. In some embodiments, the DMA engine 1420 may read tensors from the memory 1410, modify the tensors in a way that is optimized for the compute core 1430 before it writes the tensors into the local memories of the compute cores 1430. In some embodiments, the DMA engine 1420 is an accelerator in a chiplet.

[0118] The compute cores 1430 perform neural network operations in DNNs. For instance, a compute core 1430 may execute a DNN layer by running one or more deep learning operations in the DNN layer. A compute core 1430 may execute a layer, or a portion of a layer, at a time. In some embodiments, the operations of the DNN layers may be run by multiple compute cores 1430 in parallel. For instance, multiple compute cores 1430 may each perform a portion of a workload for a neural network operation. Data may be shared between the compute cores 1430. A compute core 1430 may also be referred to as a compute tile. The compute cores 1430 may be capable of running various types of neural network operations, such as convolution, matrix multiplication, Softmax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on. Neural network operations performed by the compute cores 1430 include tensor operations, i.e., operations whose inputs are tensors or operations whose outputs are tensors. In an example, the compute core 1430 receives an input tensor and one or more convolutional kernels and performs a convolution with the input tensor and convolutional kernels. The result of theconvolution may be an output tensor, which can be further computed, e.g., by the compute core 1430 or another compute core 1430. In some embodiments, the compute cores 1430 may be arranged in an array to form a compute core array, an example of which is the compute core array 210 in FIG. 2.

[0119] In the embodiments of FIG. 14, each compute core 1430 includes a local memory 1440, a digital signal processor (DSP) 1450, and a data processing unit (DPU) 1455. The DPU 1455 includes an input delivery unit (IDU) 1460, a processing engine 1470, a post-processing engine 1480, and an output delivery unit (ODU) 1490. Some or all the components of the compute core 1430 can be implemented on the same chip. In other embodiments, alternative configurations, different or additional components may be included in the compute core 1430. Further, functionality attributed to a component of the compute core 1430 may be accomplished by a different component included in the compute core 1430, a different compute core 1430, another component of the NPU 1400, or a different system. A component of the compute core 1430 may be implemented in hardware, software, firmware, or some combination thereof.

[0120] The local memory 1440 is local to the corresponding compute core 1430. The local memory 1440 is accessible to both the DSP 1450 and DPU 1455. In the embodiments of FIG.14, the local memory 1440 is inside the compute core 1430. In other embodiments, the local memory 1440 may be outside the compute core 1430. Data in the local memory 1440 may be transferred to or from the memory 1410, e.g., through the DMA engine 1420. In some embodiments, data in the local memory 1440 may be transferred to or from the local memory of another compute core 1430. The local memory 1440 may store data received, used, or generated by the IDU 1460, the processing engine 1470, the post-processing engine 1480, or the ODU 1490. Examples of the data may include input activations, weights, output activations, configuration parameters, and so on.

[0121] In some embodiments, the local memory 1440 includes one or more static randomaccess memories (SRAMs). The local memory 1440 may be byte-addressable, and each memory address identifies a single byte (eight bits) of storage. In some embodiments, the local memory 1440 may include memory banks. The number of data banks in the local memory 1440 may be 16, 64, 128, 1456, 512, 1424, 2048, or other numbers. A memory bank may include a plurality of storage units. In an example, a data bank may include 8, 16, 64, or a different number of storage units. A memory bank or a storage unit in a memorybank may have a memory address. In an example, a storage unit may store a single byte, and data larger than a single byte may be stored in storage units with consecutive memory addresses, i.e., adjacent storage units. For instance, a storage unit can store an integer number in the INT8 format, versus two storage units may be needed to store a number in the FP16 or BF16 format, which has 16 bits. In some embodiments, 16 bits can be transferred from the local memory 1440 in a single read cycle. In other embodiments, 16 bits can be transferred from the local memory 1440 in multiple read cycles, such as two cycles.

[0122] The DSP 1450 performs computations in DNN layers, including computations in group quantization-based neural network operations. In some embodiments, the DSP 1450 can perform generic computations such as addition, subtraction, multiplication, division, logical, bitwise operations, and other nonlinear computations (in terms of table look-up or polynomial approximation). The DSP 1450 may be a very long instruction word (VLIW) processor. In some embodiments, the DSP 1450 may have an architecture optimized for the operational needs of digital signal processing. In some embodiments, the DSP 1450 may perform some computations in a neural network operation, while other computations in the neural network operation may be performed by the DPU 1455. The DSP 1450 may support non-traditional operations or non-MatMul or non-convolution-based operations within DNNs.

[0123] In some embodiments, the DSP 1450 may operate in accordance with a clock signal. For instance, the timing when the DSP 1450 can execute instructions may be synchronized with the clock signal. In some embodiments, the DSP 1450 may be pipelined along with the DMA engine 1420 or the DPU 1455, thereby enabling parallel computations to improve overall performance. The DSP 1450 may be implemented on a microprocessor chip, which may be separate from a chip implementing the DPU 1455. In some embodiments, the DSP 1450 may be a Streaming Hybrid Architecture Vector Engine (SHAVE) processor. Even though FIG. 14 shows a single DSP, the compute core 1430 may include multiple DSPs. The DSPs may be arranged in an array.

[0124] The IDU 1460 loads data from the local memory 1440 to the processing engine 1470 or to the post-processing engine 1480. The IDU 1460 may read tensors from the local memory 1440. The tensors may include activation tensors, weights tensor, and so on. The IDU 1460 may perform group-wise loading of activations or weights. In some embodiments,the IDU 1460 may read data from the local memory 1440 and write the data into storage units in the processing engine 1470. For instance, the IDU 1460 may load activations into activation register files in the processing engine 1470 and load weights into weight register files in the processing engine 1470. The IDU 1460 may have an activation reader for loading activations and a weight reader for loading weights. In some embodiments, the IDU 1460 may read configuration parameters from the local memory 1440 and load the configuration parameters into configuration registers or other configurable components (e.g., LUTs) of the processing engine 1470 or post-processing engine 1480.

[0125] The processing engine 1470 performs operations in DNNs. The processing engine 1470 may include one or more processing cells. In some embodiments, the processing cells may be arranged in one or more rows and one or more columns in the processing engine 1470. Each processing cell may include processing elements (PEs) that may be arranged in an array that includes rows and columns. All the PEs in the processing engine 1470 may constitute a bigger array that includes more rows and columns. An example PE may be or may include one or more multiply-accumulate (MAC) units that can perform MAC operations. In some embodiments (e.g., embodiments where the compute core 1430 executes a convolutional layer), a computation in an MAC unit may be an MAC operation on an activation operand and a weight operand. The activation operand may be an activation tensor that may include one or more activations in the input tensor of the convolution. Different activations may be in different input channels. The weight operand may be a weight tensor that may include one or more weights in the filter of the convolution. The values of the weights are determined through training the DNN or compressing the neural network operation after training. The weights in the weight operand may be in different input channels. In some embodiments, the activation operand or weight operand is a vector along the input channel dimension.

[0126] In some embodiments, an MAC unit includes one or more multipliers for performing multiplications. An MAC unit may also include one or more accumulators ("adders") for performing accumulations. An MAC unit may also include one or more shifters to facilitate mixed-precision computations. A column of MAC units is referred to as an MAC column. An MAC column may be associated with one or more MAC lanes. A MAC lane is a path for loading data e.g., by the IDU 1460, into an MAC column. A MAC lane may be also referred to as a data transmission lane or data loading lane. An MAC column may have multiple MAClanes. The loading bandwidth of the MAC column is an aggregation of the loading bandwidths of all the MAC lanes associated with the MAC column. With a certain number of MAC lanes, data can be fed into the same number of independent MAC units simultaneously. In some embodiments where an MAC column has four MAC lanes for feeding activations or weights into the MAC column and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes can have a total loading bandwidth of 64 bytes.

[0127] In some embodiments, a processing cell may have a sparsity logic unit for accelerating computations in DNNs based on data sparsity. For instance, the sparsity logic unit may obtain or generate a sparsity bitmap and use the sparsity bitmap to identify nonzero values in the activation register files or weight registers files and send nonzero values to the PEs for performing computation, while zero values in the activation register files or weight registers files are skipped.

[0128] The post-processing engine 1480 processes outputs of the processing engine 1470. The post-processing engine 1480 may include one or more post-processing elements (PPEs). In some embodiments, the PPEs in the post-processing engine 1480 may be arranged in an array that has rows and columns. In some embodiments, the post-processing engine 1480 computes activation functions. The post-processing engine 1480 may receive outputs of the processing engine 1470 as inputs to the activation functions. In addition or alternative to activation functions, the post-processing engine 1480 may perform other types of post processing on outputs of the processing engine 1470. For instance, the post-processing engine 1480 may apply a bias on an output of the processing engine 1470. In some embodiments, the post-processing engine 1480 may be bypassed for certain neural network operations.

[0129] The ODU 1490 drains data from the processing engine 1470 or from the postprocessing engine 1480, e.g., from register files in the processing engine 1470 or from the post-processing engine 1480. The drain module may write the data to the local memory 1440. The drained data may be tensors, such as output tensors of neural network operations. In some embodiments, the ODU 1490 may drain data on a cell level. For each processing cell, the ODU 1490 may drain outputs of PEs in the processing cell based on a row index or column index of each PE. For instance, the ODU 1490 may use a sequence of cycles to drain data from a processing cell. The ODU 1490 may drain the output of some ofthe PEs in each cycle. The sequence of the cycles may be configured based on a configuration parameter indicating the operation mode of the IDU 1460.

[0130] In some embodiments, the ODU 1490 includes sparsity encoding logic that can convert outputs of the processing engine 1470 from a dense format to a sparse format. For instance, the ODU 1490 may be implemented with one or more sparsity encoders. A sparsity encoder converts dense data to compressed data based on sparsity in the dense data. For instance, the sparsity encoder may remove zeros from data computed by the processing engine 1470. The sparsity encoder may also generate sparsity maps that represent sparsity in the dense data.

[0131] In some embodiments, the data drained from the processing engine 1470 may be output data elements of a DNN layer. The sparsity encoder may generate a compressed version of the output tensor. The sparsity encoder may identify every zero activation in the output tensor and remove these activations from the output tensor to generate a compressed activation tensor (aka "sparse activation tensor"). The sparsity encoder may also generate one or more sparsity maps for the output tensor. A sparsity map may indicate sparsity in at least part of the output tensor. The sparsity map may include sparsity elements (e.g., bits), each of which corresponds to a different activation in the vector and indicates whether the corresponding activation is zeroed or not.

[0132] The ODU 1490 may write the compressed activation tensor and the one or more sparsity maps into the local memory 1440. The sparse activation tensor and the one or more sparsity maps may be further loaded to the memory 1410, e.g., through the DMA engine 1420. Additionally or alternatively, the sparse activation tensor and the one or more sparsity maps may be loaded by the IDU 1460 to the processing engine 1470 for further computation, e.g., for performing a deep learning operation in the next layer.

[0133] FIG. 15 illustrates an example sparse cell 1500, in accordance with various embodiments. The sparse cell 1500 may be a processing cell in a processing engine, e.g., the processing engine 1470 in FIG. 14. The sparse cell 1500 includes 16 MAC units 1510 (individually referred to as "MAC unit 1510"), which constitutes a MAC array having four rows and four columns. The MAC array has a spatial shape of 4x4, meaning the height of the MAC array is four and the width of the MAC array is also 15. The sparse cell 1500 also includes 16 weight register files 1520 (individually referred to as "weight register file 1520"), 16 activation register files 1530 (individually referred to as "activation register file 1530"),four row buffers 1540 (individually referred to as "row buffer 1540"), and acceleration modules 1560 (individually referred to as "acceleration module 1560"). In other embodiments, the sparse cell 1500 may include fewer, more, or different components. For example, the sparse cell 1500 may include a different number of MAC units 1510, weight register files 1520, activation register files 1530, row buffers 1540, or acceleration modules 1560. As another example, the sparse cell 1500 may include column buffers in lieu of or in addition to the row buffers 1540. Also, the shape (e.g., the height or width) of the MAC array may be different.

[0134] The MAC units 1510 are configured to perform MAC operations. Each MAC unit 1510 may include one or more multipliers and one or more adders. A multiplier may multiply an activation with a weight at a time to compute a product. In some embodiments (e.g., embodiments where the MAC unit 1510 includes multiple multipliers), the multipliers may operate simultaneously to process multiple activation-weight pairs and compute multiple products in one cycle. An adder may accumulate products computed by the multipliers. Even though not shown in FIG. 15, the sparse cell may include an adder tree including a plurality of adder tiers. The first tier may receive outputs of a plurality of MAC units 1510. The number of adders in the first tier may be half of the number of the MAC units 1510, and each adder may accumulate the outputs of two MAC units 1510. The second tier may receive outputs of adders in the first tier. The number of adders in the second tier may be half of the number of adders in the first tier, and each adder in the second tier may accumulate the outputs of two adders in the first tier. The adder tree may include one or more other tiers. The last tier may include a single adder that accumulates outputs of adders in the second last tier to compute a partial sum of the sparse cell 1500.

[0135] The weight register files 1520 store weights to be processed in MAC operations. In the embodiments of FIG. 15, four weight register files 1520 are grouped into a storage set that stores data to be used by a column of MAC units 1510. There are four storage sets corresponding to the four columns of MAC units 1510. In some embodiments, a weight register file 1520 may correspond to a MAC unit 1510 and store data to be processed by the MAC unit. In some embodiments, all the 16 weight register files 1520 constitute a weight storage unit.

[0136] The activation register files 1530 stores activations to be processed in MAC operations. In the embodiments of FIG. 15, four activation register files 1530 are groupedinto a storage set that stores data to be used by a row of MAC units 1510. There are four storage sets corresponding to the four rows of MAC units 1510. In some embodiments, an activation register file 1530 may correspond to a MAC unit 1510 and store data to be processed by the MAC unit. In some embodiments, all the 16 activation register files 1530 constitute an activation storage unit. The row buffers 1540 store outputs of the MAC units 1510. Each row buffer 1540 may drain outputs of a single row of MAC units 1510.

[0137] The acceleration module 1560 facilitates acceleration of computations in the sparse cell 1500 based on mixed formats of weights. In the embodiments of FIG. 15, each acceleration module 1560 may control acceleration of computations in a different MAC unit 1510. The number of acceleration modules 1560 in the sparse cell 1500 is the same as the number of MAC units 1510 in the sparse cell 1500. In other embodiments, an acceleration module 1560 may control acceleration in multiple MAC units 1510. As shown FIG. 15, each acceleration module 1560 includes a storage unit 1565 and a control logic 1567. The storage unit 1565 stores mixed-format maps. The control logic 1567 may control distributions of activations and weights stored from the weight register files 1520 and the activation register files 1530 to the MAC units 1510 based on mixed-format maps. In some embodiments, the control logic 1567 may distribute a weight operand and a corresponding activation operation to a MAC unit 1510 for an MAC operation. The weight operand may be a subblock (e.g., a column) of a weight block. All the weights in the weight operand may be in the same output channel and have the same spatial position, but the weights may be in different input channels from each other.

[0138] In some embodiments, a weight operand may include one or more uncompressed weight and one or more compressed weights. The control logic 1567 may distribute compressed weights to MAC units 1510 in a different manner from which the control logic 1567 distributes uncompressed weights. In some embodiments (e.g., embodiments in which the compressed weights are zeros), the control logic 1567 may select nonzero weights stored in the weight register files 1520 based on the mixed-format map and distribute these nonzero weights to the MAC unit 1510 for computation. The control logic 1567 may also distribute activations, which correspond to the nonzero weights, to the MAC unit 1510 from in the activation register files 1530. The control logic 1567 may ignore zero weights and activations corresponding the zero weights so that these weights and activations can be skipped from computation.

[0139] In other embodiments (e.g., embodiments in which the compressed weights have a lower precision than the uncompressed weights), the control logic 1567 may distribute both compressed weights and uncompressed weights to the MAC unit 1510 but in different manners. For example, the control logic 1567 may distribute one compressed weight to the MAC unit 1510 for one computation cycle of the MAC unit 1510 but distribute one uncompressed weight to the MAC unit 1510 for multiple computation cycles of the MAC unit 1510. The MAC unit 1510 may have a multiplier that can compute a product of a compressed weight with its corresponding activation in one computation cycle. The multiplier may compute multiple products for an uncompressed weight. Each of these products may be a result of multiplying a portion of the uncompressed weight with the corresponding activation in one computation cycle. One or more of these products may be shifted and then accumulated with one or more other products to compute the product of the uncompressed weight and the activation. As another example, the control logic 1567 may distribute multiple compressed weights to the MAC unit 1510 for one computation cycle of the MAC unit 1510 but distribute one uncompressed weight to the MAC unit 1510 for one computation cycle of the MAC unit 1510. The MAC unit 1510 in this example may have multiple multipliers that can compute multiple products for a uncompressed weight in one operating cycle, in which each multiplier may multiply a portion of the uncompressed weight with the corresponding activation. Each multiplier may multiply a compressed weight with the corresponding activation in one compute cycle so that multiple multipliers can handle multiple uncompressed weights in one computation cycle.

[0140] As shown in FIG. 15, the sparse cell 1500 is associated with multiplexers (MUXs) 1503, 1504, 1505, and 1506. In other embodiments, the sparse cell 1500 may be associated with a different number of MUXs or other devices. The MUX 1503 facilitates loading weights, e.g., from the local memory 1440, into the weight register files 1520. The MUX 1504 facilitates loading activations, e.g., from the local memory 1440, into the activation register files 1530. The MUX 1505 facilitates loading mixed-format maps into the storage unit 1565. The MUX 1506 may be a drain MUX that can facilitate draining outputs of the MAC units 1510, e.g., to the local memory 1440.

[0141] FIG. 16 illustrates a sparse cell array 1570, in accordance with various embodiments. The sparse cell array 1570 may be an example of the processing engine 1470 in FIG. 14. In FIG. 16, the sparse cell array 1570 includes sparse cells 1580 (individually referred to as"sparse cell 1580") arranged in four columns and four rows, an activation memory 1590, and a weight memory 1595. In other embodiments, the sparse cell array 1570 may include fewer, more, or different components. For instance, the sparse cell array 1570 may include a different number of columns, rows, or sparse cells 1580.

[0142] Each sparse cell 1580 may perform accelerated MAC operations. MAC operations in the sparse cells 1580 may be accelerated based on mixed formats of weights. An embodiment of a sparse cell 1580 may be the sparse cell 1500 in FIG. 15. The activation memory 1590 stores activations, such as activations in input tensors of neural network operations. Activations may be loaded from the activation memory 1590 to sparse cells 1580, e.g., to activation register files. The weight memory 1595 stores weights, such as weights in filters of neural network operations. Weights may be loaded from the weight memory 1595 to sparse cells 1580, e.g., to weight register files. The activation memory 1590 or weight memory 1595 may be a buffer.

[0143] FIG. 17 illustrates an example PE 1700, in accordance with various embodiments. The PE 1700 may be a unit component of a processing cell, e.g., a processing cell in the processing engine 1470 in FIG. 14. In the embodiments of FIG. 17, the PE 1700 includes an MAC unit 1705, an activation register file 1710, a weight register file 1720, an output register file 1750, and a sparsity accelerator 1760. The MAC unit 1705 includes a multiplier 1730 and an adder 1740. In other embodiments, the PE 1700 may include fewer, more, or different components.

[0144] The activation register file 1710 stores an activation operand, which may be a context. The activation register file 1710 may be an example of the activation register files 1530 in FIG. 15. The weight register file 1720 stores a weight operand. The weight register file 1720 may be an example of the weight register files 1520 in FIG. 15. The activation operand and weight operand may be loaded from a memory (e.g., the local memory 1440) into the activation register file 1710 and the weight register file 1720, respectively. The sparsity accelerator 1760 receives a sparsity bitmap 1715 that corresponds to the sparse tensor in the weight register file 1720. The sparsity bitmap 1715 may be a combined sparsity bitmap when the MAC unit 1705 operates in a combined compute mode. The sparsity bitmap 1715 may be an activation sparsity bitmap when the MAC unit 1705 operates in an activation compute mode. The sparsity bitmap 1715 may be a weight sparsity bitmap when the MAC unit 1705 operates in a weight compute mode. The sparsity bitmap 1715 may havethe same size (e.g., the same number of elements) as or a larger size than the activation operand or the weight operand.

[0145] Using the sparsity bitmap 1715, the sparsity accelerator 1760 selects four activations from the activation register file 1710 and selects four weights from the weight register file 1720. The sparsity accelerator 1760 transmits the selected activations and weights to the multiplier 1730. These selected data elements correspond to the nonzero elements of the sparsity bitmap 1715. The four selected activations and the four selected weights may constitute four activation-weight pairs. The multiplier 1730 may compute a product based on each activation-weight pair and therefore, compute four products in total. The four products may be provided to the adder 1740. Even though FIG. 17 shows a single multiplier 1730, the MAC unit 1705 may include multiple multipliers that can perform multiple multiplication operations at the same time.

[0146] The adder 1740 accumulates the four products and computes a unit-level internal partial sum. The four unselected elements of the dense tensor are not processed to save power and time, which would not impact the value of the unit-level internal partial sum. For instance, when the dense tensor is a dense activation tensor, the weights corresponding to the unselected activations are zeros so the products of the unselected activations and the weights would all be zero and have no contribution to the unit-level internal partial sum or other partial sums computed by the sparse cell. Similarly, when the dense tensor is a dense weight tensor, the activations corresponding to the unselected weights are zeros so the products of the unselected weights and the activations would all be zero and have no contribution to the unit-level internal partial sum or other partial sums computed by the sparse cell. In other embodiments, the MAC unit 1705 may operate in a dense mode in which the sparsity bitmap 1715 is not used and the sparsity accelerator 1760 is inactive. The MAC unit 1705 may process all the activations in the activation operand and all the weights in the weight operand.

[0147] The unit-level internal partial sum may be stored in the output register file 1750. In some embodiments, the unit-level internal partial sum may be used multiple times. For instance, the activation operand may represent N data blocks in the input tensor of the convolution, where N is an integer greater than 1. Instead of processing all the N data blocks to compute N unit-level internal partial sums, the unit-level internal partial sum is computed once and used N times in the convolutional layers as N unit-level internal partial sums.

[0148] In some embodiments, the PE 1700 receives one or more PE-level internal partial sums from one or more other PEs. The adder 1740 or an accumulator (not shown in FIG. 17) can accumulate the one or more PE-level internal partial sums with the PE-level internal partial sum of the PE 1700 and store the result of the accumulation (i.e., a multi-PE internal partial sum) in the output register file 1750. The one or more other PEs may be in the same column as the PE 1700 in a sparse cell. The multi-unit internal partial sum may be a column-level internal partial sum. In some embodiments, the PE-level internal partial sum of the PE 1700 or the multi-unit internal partial sum may be sent to one or more other PEs for further accumulation.

[0149] FIG. 18 illustrates an example transformer model 1800, in accordance with various embodiments. The transformer model 1800 may be an example of the DNNs described above. The transformer model 1800 may be trained or deployed using a multi-chiplet system including one or more BiFrost packages, such as the packages described above. The transformer model 1800 may transform input sequences into output sequences. In some embodiments, the transformer model 1800 is a DNN that can learn context and meaning by tracking relationships in sequential data, such as sequential words in a sentence, sequential audio signals, sequential images, and so on. In an example, the transformer model 1800 may be at least part of an LLM. As shown in FIG. 18, the transformer model 1800 includes an encoder block 1810, a decoder block 1820, and a head block 1830. In other embodiment, different or additional components may be included in the transformer model 1800.Further, functionality attributed to a component of the transformer model 1800 may be accomplished by a different component included in the transformer model 1800 or a different model or module.

[0150] The encoder block 1810 receives input sequences and generates matrix representations of the input sequences. In the embodiments of FIG. 18, the encoder block 1810 receives an input 1801 and generates an encoder output 1802. The input 1801 may be an input prompt. In some embodiments, the input 1801 may include one or more input tokens, such as words, phrases, sentences, images, audio signals, other types of input tokens, or some combination thereof. In an example, the input 1801 may include a prompt received from a user of the transformer model 1800. The prompt may include a question or request made by the user. A word in the prompt may be an input token. The encoder output 1802 may include one or more vectors that are contextualized representations of the input1801. Each vector in the encoder output 1802 may represent a token in the input 1801 with contextual understanding.

[0151] The encoder block 1810 includes an embedding layer 1813, a positional encoding layer 1815, and a plurality of layers 1840 (individually referred to as "layer 1840"). In other embodiments, the encoder block 1810 may have different, fewer, or more components. Also, the arrangement of the components in the encoder block 1810 may be different from the arrangement shown in FIG. 18. For the purpose of illustration, the encoder block 1810 has N layers in FIG. 18, where N is an integer. Each layer 1840 may include one or more neural network operations. The layers 1840 may transform a sequence of embeddings into a representation that encapsulates the learned information from the input 1801. Different layers 1840 may have different internal parameters, e.g., different weights, bias, or other types of internal parameters. In some embodiments, the layers 1840 have identical components. The components in a layer 1840 may be layers and may also be referred to as sub-layers of the layer 1840. As shown in FIG. 18, a layer 1840 includes four sub-layers: a multi-head attention (MHA) layer 1841, an add & norm layer 1842, a feed forward layer 1843, and another add & norm layer 1844.

[0152] The decoder block 1820 iteratively generates outputs 1803 using encoded representations generated by the encoder block 1810. The decoder block 1820 includes an embedding layer 1823, a positional encoding layer 1825, and a plurality of layers 1850 (individually referred to as "layer 1850"). For the purpose of illustration, the decoder block 1820 has N layers in FIG. 18, where N is an integer. In the embodiments of FIG. 18, the number of layers 1850 in the decoder block 1820 is the same as the number of layers 1840 in the encoder block 1810. In other embodiments, the number of layers 1850 in the decoder block 1820 may be different from the number of layers 1840 in the encoder block 1810. Each layer 1850 may include one or more neural network operations. Different layers 1850 may have different internal parameters. In some embodiments, the layers 1850 may have identical components. The components in a layer 1850 may be layers and may also be referred to as sub-layers of the layer 1850. As shown in FIG. 18, a layer 1850 includes six sub-layers: an MHA layer 1851, an add & norm layer 1852, another MHA layer 1853, another add & norm layer 1854, a feed forward layer 1855, and another add & norm layer 1856.

[0153] In some embodiments, a sequence of inference stages is performed in the decoder block 1820 using encoder outputs, e.g., the encoder output 1802. A matrix may be predicted through each inference stage. The outputs 1803 may include a plurality of matrices. Each matrix may be further processed in the head block 1830 to predict a token. The plurality of matrices may be used to predict a sequence of tokens. For the first inference stage, the decoder block 1820 may receive one or more start tokens as input tokens and compute a first matrix from the input tokens and the output of the encoder block 1810. The first matrix may be used by the head block 1830 to predict a first token. The predicted token may be used as a new input token, in addition to the start token(s), in the second inference stage. Similarly, a second token may be predicted through the second inference stage and may be used in the third inference stage. This iteration may continue till all the inference stages are complete.

[0154] The head block 1830 receives the output of the decoder block 1820 and processes it in a linear layer 1833 and a Softmax layer 1835. A linear operation may be performed on the output of the decoder block 1820 in the linear layer 1833. The linear operation may include a multiplication of the output of the decoder block 1820 with a weight matrix. The output of the linear layer 1833 may be a vector. In some embodiments, the head block 1830 may function as a classifier. The number of data elements in the vector computed in the linear layer 1833 may depend on the number of classes involved. In an example where there are M classes, where M is an integer, the vector computed in the linear layer 1833 may have M data elements representing the prediction for the M classes, respectively.

[0155] The output of the linear layer 1833 may be input into the Softmax layer 1835. A Softmax function may be applied on the output of the linear layer 1833 to compute probability scores. A probability score may have a value in the range from 0 to 18. In some embodiments, a probability value is computed for each data element in the vector computed in the linear layer 1833. The highest one of the probability scores may be the key. The corresponding index of the key may point to the token that the transformer model 1800 predicts as the next in the sequence. The final output of the transformer model 1800 may be the sequence of predicted tokens. In some embodiments, the head block 1830 may be a language modeling head.

[0156] An embedding layer (e.g., the embedding layer 1813 or the embedding layer 1823) converts an input of the embedding layer (e.g., the input 1801 or the outputs 1803) into oneor more embeddings. An embedding may be a vector, which is also referred to as an embedding vector or a vector embedding. The vector embedding may include a sequence of data elements. In some embodiments, the embedding layer 1813 may generate a plurality of embeddings, each of which may be converted from a different input token in the input 1801. The embeddings may capture the semantic meaning of the tokens in the input 1801. The embeddings may be numerical representations that capture the relationships or meanings of words, phrases, or other data types. In an example where the input 1801 is a prompt including a sequence of words, the embedding layer 1813 may generate an embedding from each word in the input 1801. The embedding layer 1823 in the decoder block 1820 may generate a plurality of embeddings from tokens received by the decoder block 1820 in a similar manner as the embedding layer 1813.

[0157] A positional encoding layer (e.g., the positional encoding layer 1815 or the positional encoding layer 1825) performs positional encoding on embeddings generated in the corresponding embedding layer. In some embodiments, the positional encoding layer may apply one or more positional encoding vectors (e.g., a positional encoding vector 1804 or positional encoding vector 1805) on vector embeddings from the corresponding embedding layer to generate new vector embeddings that represent the embeddings with positional context. The positional encoding vector may encode information about the position of the embedding in a sequence of embeddings. In some embodiments, the positional encoding layer performs an addition operation on a positional encoding vector and a vector embedding. The addition operation may be elementwise addition. The positional encoding layer may output an embedding matrix that includes the vector embeddings computed in the positional encoding layer.

[0158] An MHA layer (e.g., the MHA layer 1841, the MHA layer 1851, or the MHA layer 1853) may implement a multi-head attention mechanism, which may be a multi-head self-attention mechanism or a multi-head cross-attention mechanism. In some embodiments, the MHA layer 1841 or the MHA layer 1851 may implement a self-attention mechanism. For self-attention, the queries, keys, and values may come from the same place. For instance, for the MHA layer 1841, the queries, keys, and values may all come from the positional encoding layer 1815. For the MHA layer 1851, the queries, keys, and values may all come from the positional encoding layer 1825. The self-attention mechanism may enable the transformer model 1800 to relate each token with other tokens. The MHA layer maycompute attention scores from embeddings generated in the corresponding positional encoding layer. In some embodiments, the MHA layer may receive one or more queries, one or more keys, and one or more values. In some embodiments, the MHA layer has a number of heads that receive different linearly projected versions of the queries, keys, and values and produce outputs in parallel that are then used to generate the final result.

[0159] In some embodiments, the queries, keys, and values input into the MHA layer 1841 may be computed from vector embeddings generated by the positional encoding layer 1815. The queries, keys, and values input into the MHA layer 1851 may be computed from vector embeddings generated by the positional encoding layer 1825. A query, key, or value may be a vector the represents a token in a sequence. In some embodiments, a query matrix Q G IRWx / lmay be computed by multiply an embedding matrix X G IRWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WqG IRdx / l, where d is the dimension of a vector embedding, N is the number of vector embeddings in the embedding matrix, and h is the number of attention heads. Each row in the query matrix may be a query. A key matrix K G IRWx / lmay be computed by multiple an embedding matrix X G IRWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WkE IRdx / l. Each row in the key matrix may be a key. A value matrix V E IRWx / lmay be computed by multiple an embedding matrix X E IRWxd(e.g., an embedding matrix computed in a positional encoding layer) with a weight matrix WvE IRdx / l. Each row in the value matrix may be a value.

[0160] In some embodiments, the MHA layer 1851 may implement masked multi-head selfattention. The MHA layer 1851 may prevent positions from attending to subsequent positions. For instance, each token in the sequence may not be influenced by future tokens. This masking can ensure that the predictions of a particular position can depend on known outputs at positions before it and not depend on unknown outputs at positions after it.

[0161] In some embodiments, the MHA layer 1853 may implement a cross-attention mechanism, such as encoder-decoder cross-attention. The MHA layer 1853 may use outputs from the previous layer (i.e., the add & norm layer 1852) as queries and use outputs from the encoder block 1810 as keys and values. The cross-attention can align the encoder's input with the decoder's, empowering the decoder block 1820 to identify and emphasize the most relevant parts of the encoder's input.

[0162] In some embodiments, an MHA layer includes linear layers, a MatMul layer, a scale layer, a Softmax layer, another MatMul layer, a concatenation layer, and another linear layer. These layers may be arranged in a sequence. The MHA layer may receive three input matrices: a query matrix, a key matrix, and a value matrix, which are inputs of three linear layers, respectively. The linear layers may include matrix multiplication (MatMul) operations. For instance, a first linear layer may perform a multiplication of the query matrix with a weight matrix to compute a first parameter matrix. The first parameter matrix may be denoted as QW, where Q is the query matrix and W G ^dmodei^q js t^e wejghtmatrix. A second linear layer may perform a multiplication of the key matrix with a weight matrix to compute a second parameter matrix. The second parameter matrix may be denoted as KWtK, where K is the key matrix and W G ^dmodei> dk jsthe weight matrix. A third linear layer may perform a multiplication of the value matrix with a weight matrix to compute a third parameter matrix. The third parameter matrix may be denoted as VW-', where V is the value matrix and WtvG ]dmodeixdk js te weight matrix, i may indicate the index of the head. dqis the dimension of a query vector. dkis the dimension of a key vector. dvis the dimension of a value vector. In some embodiments, dq= dk= dv= dmodei / h. In some embodiments, the linear layers may be in a linear block of the MHA layer. In some embodiments, the MHA layer may include multiple linear blocks. For instance, the MHA layer includes h linear blocks. The linear blocks may have the same layers as each other. Each linear block may compute three parameter matrices from the query matrix, key matrix, and value matrix, respectively.

[0163] The MatMul layer, scale layer, mask layer, Softmax layer, and MatMul layer may be in an attention block of the MHA layer. The attention block may implement a scaled dotproduct attention mechanism. In some embodiments, the MHA layer includes a plurality of attention blocks that includes the attention block. For the purpose of illustration, the MHA layer includes h attention blocks. The attention blocks may have the same layers as each other. A linear block and an attention block may constitute a head of the MHA layer. When the MHA layer has h linear blocks and h attention blocks, the MHA layer has h heads. A head may be denoted as headt= Attention QW, KW, VW-').

[0164] A matrix multiplication operation may be performed on parameter matrices in the MatMul layer, which computes a score matrix. In some embodiments, the score matrix mayestablish the degree of emphasis each token should place on other tokens. The score matrix may include a plurality of scores. Each token may be assigned a score in relation to other tokens within the same time step. A higher score may indicate a higher focus or emphasis. The score matrix may be scaled in the scale layer. In some embodiments, the score matrix is scaled down in the scale layer by dividing the scores in the score matrix by the square root of the dimension of the query vector and the key vector, which may be denoted asThe output of the scale layer may be a scaled matrix, which includes adjusted scores. The mask layer may be optional in some embodiments. The mask layer may add an attention mask (which may be an input to the attention block) to the output of the scale layer to mask out some elements in the output of the scale layer. The positions of the masked-out elements may be defined by the attention mask. A Softmax function may be applied on the scaled matrix in the Softmax layer to compute an attention weight matrix. The attention weight matrix includes attention weights. The attention weights may be probability values ranging from 0 to 1. The Softmax function may emphasize high scores while diminishing low scores, which can enhance the model's ability to determine which tokens should get more attention.

[0165] In the MatMul layer, a matrix multiplication operation is performed on the attention weight matrix computed in the Softmax layer and the parameter matrix computed from value matrix in the corresponding linear layer. The result of the matrix multiplication operation is a single-head output matrix, which is an output of the attention block.

[0166] When the MHA layer has h attention blocks, there may be h single-head output matrices. The single-head output matrices are concatenated in the concatenation layer to form a concatenated matrix. A linear operation (also referred to as "linear transformation") is performed on the concatenated matrix using a weight matrix in the linear layer. In some embodiments, the MHA may be denoted as MultiHead Q, K, 7) =Concat (head-, head2,..., headh~)W°, where Concat denotes concatenation, and W° G ^hdvxdmodeijs t e wejght matrix in the corresponding linear layer.

[0167] An add & norm layer in the transformer model 1800, such as the add & norm layer 1842, 1844, 1852, 1854, and 1856, has an addition operation followed by a layer normalization operation. The addition operation may be an addition of the output of the preceding layer and the input of the preceding layer. The preceding layer is a layer that isarranged right before the add & norm layer. For example, the preceding layer of the add & norm layer 1842 is the MHA layer 1841. As another example, the preceding layer of the add & norm layer 1854 is the MHA layer 1853.

[0168] Then the layer normalization operation is applied on the result of the addition operation, which may be denoted as LayerNorm(x + sublayer(x)), where LayerNorm denotes layer normalization, x is the input of the preceding layer, and sublayer(x) denotes the output of the preceding layer. In some embodiments, the layer normalization operation may include a sequence of computations. In an example, the layer normalization operation may include a mean computation, which may be denoted as y.xy= - X z=i Axyz, where Axyzdenotes a data element in the input tensor, x may be the positional index of the data element in one of the spatial dimensions, y may be the positional index of the data element in the other one of the spatial dimensions, z may be the positional index of the data element in the channel dimension, and ixydenotes the output of the mean computation, which may be a 2D matrix. The mean computation may be channel-wise reduction operation. The layer normalization operation may convert ixyto a 3D tensor [ixyz, e.g., by replicating every data element over z output points.

[0169] The layer normalization operation may also include an elementwise subtraction, which may be denoted as Dxyz= Axyz— [ixyz. The layer normalization operation may further include a variance computation denoted as <j2%y= z=iD2xyzand a division1computation denoted as Mxy=. ^=. Mxymay be a 2D tensor. The layerx< AzXy+exZ)normalization operation may also convert Mxyto a 3D tensor Mxyz, e.g., by replicating every data element over z output points. Further, the layer normalization operation may have an element multiplication denoted as A'xyz= X1. = = Dxyzx Mxyz. The layer normalization operation may further compute jx(ff2xy+e)A"xyz= and LNxyz= A"xyzx yz. LNxyzmay be the output of the layernormalization operation.

[0170] A feed forward layer (e.g., the feed forward layer 1843 and the feed forward layer 1855) may be a position-wise fully-connected feed forward network. In an example, thefeed forward layer may include two linear layers with an activation function in between. An example of the activation function is Rectified Linear Unit (ReLU).

[0171] FIG. 19 illustrates an example CNN 1900, in accordance with various embodiments. The CNN 1900 may be an example of the DNNs described above. The CNN 1900 may be trained or deployed using a multi-chiplet system including one or more BiFrost packages, such as the packages described above. For the purpose of illustration, the CNN 1900 includes a sequence of layers comprising a plurality of convolutional layers 1910 (individually referred to as "convolutional layer 1910"), a plurality of pooling layers 1920 (individually referred to as "pooling layer 1920"), and a plurality of fully-connected layers 1930 (individually referred to as "fully-connected layer 1930"). In other embodiments, the CNN 1900 may include fewer, more, or different layers. In an execution of the CNN 1900, the layers of the CNN 1900 execute tensor computation that includes many tensor operations, such as convolutions, interpolations, pooling operations, elementwise operations (e.g., elementwise addition, elementwise multiplication, etc.), other types of tensor operations, or some combination thereof.

[0172] The convolutional layers 1910 summarize the presence of features in inputs to the CNN 1900. The convolutional layers 1910 function as feature extractors. The first layer of the CNN 1900 is a convolutional layer 1910. In an example, a convolutional layer 1910 performs a convolution on an input tensor 1940 (also referred to as IFM 1940) and a filter 1950. As shown in FIG. 19, the IFM 1940 is represented by a 7x7x3 three-dimensional (3D) matrix. The IFM 1940 includes 3 input channels, each of which is represented by a 7x7 two-dimensional (2D) matrix. The 7x72D matrix includes 7 input elements (also referred to as input points) in each row and 7 input elements in each column. The filter 1950 is represented by a 3x3x33D matrix. The filter 1950 includes 3 kernels, each of which may correspond to a different input channel of the IFM 1940. A kernel is a 2D matrix of weights, where the weights are arranged in columns and rows. A kernel can be smaller than the IFM. In the embodiments of FIG. 19, each kernel is represented by a 3x32D matrix. The 3x3 kernel includes 3 weights in each row and 3 weights in each column. Weights can be initialized and updated by backpropagation using gradient descent. The magnitudes of the weights can indicate importance of the filter 1950 in extracting features from the IFM 1940.

[0173] The convolution includes multiply-accumulate (MAC) operations with the input elements in the IFM 1940 and the weights in the filter 1950. The convolution may be astandard convolution 1963 or a depthwise convolution 1983. In the standard convolution 1963, the whole filter 1950 slides across the IFM 1940. All the input channels are combined to produce an output tensor 1960 (also referred to as output feature map (OFM) 1960). The OFM 1960 is represented by a 5x5 2D matrix. The 5x52D matrix includes 5 output elements (also referred to as output points) in each row and 5 output elements in each column. For the purpose of illustration, the standard convolution includes one filter in the embodiments of FIG. 19. In embodiments where there are multiple filters, the standard convolution may produce multiple OCs in the OFM 1960.

[0174] The multiplication applied between a kernel-sized patch of the IFM 1940 and a kernel may be a dot product. A dot product is the elementwise multiplication between the kernel-sized patch of the IFM 1940 and the corresponding kernel, which is then summed, always resulting in a single value. Because it results in a single value, the operation is often referred to as the "scalar product." Using a kernel smaller than the IFM 1940 is intentional as it allows the same kernel (set of weights) to be multiplied by the IFM 1940 multiple times at different points on the IFM 1940. Specifically, the kernel is applied systematically to each overlapping part or kernel-sized patch of the IFM 1940, left to right, top to bottom. The result from multiplying the kernel with the IFM 1940 one time is a single value. As the kernel is applied multiple times to the IFM 1940, the multiplication result is a 2D matrix of output elements. As such, the 2D output matrix (i.e., the OFM 1960) from the standard convolution 1963 is referred to as an OFM.

[0175] In the depthwise convolution 1983, the input channels are not combined. Rather, MAC operations are performed on an individual input channel and an individual kernel and produce an OC. As shown in FIG. 19, the depthwise convolution 1983 produces a depthwise output tensor 1980. The depthwise output tensor 1980 is represented by a 5x5x3 3D matrix. The depthwise output tensor 1980 includes 3 OCs, each of which is represented by a 5x52D matrix. The 5x5 2D matrix includes 5 output elements in each row and 5 output elements in each column. Each OC is a result of MAC operations of an input channel of the IFM 1940 and a kernel of the filter 1950. For instance, the first OC (patterned with dots) is a result of MAC operations of the first input channel (patterned with dots) and the first kernel (patterned with dots), the second OC (patterned with horizontal strips) is a result of MAC operations of the second input channel (patterned with horizontal strips) and the second kernel (patterned with horizontal strips), and the third OC (patterned with diagonal stripes) is aresult of MAC operations of the third input channel (patterned with diagonal stripes) and the third kernel (patterned with diagonal stripes). In such a depthwise convolution, the number of input channels equals the number of OCs, and each OC corresponds to a different input channel. The input channels and output channels are referred to collectively as depthwise channels. After the depthwise convolution, a pointwise convolution 1993 is then performed on the depthwise output tensor 1980 and a 19x1x3 tensor 1990 to produce the OFM 1960.

[0176] The OFM 1960 is then passed to the next layer in the sequence. In some embodiments, the OFM 1960 is passed through an activation function. An example activation function is rectified linear unit (ReLU). ReLU is a calculation that returns the value provided as input directly, or the value zero if the input is zero or less. The convolutional layer 1910 may receive several images as input and calculate the convolution of each of them with each of the kernels. This process can be repeated several times. For instance, the OFM 1960 is passed to the subsequent convolutional layer 1910 (i.e., the convolutional layer 1910 following the convolutional layer 1910 generating the OFM 1960 in the sequence). The subsequent convolutional layers 1910 perform a convolution on the OFM 1960 with new kernels and generate a new feature map. The new feature map may also be normalized and resized. The new feature map can be kernelled again by a further subsequent convolutional layer 1910, and so on.

[0177] In some embodiments, a convolutional layer 1910 has four hyperparameters: the number of kernels, the size F kernels (e.g., a kernel is of dimensions FxFxD pixels), the S step with which the window corresponding to the kernel is dragged on the image (e.g., a step of one means moving the window one pixel at a time), and the zero-padding P (e.g., adding a black contour of P pixels thickness to the input image of the convolutional layer 1910). The convolutional layers 1910 may perform various types of convolutions, such as 2-dimensional convolution, dilated or atrous convolution, spatial separable convolution, depthwise separable convolution, transposed convolution, and so on. The CNN 1900 includes 196 convolutional layers 1910. In other embodiments, the CNN 1900 may include a different number of convolutional layers.

[0178] The pooling layers 1920 down-sample feature maps generated by the convolutional layers, e.g., by summarizing the presence of features in the patches of the feature maps. A pooling layer 1920 is placed between two convolution layers 1910: a precedingconvolutional layer 1910 (the convolution layer 1910 preceding the pooling layer 1920 in the sequence of layers) and a subsequent convolutional layer 1910 (the convolution layer 1910 subsequent to the pooling layer 1920 in the sequence of layers). In some embodiments, a pooling layer 1920 is added after a convolutional layer 1910, e.g., after an activation function (e.g., ReLU, etc.) has been applied to the OFM 1960.

[0179] A pooling layer 1920 receives feature maps generated by the preceding convolution layer 1910 and applies a pooling operation to the feature maps. The pooling operation reduces the size of the feature maps while preserving their important characteristics.Accordingly, the pooling operation improves the efficiency of the DNN and avoids over-learning. The pooling layers 1920 may perform the pooling operation through average pooling (calculating the average value for each patch on the feature map), max pooling (calculating the maximum value for each patch of the feature map), or a combination of both. The size of the pooling operation is smaller than the size of the feature maps. In various embodiments, the pooling operation is 2x2 pixels applied with a stride of two pixels, so that the pooling operation reduces the size of a feature map by a factor of 2, e.g., the number of pixels or values in the feature map is reduced to one quarter the size. In an example, a pooling layer 1920 applied to a feature map of 6x6 results in an output pooled feature map of 3x3. The output of the pooling layer 1920 is inputted into the subsequent convolution layer 1910 for further feature extraction. In some embodiments, the pooling layer 1920 operates upon each feature map separately to create a new set of the same number of pooled feature maps.

[0180] The fully-connected layers 1930 are the last layers of the DNN. The fully-connected layers 1930 may be convolutional or not. The fully-connected layers 1930 receive an input operand. The input operand defines the output of the convolutional layers 1910 and pooling layers 1920 and includes the values of the last feature map generated by the last pooling layer 1920 in the sequence. The fully-connected layers 1930 apply a linear combination and an activation function to the input operand and generate a vector. The vector may contain as many elements as there are classes: element i represents the probability that the image belongs to class i. Each element is therefore between 0 and 19, and the sum of all is worth one. These probabilities are calculated by the last fully-connected layer 1930 by using a logistic function (binary classification) or a SoftMax function (multi-class classification) as an activation function. In some embodiments, thefully-connected layers 1930 multiply each input element by weight, make the sum, and then apply an activation function (e.g., logistic if N=2, SoftMax if N>2). This is equivalent to multiplying the input operand by the matrix containing the weights.

[0181] FIG. 20 is a block diagram of an example computing device 2000, in accordance with various embodiments. In some embodiments, the computing device 2000 can be implemented on one or more chiplets, such as the chiplets described above. A number of components are illustrated in FIG. 20 as included in the computing device 2000, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 2000 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 2000 may not include one or more of the components illustrated in FIG. 20, but the computing device 2000 may include interface circuitry for coupling to the one or more components. For example, the computing device 2000 may not include a display device 2006, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 2006 may be coupled. In another set of examples, the computing device 2000 may not include an audio input device 2018 or an audio output device 2008 but may include audio input or output device interface circuitry to which an audio input device 2018 or audio output device 2008 may be coupled.

[0182] The computing device 2000 may include a processing device 2002 (e.g., one or more processing devices). The processing device 2002 processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. The computing device 2000 may include a memory 2004, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid state memory, and / or a hard drive. In some embodiments, the memory 2004 may include memory that shares a die with the processing device 2002. In some embodiments, the memory 2004 includes one or more non-transitory computer-readable media storing executable instructions. The instructions stored in the one or more non-transitory computer-readable media may be executed by the processing device 2002.

[0183] In some embodiments, the computing device 2000 may include a communication chip 2012 (e.g., one or more communication chips). For example, the communication chip 2012 may be configured for managing wireless communications for the transfer of data to and from the computing device 2000. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

[0184] The communication chip 2012 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 2012 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. The communication chip 2012 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 2012 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 2012 may operate in accordance with other wireless protocols in other embodiments. The computing device 2000 may include an antenna 2022 to facilitate wireless communications and / or to receive other wireless communications (such as AM or FM radio transmissions).

[0185] In some embodiments, the communication chip 2012 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, the communication chip 2012 may include multiple communication chips. For instance, a first communication chip 2012 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 2012 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication chip 2012 may be dedicated to wireless communications, and a second communication chip 2012 may be dedicated to wired communications.

[0186] The computing device 2000 may include battery / power circuitry 2014. The battery / power circuitry 2014 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 2000 to an energy source separate from the computing device 2000 (e.g., AC line power).

[0187] The computing device 2000 may include a display device 2006 (or corresponding interface circuitry, as discussed above). The display device 2006 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.

[0188] The computing device 2000 may include an audio output device 2008 (or corresponding interface circuitry, as discussed above). The audio output device 2008 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0189] The computing device 2000 may include an audio input device 2018 (or corresponding interface circuitry, as discussed above). The audio input device 2018 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0190] The computing device 2000 may include a GPS device 2016 (or corresponding interface circuitry, as discussed above). The GPS device 2016 may be in communication with a satellite-based system and may receive a location of the computing device 2000, as known in the art.

[0191] The computing device 2000 may include another output device 2010 (or corresponding interface circuitry, as discussed above). Examples of the other output device 2010 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0192] The computing device 2000 may include another input device 2020 (or corresponding interface circuitry, as discussed above). Examples of the other input device 2020 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0193] The computing device 2000 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultramobile personal computer, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 2000 may be any other electronic device that processes data.

[0194] The following paragraphs provide various examples of the embodiments disclosed herein.

[0195] Example 1 provides an IC device, including a compute core; a switch network, the switch network including a plurality of switches configured in a hierarchical tree structure; and an address module coupled to the switch network, the address module to translate a physical address for a job performed by the compute core to a system physical address in a system memory, the system memory shared by a plurality of IC devices including the IC device, the plurality of IC devices communicating with each other through the switch network.

[0196] Example 2 provides the IC device of example 1, in which the hierarchical tree structure includes two edge levels and a spine level, in which the spine level is between the two edge levels.

[0197] Example 3 provides the IC device of example 2, in which the address module is coupled to a switch in one of the two edge levels.

[0198] Example 4 provides the IC device of any one of examples 1-3, in which the address module is further coupled to a scale-up network.

[0199] Example 5 provides the IC device of any one of examples 1-4, in which the IC device and another IC device of the plurality of IC devices are in separate IC packages.

[0200] Example 6 provides the IC device of any one of examples 1-5, in which the address module includes an address translation unit to translate the physical address for the job to the system physical address; a route compute unit to determine a data transfer path based on the system physical address; and a packet generation unit to generate a packet based on the system physical address and the data transfer path.

[0201] Example 7 provides the IC device of example 6, in which the address module includes a configurable route table configured to map an identifier of the job to a memory range, in which the route compute unit is further to perform a security check on the system physical address based on the memory range.

[0202] Example 8 provides the IC device of example 6 or 7, in which the route compute unit is further to determine an alternative data transfer path for the job in response to a memory failure associated with the data transfer path.

[0203] Example 9 provides the IC device of any one of examples 1-7, further including a memory, in which the compute core and the memory communicate through the switch network.

[0204] Example 10 provides the IC device of example 9, further including an additional address module coupled to the memory and the switch network.

[0205] Example 11 provides an IC package, including a first chiplet, the first chiplet including one or more compute cores, a switch network, the switch network including a plurality of switches configured in a hierarchical tree structure, and an address module coupled to the switch network, the address module to translate a physical address for a job performed by the one or more compute cores to a system physical address; a second chiplet, in which the first chiplet and the second chiplet communicate through the switch network.

[0206] Example 12 provides the IC package of example 11, in which the second chiplet including one or more additional compute cores, an additional switch network; and an additional address module coupled to the additional switch network, the additional addressmodule to translate a physical address for a job performed by the one or more additional compute cores to a system physical address, in which the first chiplet and the second chiplet communicate further through the additional switch network.

[0207] Example 13 provides the IC package of example 11 or 12, in which the IC package communicates with another IC package through the switch network.

[0208] Example 14 provides the IC package of example 13, further including an input / output chiplet, in which the input / output chiplet is coupled to the switch network and to the another IC package.

[0209] Example 15 provides the IC package of example 14, in which the input / output chiplet is associated with a scale-up network or a scale-out network.

[0210] Example 16 provides the IC package of any one of examples 11-15, in which the address module includes an address translation unit to translate the physical address for the job to the system physical address; a route compute unit to determine a data transfer path based on the system physical address; and a packet generation unit to generate a packet based on the system physical address and the data transfer path.

[0211] Example 17 provides the IC package of example 16, in which the route compute unit is further to determine an alternative data transfer path for the job in response to a memory failure associated with the data transfer path.

[0212] Example 18 provides the IC package of any one of examples 11-17, in which the first chiplet is to execute an operating system that manages the second chiplet through a device driver.

[0213] Example 19 provides the IC package of any one of examples 11-18, in which the IC package has a plurality of multi-chiplet subsystems, in which a chiplet in a multi-chiplet system is to execute an operating system that manages one or more other chiplets in the multi-chiplet system through a device driver.

[0214] Example 20 provides the IC package of any one of examples 11-19, in which the first chiplet has a central processing unit, in which the second chiplet has a graphics processing unit or neural processing unit.

[0215] Example 21 provides a system, including a first rack of IC packages, an IC package including a plurality of chiplets, each chiplet including one or more compute cores, a switch network, the switch network including a plurality of switches configured in a hierarchical tree structure, and an address module coupled to the switch network, the address moduleto translate a physical address for a job performed by the one or more compute cores to a system physical address; and a second rack of IC packages, in which the first rack of IC packages and the second rack of IC packages communicate through the switch network.

[0216] Example 22 provides the system of example 21, in which the first rack of IC packages or the second rack of IC packages is in a scale-up network or a scale-out network.

[0217] Example 23 provides the system of example 21 or 22, in which the address module includes an address translation unit to translate the physical address for the job to the system physical address; a route compute unit to determine a data transfer path based on the system physical address; and a packet generation unit to generate a packet based on the system physical address and the data transfer path.

[0218] Example 24 provides the system of example 23, in which the route compute unit is further to determine an alternative data transfer path for the job in response to a memory failure associated with the data transfer path.

[0219] Example 25 provides the system of any one of examples 21-24, in which the switch network provides connectivity between the plurality of chiplets.

[0220] The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art can recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

1. Claims1. An integrated circuit (IC) device, comprising:3.a compute core;4.a switch network, the switch network comprising a plurality of switches configured in a hierarchical tree structure; and5.an address module coupled to the switch network, the address module to translate a physical address for a job performed by the compute core to a system physical address in a system memory, the system memory shared by a plurality of IC devices including the IC device, the plurality of IC devices communicating with each other through the switch network.

2. The IC device of claim 1, wherein the hierarchical tree structure includes two edge levels and a spine level, wherein the spine level is between the two edge levels.

3. The IC device of claim 2, wherein the address module is coupled to a switch in one of the two edge levels.

4. The IC device of any one of claims 1-3, wherein the address module is further coupled to a scale-up network.

5. The IC device of any one of claims 1-4, wherein the IC device and another IC device of the plurality of IC devices are in separate IC packages.

6. The IC device of any one of claims 1-5, wherein the address module comprises:11.an address translation unit to translate the physical address for the job to the system physical address;12.a route compute unit to determine a data transfer path based on the system physical address; and13.a packet generation unit to generate a packet based on the system physical address and the data transfer path.

7. The IC device of claim 6, wherein the address module comprises:14.a configurable route table configured to map an identifier of the job to a memory range,15.wherein the route compute unit is further to perform a security check on the system physical address based on the memory range.

8. The IC device of claim 6 or 7, wherein the route compute unit is further to determine an alternative data transfer path for the job in response to a memory failure associated with the data transfer path.

9. The IC device of any one of claims 1-7, further comprising:18.a memory,19.wherein the compute core and the memory communicate through the switch network.

10. The IC device of claim 9, further comprising:21.an additional address module coupled to the memory and the switch network.

11. An integrated circuit (IC) package, comprising:23.a first chiplet, the first chiplet comprising:24.one or more compute cores,25.a switch network, the switch network comprising a plurality of switches configured in a hierarchical tree structure, and26.an address module coupled to the switch network, the address module to translate a physical address for a job performed by the one or more compute cores to a system physical address;27.a second chiplet,28.wherein the first chiplet and the second chiplet communicate through the switch network.

12. The IC package of claim 11, wherein the second chiplet comprising:30.one or more additional compute cores, an additional switch network; and31.an additional address module coupled to the additional switch network, the additional address module to translate a physical address for a job performed by the one or more additional compute cores to a system physical address,32.wherein the first chiplet and the second chiplet communicate further through the additional switch network.

13. The IC package of claim 11 or 12, wherein the IC package communicates with another IC package through the switch network.

14. The IC package of claim 13, further comprising:35.an input / output chiplet,36.wherein the input / output chiplet is coupled to the switch network and to the another IC package.

15. The IC package of claim 14, wherein the input / output chiplet is associated with a scale-up network or a scale-out network.

16. The IC package of any one of claims 11-15, wherein the address module comprises: an address translation unit to translate the physical address for the job to the system physical address;39.a route compute unit to determine a data transfer path based on the system physical address; and40.a packet generation unit to generate a packet based on the system physical address and the data transfer path.

17. The IC package of claim 16, wherein the route compute unit is further to determine an alternative data transfer path for the job in response to a memory failure associated with the data transfer path.

18. The IC package of any one of claims 11-17, wherein the first chiplet is to execute an operating system that manages the second chiplet through a device driver.

19. The IC package of any one of claims 11-18, wherein the IC package has a plurality of multi-chiplet subsystems, wherein a chiplet in a multi-chiplet system is to execute an operating system that manages one or more other chiplets in the multi-chiplet system through a device driver.

20. The IC package of any one of claims 11-19, wherein the first chiplet has a central processing unit, wherein the second chiplet has a graphics processing unit or neural processing unit.

21. A system, comprising:45.a first rack of integrated circuit (IC) packages, an IC package comprising a plurality of chiplets, each chiplet comprising:46.one or more compute cores,47.a switch network, the switch network comprising a plurality of switches configured in a hierarchical tree structure, and48.an address module coupled to the switch network, the address module to translate a physical address for a job performed by the one or more compute cores to a system physical address; and49.a second rack of IC packages,50.wherein the first rack of IC packages and the second rack of IC packages communicate through the switch network.

22. The system of claim 21, wherein the first rack of IC packages or the second rack of IC packages is in a scale-up network or a scale-out network.

23. The system of claim 21 or 22, wherein the address module comprises:53.an address translation unit to translate the physical address for the job to the system physical address;54.a route compute unit to determine a data transfer path based on the system physical address; and a packet generation unit to generate a packet based on the system physical address and the data transfer path.

24. The system of claim 23, wherein the route compute unit is further to determine an alternative data transfer path for the job in response to a memory failure associated with the data transfer path.

25. The system of any one of claims 21-24, wherein the switch network provides connectivity between the plurality of chiplets.

Images