Integrating hardware devices into a thread execution environment of a heterogeneous computing platform

Abstract

Methods and systems, including computer-readable media, are described for integrating hardware Input / Output ("I / O") devices into a thread execution environment of a heterogeneous computing platform. The methods are implemented at the computing platform using a central processor that communicates with the hardware I / O devices. The central processor instantiates a first processing thread that is used to execute a task of a machine-learning ("ML") workload and generates a first instruction that allocates the first processing thread to a hardware I / O device of the platform. The hardware I / O device is integrated into a thread execution environment of the central processor in response to executing the first instruction at the hardware I / O device. The hardware I / O device executes the task of the ML workload by performing vector multiplications using a matrix computing unit of the hardware I / O device.

Description

Attorney Docket No.: 56113-0796WO1INTEGRATING HARDWARE DEVICES INTO A THREAD EXECUTION ENVIRONMENT OF A HETEROGENEOUS COMPUTING PLATFORMBACKGROUND

[0001] This specification relates to thread execution on a heterogeneous computing platform.

[0002] A heterogeneous computing platform includes one or more general-purpose hardware computing units and one or more special-purpose hardware computing units. Examples of general-purpose hardware computing units include central processing units (CPUs), e.g., a single or multi-core CPU. Special-purpose hardware computing units, also referred to as hardware accelerators, are computing devices having specialized hardware configured to perform specialized computations including, e.g.. machine-learning computations. Examples of special-purpose hardware computing units include graphics processing units (GPUs), field-programmable gate arrays (FGPAs), and application-specific integrated circuits (ASICs), including tensor processing units (TPUs).

[0003] For various reasons, general-purpose hardware computing units (e.g.. central processors) and special-purpose hardware computing units (e.g., hardware accelerators) have different design constraints and are optimized to perform different processor functions. For example, GPUs often include support for wide vectors and substantial register files to optimize executing compute intensive workloads, whereas CPUs are optimized for instruction throughput and low-latency scalar operations, dedicating logic to caches and out- of-order dependence control. Also, general-purpose processors and special-purpose hardware routinely employ different instruction set architectures.SUMMARY

[0004] This specification describes a heterogeneous computing platform that implements techniques for seamlessly integrating multiple hardware input / output (“I / O”) devices into a thread execution environment of a central processor of the heterogeneous platform. For example, the techniques for thread integration can be implemented using a minimum, streamlined instruction set of the central processor that is configured for recognition by each hardware I / O device to facilitate thread-level integration.

[0005] The central processor can instantiate processing threads that trigger executing artificial intelligence (“Al”) and / or machine-learning (“ML”) workloads on the I / O device that implements the disclosed thread integration techniques. Relative to prior approaches, theAttorney Docket No.: 56113-0796WO1 thread integration techniques described in this specification can provide a more fine-grained extension of the ML compute capability of the central processor. The streamlined instruction set enables an operating system scheduler (“OS scheduler”) of the computing platform to configure or set up a thread execution environment directly on the hardware I / O device.

[0006] The OS scheduler can implement its scheduling operations based on a timesharing technique, such as time multiplexing among multi-process or multi-thread execution. For example, the OS scheduler can allow a particular process or thread to run for a short time duration called a time quantum (or time slice). This time-sharing technique relies on a timer interrupt that is triggered at each time interval. Notably, timer interrupt differs conventional interrupts that are normally used for communication a CPU or processor and a I / O device, such as an ML accelerator. This timer interrupt exists and functions regardless and independent of any I / O device. In some implementations, the OS scheduler is a Linux CPU scheduler that maintains an ordered hierarchy of scheduling domains and processing threads based in part on a hardware layout of the physical cores of the central processor.

[0007] The time-sharing technique is implemented using a timer interrupt handler, which executes the OS scheduler as well as many other functions associated with thread management of a central processor. The OS scheduler can execute a scheduling algorithm to detect new tasks or workloads for running on a core of the central processor or a hardware I / O device of the computing platform. In some cases, the OS scheduler’s selecting of new tasks is algorithm dependent. For example, based on its scheduling algorithm, the OS scheduler can detect or identify a new task (or workload) in a work queue of the central processor, select the new task from the queue, and associate the new task with a thread pool of the central processor.

[0008] For a current thread being executed, the OS scheduler is configured to save information of the current thread, such as an execution state(s) of that thread and a program counter (PC) value(s) for that thread. The OS scheduler is configured to establish the thread execution environment for the newly selected task, for example, by spawning or initiating a new processing thread from the thread pool to execute the task. This can include configuring a set of registers of the central processor (or a core of the processor) and establishing a starting PC address value. The central processor can be a multi-threaded processor with multiple program counters that each point to a next instruction to execute for a given thread, including a given task or workload for that thread.

[0009] The hardware I / O device can spawn or instantiate a new processing thread within a thread execution environment of the central processor based on an integration instructionAttorney Docket No.: 56113-0796WO1 indicated or pointed to by the PC. For example, when the timer interrupt handler exits its interrupt sequence, the central processor (or hardware I / O device) will execute an instruction from the starting PC, and thus run the new processing thread. The instruction can be a thread-level integration that is uniquely configured to spawn a new thread at a hardware I / O device that is distinct from processing cores of the central processor. In some implementations, a thread is spawned from the thread pool to establish an execution environment of the central processor on the hardw are I / O device when a particular instruction pointed to by the PC address is executed on the I / O device. For example, the I / O device can be a pluggable AI / ML accelerator engine (“PAAE”) that is configured to execute a minimum set of integer, scalar, or vector instructions that allow the PAAE to configure an execution environment of the central processor on the PAAE.

[0010] The thread-level integration can be triggered when the hardware I / O device (e.g., a PAAE) executes the unique instruction locally at the I / O device. The thread-level integration instruction can be encoded among vector instructions that configure and perform vector and matrix operations for executing an AI / ML workload at an I / O device such as a PAAE. For example, a unique integer or scalar thread-level instruction can be read from the PC starting address pointer and executed locally at the I / O device to integrate the device into a main processor’s thread execution environment. The I / O device (or PAAE) can receive or generate subsequent instructions that are implemented at the I / O device to execute ML workloads by performing vector / matrix multiplications using a matrix computing unit of the I / O device.

[0011] One aspect of the subject matter described in this specification can be embodied in a method implemented using a central processor that communicates with a hardw are Input / Output (“I / O”) device. The method includes instantiating, at the central processor, a first processing thread used to execute a task of a machine-learning (“ML”) workload and generating, by the central processor, a first instruction that allocates the first processing thread to the hardw are I / O device. The method further includes integrating the hardw are I / O device into a thread execution environment of the central processor in response to executing the first instruction at the hardware I / O device and executing, at the hardware I / O device, the task of the ML w orkload by performing vector multiplications using a matrix computing unit of the hardw are I / O device.

[0012] These and other implementations can each optionally include one or more of the following features. For example, in some implementations, the central processor is a CPUAttorney Docket No.: 56113-0796WO1 and integrating the hardware I / O device into a thread execution environment of the central processor comprises: dynamically configuring the hardware I / O device to execute vector arithmetic operations within an instance of a CPU thread. The first instruction can be a thread integration instruction adapted for local execution at the hardware I / O device using a scalar unit of the hardware I / O device. In some implementations, i) the thread execution environment of the central processor comprises a first processor core configured to execute multi-threaded operations; and ii) the method further comprises, instantiating the first processing thread as a subordinate context of the multi-threaded operations that the first processor core is configured to execute.

[0013] In some implementations, the method includes allocating, to the first processor core, a set of resources that are used to execute the multi-threaded operations; and configuring the set of resources as a shared resource set that is accessible to the first processor core and hardware I / O device. Configuring the set of resources as a shared resource set of the central processor can include comprises: i) generating a second instruction that configures resource sharing at the central processor; ii) in response to executing the second instruction, performing thread-based shared memory mapping within the thread execution environment of the central processor; and iii) exposing an address space of the shared resource set to the hardware I / O device concurrent with initiating the thread-based shared memory mapping.

[0014] In some implementations, the method further includes executing, by the first processor core and the hardware I / O device, fined-grained memory access requests concurrently against the shared resource set of the central processor. The concurrent, fined- grained memory access requests are executed within the thread execution environment of the central processor. The method can further include provisioning, using the shared resource set of the central processor, fined-grained memory access allocations concurrently to the first processor core and the hardware I / O device.

[0015] In some implementations, the hardware I / O device is an ML hardware accelerator configured to accelerate computations for the ML workload. The hardware I / O device can be a hardware ML tensor processing unit C’TPU”) configured to accelerate ML workload computations that use multi-dimensional tensors. In some implementations, the hardware I / O device is a general-purpose graphics processing unit (GPU). In some other implementations, the ML w orkload is an ML inference workload or an ML training w orkload.Attorney Docket No.: 56113-0796WO1

[0016] One aspect of the subject matter described in this specification can be embodied in a method implemented using a scalar unit of a hardware I / O device that communicates with a central processing unit (‘'CPU”). The method includes identifying a request to execute a machine-learning (“ML”) inference workload; receiving, by the hardware I / O device, an instruction indicating CPU allocation of a processor thread; based on the instruction, instantiating, by a scalar unit of the hardware I / O device, the allocated processor thread as a CPU thread implemented at the hardware I / O device; and executing the ML inference workload at the hardware I / O device but as an operation under the CPU thread and that occurs within a thread execution environment of the CPU.

[0017] Another aspect of the subject matter described in this specification can be embodied in a hardware compute engine of a hardware neural network (“NN”) processor. The hardware compute engine includes a scalar unit configured to: i) receive an instruction from a central processing unit (“CPU”); ii) execute the instruction to initiate CPU thread support at the NN processor; and iii) integrate the NN processor into a thread execution environment of the CPU in response to executing the instruction at the scalar unit. The hardware compute engine further includes a vector unit configured to process vector instructions for executing a NN inference; and a matrix computing unit configured to perform vector multiplications to execute the NN inference based on the vector instructions.

[0018] Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation causes the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0019] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages.

[0020] In contrast to prior approaches for offloading tasks, the pluggable AI-ML acceleration engine can be integrated with any existing computing devices, e.g., specialpurpose processors or processing units, to improve overall device performance and increase utilization of processing and / or memory resources of a heterogeneous computing platform. The pluggable architecture of the AI-ML acceleration engine allows the thread execution strategy to be universally applicable across different heterogeneous computing platforms.Attorney Docket No.: 56113-0796WO1

[0021] The thread-level integration techniques allow for dynamic integration of pluggable AI-ML acceleration engines into thread execution environment of the heterogeneous computing platform in a manner that reduces the operational overhead of the platform. For example, hardware I / O devices such as AI / ML accelerator engines can be integrated into main / central processor's execution environment without the added complexity, latency, and overhead of doorbell interrupt mechanisms and cross-mapping of memory that are required for conventional offload approaches.

[0022] The techniques for seamless thread-level integration of hardware I / O devices utilize semaphore integer variables to efficiently control access to resources that are shared by multiple processing threads, cores, and / or I / O devices of the computing platform. The thread-level integration techniques also incorporate key sharing mechanisms in the platform's operating system to implement enhanced resource sharing and memory mapping between a main / central processor and pluggable AI / ML acceleration engines of the platform. The key sharing mechanisms can be adapted to implement simplified and streamlined resource sharing with higher efficiency over conventional sharing techniques that use static, fixed-size carve outs of system memory.

[0023] Further, integrating a PAAE or hardware I / O device into a thread execution environment of a central processor enables memory' sharing / mapping techniques that use a uniform virtual address (VA) to physical address (PA) view among various components of the platform. This uniform VA to PA mapping can be used to implement demand paging techniques, where the uniform VA to PA view simplifies the corresponding VA to PA address translation and existing OS scheduler support can be utilized to adaptively manage main memory' mappings and shared allocations among threads of a thread pool. These features increase memory usage efficiency across the heterogenous computing platform.

[0024] The thread-level integration techniques can be implemented using a common instruction set architecture (‘’ISA”) that covers the instructions for both central processor and a PAAE or other hardware I / O device. This mitigates or eliminates the need to address potential issues with instruction portability’ and backward compatibility, which in turn encourages broader adoption of the disclosed heterogeneous platform and thread-level integration techniques over conventional platforms that use traditional off-load techniques.

[0025] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.Attorney Docket No.: 56113-0796WO1BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Fig. 1 is a block diagram of an example heterogeneous computing platform.

[0027] Fig. 2 illustrates an example of a conventional implementation of a heterogeneous computing platform that offloads tasks to an example I / O device.

[0028] Fig. 3 is a block diagram of an example pluggable AI / ML acceleration engine (PAAE).

[0029] Fig. 4 is a block diagram of another example pluggable AI / ML acceleration engine (PAAE).

[0030] Fig. 5 is a block diagram of an example heterogeneous computing platform that includes processing cores of a central processor and one or more PAAEs.

[0031] Fig. 6 illustrates an example framework for end-to-end thread execution.

[0032] Fig. 7 illustrates an example implementation of thread-level shared memory’ mapping.

[0033] Fig. 8 is an example process for integrating hardware I / O devices into a thread execution environment.

[0034] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0035] Fig. 1 is a block diagram of an example heterogeneous computing platform 100 that includes a system-on-chip 102 (“SoC 102”). The heterogeneous computing platform 100 can be an example computing system of a mobile, client, or server computing device and is referred to alternatively as system 100. The SoC 102 includes a central processor 104, such as a central processing unit (“CPU”), a memory controller 105, a shared memory 106 (“memory 106”). and an IP / circuit block 110. In some implementations, system 100 can include multiple SoCs and any descriptions for the SoC 102 will apply equally to each of the multiple SoCs that may be included at heterogeneous computing platform 100.

[0036] The central processor 104 can be a general-purpose CPU (e.g., a single or multicore CPU). The central processor 104 generates one or more indicators, such as an app- launch indicator or a function call that is triggered in response to executing or launching an application at a user device. For example, the application can be a camera application that uses an imaging sensor to generate image data or a gaming application that requires substantial memory' and graphics processing resources to render graphical content of the game. The central processor 104 also generates one or more application values, such as pixelAttorney Docket No.: 56113-0796WO1 values or frame rate. The application values may be associated with a function call, may be descriptive of an event that occurs during execution of the application, or both.

[0037] The memory 106 is a system memory, shared memory, or both. In the example of Fig. 1, memon 106 is depicted external to circuit block 110. However, memory 106 can include portions of memory that are: i) specific to circuit block 110, ii) external to circuit block 110, or iii) both. The memory 106 can be random access memory of the SoC 102, such as static random-access memory (SRAM), dynamic random-access memory (DRAM), a synchronous DRAM (SDRAM), or double data rate (DDR) SDRAM.

[0038] In some implementations, aspects of memory 106 are configured as a shared scratchpad memory' that supports parallel access of its memory resources by two or more processors of the circuit 110. The memory 106 can also include various other types of memory, such as high bandwidth memory (HBM), narrow memory (e.g., for storing 8-bit values), wide memory (e.g., for storing 16-bit or 32-bit values), etc.

[0039] The SoC 102 also includes an artificial intelligence / machine learning (AI / ML) thread control manager 108 that is implemented in hardware and software. An operating system, such as Linux or Android, runs on the central processor 104 and includes an OS scheduler that arranges and manages processes in a work queue or thread pool of the central processor 104. Aspects of the AI / ML thread control manager 108 are executed by the OS scheduler. In some implementations, functions of the AI / ML thread control manager 108 are executed as an integrated sub-routine of the OS scheduler. Aspects of the AI / ML thread control manager 108 can be also implemented as firmware of the SoC 102 or firmware of a device of the SoC 102, such as a proprietary' processor or IP device of the SoC 102 or the central processor 104. The AI / ML thread control manager 108 schedules the execution of processing threads within a thread execution environment of the central processor 104.

[0040] The central processor 104 uses its OS scheduler (e.g., AI / ML thread control manager 108) to dynamically integrate example hardware I / O devices such as the PAAEs 118 into its thread execution environment. The system 100 can include a cluster of PAAEs 118, where each PAAE 118 includes a special-purpose processing unit (SPPU) 116. The SPPU 116 can be a hardware I / O device that is configured as an individual PAAE 118. In some implementations, system 100 includes multiple PAAE 118, where each PAAE 118 includes one or more individual compute engines. In the example of Fig. 1, the individual compute engines can be represented by SPPU 116, but other types of compute engines or processing devices may be used. The OS scheduler, including the AI / ML thread control manager 108, of the central processor 104 can establish a processing thread and generate or access instructionsAttorney Docket No.: 56113-0796WO1 that are used to seamlessly integrate multiple hardware I / O devices (or PAAEs) into a thread execution environment of central processor 104. This is described in detail below at least with reference to the examples of Fig. 3 and Fig. 4.

[0041] The special-purpose processing unit (SPPU) 116 can generally be any computing device having specialized hardware configured to perform specialized computations including, e.g., neural network computations or other machine-learning ( ’ML") computations. For example, the SPPU 116 can be a GPU or can include computational functionality of a GPU. As another example, the SPPU 1 16 can be a tensor processing unit (TPU) or can include the tensor-based computational functionality of TPU. As another example, the SPPU 116 can be a neural processing unit (NLP) that implements neural network ML models. Further examples of the SPPU 116 include other specialized processing units, such as application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0042] The circuit block 110 generally includes individual IP devices such as processors, processor cores, or special-purpose processing devices. For example, the circuit block 110 can include one or more of an image signal processor (ISP) 112. a digital signal processor (DSP) 114, or a graphics processing unit (GPU) 117, such as a general-purpose GPU. The circuit block 110 is referred to alternatively as an IP block 110, where the IP block can include one or more proprietary hardware elements. For example, each of the ISP 112, DSP 114, GPU 117 can be a respective proprietary IP block (or IP device) of a particular entity or device manufacturer. In some implementations, the PAAE 1 18 and / or SPPU 1 16 can be included among the IP devices of IP block 110.

[0043] The SoC 102 cooperates with the memory device 122 to perform computations across one or more bank groups of the memory device 122. The computations can be for operations or workloads that involve one or more of the processors at IP block 110, including the PAAE 118. Additionally, the computations can be for a heterogenous operation that spans multiple processors of IP block 110, multiple IP blocks 110, or both. In at least one example the memory device 122 may be external to the SoC 102, whereas in another example the memory device 122 may be internal to the SoC 102.

[0044] In the example of Fig. 1, system 100 and the SoC 102 is an integrated circuit of an example computing device 130, which can be a consumer electronic device or mobile device, and can be represented by items such as a smartphone 130a, tablet 130b, laptop 130c, or smartwatch (or wearable device) 130d. The devices 130 may also include other items such as an eNotebook, Netbook, smart speaker, or mobile computer, or any electronic device with anAttorney Docket No.: 56113-0796WO1SoC having a central processor that communicates with an integrated or pluggable hardware I / O device. In some implementations, the system 100 and the SoC 102 are integrated circuits of a desktop computer, network server, or related cloud-based asset.

[0045] Fig. 2 illustrates an example of a conventional implementation of a heterogeneous computing platform 200, where a processor interacts with an offload engine (I / O device) to offload tasks to the I / O device. The heterogeneous computing platform 200 is referred to alternatively as system 200. The heterogeneous computing platform 200 includes a central processing unit 204 ('‘CPU 204”) and an example hardware I / O device 216, such as an ML hardware accelerator or special-purpose processing unit (SPPU) 216. In the example of Fig. 2, the heterogeneous computing platform is shown as being integrated in a host computing device 200, such as a mobile / client device 202. This traditional heterogeneous computing platform 200 implements conventional brute force approaches to offload computing tasks from the CPU 204 to the hardware I / O device 216.

[0046] In some implementations, to offload an example compute task, the CPU 204 loads program code and / or data for the task from disk memory of the computing device 200 and stores the code / data in memory device 122. The hardware I / O device is configured to access the code / data via a set of address spaces that differ from address spaces allocated to the CPU 204. An operating system (“OS”) kernel of the CPU 204 communicates with the hardware I / O device through a device driver, which introduces additional latency and complexity when offloading tasks to the hardware I / O device from the CPU 204.

[0047] For example, the device driver is required to send a doorbell signal to the hardware I / O device to provide information for executing the task. The information can include a starting address for a program counter (“PC”) and pointers to the code / data required to execute the task at the hardware I / O device. After executing the task, the hardware I / O device can generate and / or pass an interrupt signal to the CPU 204. In some cases, the relevant compute task is requested by an application or program of computing device 200. An interrupt service routine of the CPU 204 processes the intermpt signal to initiate a callback function of the device driver, which informs the requesting application the compute task is complete.

[0048] The CPU 204 includes a scalar processing unit 232 (or “scalar unit 232” for short) that executes scalar instructions and / or a vector processing unit 234 (or “vector unit 234” for short) that executes vector instructions. The scalar unit 232 and the vector unit 234 can each include a number of hardware components, including one or more of: register files, units of memory, arithmetic units, multiplexors, decoders, or other hardware.Attorney Docket No.: 56113-0796WO1

[0049] Vector instructions may be associated with different operations, operands, modifiers, and other properties than scalar instructions. For example, vector and scalar instructions may have different operation codes, memory object types, constant types, destination modifiers, source modifiers, synchronization modes, instruction compression schemes, lock modes, or any combination thereof.

[0050] As shown in the example of Fig. 2, a compute engine 240 of the SPPU 216 includes a vector unit 242 that executes vector instructions and an AI / ML processing unit 244 (or '‘AI / ML unit 244’’ for short) that executes AI / ML instructions. For example, the AI / ML instructions can include instructions for tensor acceleration, e.g., instructions for speeding up the processing, e g., matrix multiplication or accumulation, of the tensors. The AI / ML unit 244 can include the same or different hardware components than the scalar unit 232 or the vector unit 234 mentioned above.

[0051] The conventional heterogeneous computing platform 200 includes an operating system 220 that supports one or more applications 225. For example, as mentioned above, the one or more applications 225 can include a camera application that uses an imaging sensor to generate image data or a gaming application that requires substantial memory and graphics processing resources to render graphical content of the game. As another example, the one or more applications 225 can include artificial intelligence (Al) applications that execute ML models that are deployed locally at host computing device 200, or execute ML models that are deployed at least partly a cloud server remote from the host computing device 200.

[0052] The CPU 204 can offload processing operations to the SPPU 216 to reduce the processing workload of the CPU 204 when requests to execute an application 225 is received by the CPU 204. As discussed above, the host computing device 200 uses a doorbell intermpt mechanism to offload the ML workload to the SPPU 216. The SPPU 216 includes a doorbell block 246 that can receive a doorbell signal from the CPU 204. The doorbell block 246 can include any number of doorbells. For example, an application 205 can be assigned one or more doorbells, and the application 205 can ring the appropriate doorbell based on the type of workload the application is submitting. The SPPU 216 also includes an interrupt block 248 that can generate an interrupt signal to the CPU 204.

[0053] As illustrated in Fig. 2, the CPU 204 is executing a main thread. To submit a ML workload to the SPPU 216 while executing the main thread, the CPU 204 generates a doorbell signal 252 and sends the doorbell signal 252 to the doorbell block of the SPPU 216. In doing so, the CPU 204 sends a message to the SPPU 216 to initiate the ML workload. In some cases, the CPU 204 continues executing the main thread after sending the doorbellAttorney Docket No.: 56113-0796WO1 signal whereas, in other cases, the CPU 204 temporarily halts the execution of the main thread after sending the doorbell signal.

[0054] Upon receiving the doorbell signal 252, the SPPU 216 will then begin executing the ML workload. Once done, the SPPU 216 sends an interrupt signal 254 to the CPU 204 to notify the CPU 204 of the completion of the ML workload. Upon receiving the intermpt signal 254, in the cases where the main thread is still being executed at the CPU 204. the CPU interrupts any existing threads and swaps out that existing thread to resume execution of the main thread. For example, the main thread resumes execution using the results that are received as an output of the ML workload executed by the SPPU 216. Alternatively, in other cases where the execution of the main thread has been halted, the CPU 204 restarts the main thread that may be on halt, and then begins executing the restarted thread based on the new context.

[0055] A number of problems may arise, however, with this doorbell intermpt mechanism. One of the problems is the latency between the CPU 204 and SPPU 216. The CPU microarchitecture and the physical properties of the data communication channel between the CPU 204 and SPPU 216 limit how fast a doorbell signal can be sent to the SPPU 216 and an interrupt can be received from the SPPU 216.

[0056] Another problem is shared memory' management between the CPU 204 and SPPU 216. Conventionally, an off-load engine (e.g., SPPU) based heterogeneous computing platform uses carved out system memory for ease of memory usage by the off-load engines. Such carved out memory is no longer visible to the other applications; hence the user / client device 130 often appears to have much less system memory' capacity than it actually has. The management of a shared memory' region between the CPU 204 and SPPU 21 is not only complicated but also introduces delay.

[0057] Another problem is with the interrupt service routine. When the CPU 204 receives an interrupt signal, it executes an interrupt service routine (ISR) corresponding to the received interrupt signal. Because executing the ISR involves calling a device driver by way of function callbacks, the CPU 204 may experience a significant interrupt latency, which is the time it takes for the CPU 204 to finish the execution of the ISR since the interrupt signal is received.

[0058] A device driver is a computer program that controls or operates one or more hardware components in a computing system. The device driver provides a software interface between the hardware components and an operating system and / or applications and utilities. Device drivers and the operating system components may interact via function calls, such asAttorney Docket No.: 56113-0796WO1 application programming interfaces (APIs) function calls that are published and updated over time by the operating system vendor, the hardware vendor, or standards groups; and function callbacks by an operating system component to function addresses provided by the device driver.

[0059] Yet another problem is that most off-load engines (e.g., SPPUs) implement proprietary instruction set architectures (ISA) that are entirely different from the ISA of the CPU 104. For example, the off-load engines often employ ISAs that differ from one chip vendor to another, and the ISAs often lack backward compatibility even from the same chip vendor. As a result, applications that use a particular off-load engine may lack portability across other off-load engines due to binary incompatibility'.

[0060] Fig. 3 is a block diagram that shows an example implementation of a pluggable AI / ML acceleration engine. In the example of Fig. 3, the PAAE 118 can include multiple hardware I / O devices 318, where each hardware I / O device can be an AI / ML compute engine, such as an ML hardware accelerator, SPPU, or the like. In this and other examples, each of the multiple hardware I / O devices 318 can be application-specific I / O devices (e g., pluggable AI / ML accelerators) that are configured to recognize and execute a unique subset of instructions that efficiently extend a central processor’s thread-level computing capabilities. In some implementations, one or more of the hardware I / O devices 318 are special-purpose processing units, such as SPPU 116 or another processing device, that are configurable as a pluggable hardware I / O device of the central processor 104.

[0061] In contrast to the conventional computing platform 200 of Fig. 2, the central processor 104 of system 100 uses its I / O thread control manager 108 and the PAAE 118 to implement efficient, streamlined thread-level integration of hardware I / O devices of an SoC. In the example of Fig. 3. the central processor 104 leverages control logic represented by its I / O thread control manager 108 to generate instructions that are used to seamlessly integrate multiple hardware I / O devices into a thread execution environment of central processor 104. Notably, using the disclosed techniques, the central processor 104 can implement the threadlevel integration without the long latency, signaling complexity, and memory management pitfalls of the traditional platforms described above.

[0062] For example, the central processor 104 is configured to establish one or more processing threads and dynamically allocate or assign one or more of those processing threads to the PAAE 118 for instantiation and / or execution of the processing thread at the PAAE 118. The processing thread can be assigned to the PAAE 118 to implement threadlevel execution of the vector and matrix computations that are performed to execute AI / MLAttorney Docket No.: 56113-0796WO1 workloads. For example, the AI / ML workload can be a request from an application to compute an ML inference for a generative speech or computer vision task.

[0063] The central processor 104 includes an operating system 310 that supports one or more applications 225. In addition to the example applications described above, the one or more applications 225 can also include applications that request or leverage neural networks and other ML models and resources, including certain serv er- / cloud-based resources. More generally, the applications 225 can be any application that submits a request to the central processor 104 to execute an AI / ML task or workload. The central processor 104 can leverage the disclosed techniques to efficiently offload ML tasks or workloads to dedicated ML hardware of the PAAE 118 to accelerate execution of the requested ML workload.

[0064] In some implementations, the operating system 310 is configured to associate one or more application requests with a corresponding processing thread(s) / context(s) and, for each application request, dynamically allocate or assign distinct processing threads to different AI / ML engines 318 of the PAAE 118. The operating system 310 cooperates with the I / O thread control manager 108 and its scalar unit 232 to implement a minimum instruction set that is configured to instantiate a meaningful compute sequence or processing thread at an AI / ML engine 318 of the PAAE 118. For example, the central processor 104 can have a subset of dedicated thread-level integration instructions that include a predefined opcode sequence for dynamically configuring an I / O device (e.g.. PAAE 118) to execute within an instance of a CPU context or processing thread of the central processor 104.

[0065] In some implementations, the PAAE 1 18 (or hardware I / O device) is configured to execute actions associated with a routine of a timer interrupt handler to instantiate a new thread on the PAAE 118 or AI / ML engine 318. For example, the PAAE 118 can load required data values and workload execution information from memory, such as memory’ device 122 (e.g., DRAM). The PAAE can then write the relevant data values into a set of system registers of the PAAE that are configured to support task execution under the new thread. The PAAE can also initialize a set of general-purpose architecture registers and configure or set PC to point to a particular address or instruction. In response to performing these initialization actions, the PAAE may then exit from the timer interrupt handler routine.

[0066] As shown in the example of Fig. 3, each AI / ML engine 318 of the PAAE 118 includes a scalar unit 320, a vector unit(s) 330, and an AI / ML processing unit(s) 340 (“AI / ML unit 340’'). The scalar unit 320 is operable to execute a streamlined, subset of instructions that are uniquely’ configured to enable seamless integration of the PAAE 118 intoAttorney Docket No.: 56113-0796WO1 the thread execution environment managed by operating system 310 of the central processor 104.

[0067] The vector unit(s) 330 executes vector instructions 335 to perform various vector operations that are typically associated with ML computations, whereas the AI / ML unit(s) 340 executes AI / ML instructions 345, for example, to perform matrix multiplication and other arithmetic operations normally associated with ML computations. In some implementations, the vector unit 330 includes one or more non-linear units (e.g.. for applying activation functions) and dedicated reduction circuitry for performing reduction operations, for example, on accumulated values that are computed based on matrix multiplication executed by the AI / ML unit 340. Each of the vector unit(s) 330 and AI / ML unit(s) 340 can include a number of hardware components, including one or more of: register files, units of memory (e.g., activation registers, weight registers), arithmetic units (e.g., adders, multipliers), multiplexors, decoders, or other hardware.

[0068] The AI / ML instructions 345 can include instructions for performing computations to execute ML inferences (and / or training). For example, this can include neural network computations for processing neural network inputs through one or more layers of a neural network. In some implementations, neural network inputs are processed using data structures such as multi-dimensional tensors (or matrices) and the AI / ML instructions 345 are for tensor acceleration, e.g., instructions for accelerating tensor computations, e.g., matrix multiplication or accumulation, of the tensors. The AI / ML unit 340 can include the same or different hardware components than the scalar unit 320 or the vector unit 330 of the AI / ML engine 318.

[0069] The vector instructions and the AI / ML instructions executed by the compute engine 240 may be any instructions included in an instruction set that is specific to the SPPU 216. For example, the instruction set may be based on a Complex Instruction Set Computing (CISC) instruction set architecture (ISA), a Reduced Instruction Set Computing (RISC) ISA, or a Very Long Instruction Word (VLIW) ISA designed for instruction-level parallelism.

[0070] Fig. 4 shows another example implementation of a pluggable AI / ML acceleration engine. In the example of Fig. 4, the PAAE 400 includes multiple processing pipelines that are each capable of executing one or more types of instructions. In some implementations, each pipeline can execute a different type of instruction, whereas in some other implementations, two or more of the multiple pipelines can execute the same type of instruction. In some implementations, the multiple processing pipelines of the PAAE 400 operate on data that is accessible from a shared memory that is visible to, and accessible by, aAttorney Docket No.: 56113-0796WO1 particular processing thread or core of the central processor 104 and to the PAAE 400. This is described in detail below with reference to the examples of Fig. 6 and Fig. 7.

[0071] Processors typically process instructions by executing the instruction in a series of small steps. In some cases, to increase the number of instructions being processed by the processor (and therefore increase the speed of the processor), the processor may be pipelined. Pipelining refers to providing separate stages in a processor where each stage performs one or more of the small steps necessary to execute an instruction. In some cases, the pipeline (in addition to other circuitry) may be placed in a processor core of the processor. Some processors may have multiple processor cores, and in some cases, each processor core may have multiple pipelines. Where a processor core has multiple pipelines, groups of instructions (also referred to as issue groups) may be issued to the multiple pipelines in parallel and executed by each of the pipelines in parallel.

[0072] In the example of FIG. 4, the PAAE 400 has four pipelines where each pipeline is configured to execute a respective subset of the instructions. The first pipeline is configured to execute all of the instructions - including the scalar instructions 325. the vector instructions 335. and the AI / ML instructions 345 - included in an instruction set that corresponds to the PAAE 400. The first pipeline includes a scalar unit that executes scalar instructions, a vector unit that executes vector instructions, and an AI / ML unit that executes AI / ML instructions. Moreover, the first pipeline includes a full instruction decoder 402 that is configured to decode all types of instructions for execution on the scalar unit 320, the vector unit 330, and the AI / ML unit 340.

[0073] The second pipeline is configured to execute a portion of the instructions that include both the AI / ML instructions 345 and the vector instructions 335 in an instruction set that corresponds to the PAAE 400. The second pipeline implements instruction decoding and processing for the vector unit 330 to execute a portion of the vector instructions 335 that are assigned to the second pipeline and for the AI / ML unit 340 to execute a portion of the AI / ML instructions 345 that are assigned to the second pipeline. In some implementations, the second pipeline includes a partial instruction decoder 404 configured to decode the respective portions of vector and AI / ML instructions that are assigned for execution on the vector unit 330 and the AI / ML unit 340 along the second processing pipeline.

[0074] The third pipeline is configured to execute a portion of the instructions that include the AI / ML instructions 345 in the instruction set corresponding to the PAAE 400. The third pipeline implements instruction decoding and processing for the AI / ML unit 340 to execute the portion AI / ML instructions 345 that are assigned to the third pipeline. In someAttorney Docket No.: 56113-0796WO1 implementations, the third pipeline includes a partial instruction decoder 406-3 configured to decode the respective portions of AI / ML instructions 345 that are assigned for execution on the AI / ML unit 340 along the third processing pipeline.

[0075] The fourth pipeline is configured to execute a portion of the instructions that also include the AI / ML instructions 345 in the instruction set corresponding to the PAAE 400. The fourth pipeline also implements instruction decoding and processing for the AI / ML unit 340 to execute the portion AI / ML instructions 345 that are assigned to the fourth pipeline. In some implementations, the fourth pipeline includes a partial instruction decoder 406-4 configured to decode the respective portions of AI / ML instructions 345 that are assigned for execution on the AI / ML unit 340 along the fourth processing pipeline.

[0076] Fig. 5 is a block diagram of an example heterogeneous computing platform 500. The heterogeneous computing platform 500 is an example implementation of system 100 that includes two or more processing cores of a central processor 104 that can form a cluster of CPU / processor cores and two or more PAAEs 118 that can form a cluster of PAAEs. In the example of Fig. 5, the central processor 104 of the heterogeneous computing platform 500 includes a cluster of processing cores (e.g., core 1, core 2. core 3. and core A and a cluster of PAAEs (e g., PAAE 1, PAAE 2, PAAE 3, . .. , PAAE N). In this example, each PAAE in the cluster of PAAEs corresponds to the AI / ML engines 318 described above with reference to Fig. 3.

[0077] In the conventional heterogeneous computing platform 200 described above, portions of system memory that are can ed out for usage by an off-load engine (or I / O device) are only visible to, and / or accessible by, that off-load engine (or I / O device). In these conventional implementations of memory allocation, the carved-out portions of system memory (e g., DRAM) are no longer visible to, or accessible by, the processor cores of the central processor 104. For example, when the operating system 310 has limited visibility to I / O device memory mapping, this often leads to rigid memory mapping and potential overallocation of memory resources to I / O devices. This inefficient approach to memory allocation and usage needlessly restricts the pool of available memory’ resources within system 100 and ultimately degrades the system’s overall performance.

[0078] In contrast to traditional / conventional techniques, the heterogeneous computing platform 500 includes a thread-level integration technique for shared memory mapping. This thread-level memory sharing technique allocates memory resources within the context or processing thread of a CPU / central processor’s thread execution environment. This allows for fine-grained, thread-level memory allocation such that the memory resources (e.g.,Attorney Docket No.: 56113-0796WO1DRAM resources) can be made visible to one or more processing cores of the central processor 104 and to one or more AI / ML engines 318 in a cluster of PAAEs 118.

[0079] For example, one or more AI / ML engines 318 in a cluster of PAAEs can be integrated under a control thread of operating system 310 to expose the AI / ML engines 318 to the central processor’s memory management control schemes. In some implementations, this thread-level integration for memory management allows the PAAE 118 to implement more fine-grained operations for memory requests and allocations, e.g., 4 KB granularity, per application.

[0080] Fig. 6 is an example illustration of framework 600 for end-to-end thread execution that can be executed at the heterogeneous computing platform 100. In the example of Fig. 6, a CPU or central processor 104 is configured to execute multiple threads, including a main thread and multiple processing threads that are used to execute respective tasks of one or more machine-learning (“ML”) workloads.

[0081] As illustrated, the central processor 104 can initiate framework 600 by executing a main thread. The multiple processing threads that are used to execute, by multiple PAAEs, respective tasks of the one or more ML workloads are spawned from the main thread. The spawned threads execute in parallel with each other. After all of the spawned threads terminate, e.g., upon reaching a synchronization point, execution of the main thread resumes. A “synchronization point” refers to a time point where multiple independently executing threads arrive at a common position. Because the multiple independently executing threads may arrive at the common position at different times, some of the threads can enter an idle state as those threads wait for one or more other threads until all of the threads arrive at the common position.

[0082] Each PAAE (or hardware I / O device) can be configured to have an independent power domain (e.g., an independent voltage and / or frequency domain) that is independently powered and independently activatable. Each PAAE 318 can be powered on independently from other PAAEs. For example, tw o PAAEs can operate in parallel with two different voltages supplied to them, e.g., one of the PAAEs may be powered on to allow it to continue its execution of a corresponding ML workload while another PAAE is inactive and / or powered off after having completed its execution of a corresponding ML workload.

[0083] In some implementations, each PAAE can be further configured to have an independent power domain that is independent from the power domain of the central processor 104 or other components of the heterogeneous computing platform 100. For example, the central processor 104 that is executing the main thread can be powered off whileAttorney Docket No.: 56113-0796WO1 the threads corresponding to the PAAEs are running, and can be later powered back on after the threads corresponding to the PAAEs have terminated.

[0084] The independent power domains allow power management at a finer granularity, which improves the power management of the heterogeneous computing platform 100. For example, a power management unit of the platform can more adaptively transition different PAAEs of the platform between different power states. In some implementations, this enables various PAAEs of the platform to transition between different activity states, such as between normal operational modes, hibernation modes, or power-off modes, based on their respective progresses toward the completion of the ML workloads execution. In addition, the independent power domains enable the use of advanced or intelligent power management algorithms.

[0085] Fig. 7 illustrates an example implementation of thread-level shared memory mapping. In the example of Fig. 7, the PAAE 118 operates on and processes data for working sets that are stored at memory locations that are among a set of shared memory resources from the memory units of system 100. For example, the set of shared memory resources can be included in system / shared memory 106, memory device 122 (e.g., DRAM), or both. As described above, the PAAE 118 can be integrated within the same thread execution environment of the central processor 104 such that the set of shared memory' resources is visible to, and accessible by, the PAAE 118 as well as the particular processing thread or core under which the PAAE 118 is integrated.

[0086] The central processor 104 can include a set of I / O integration instructions that are uniquely configured to implement thread-level I / O device memory mapping and resource sharing within the thread execution environment of the central processor 104. The I / O integration instructions can be a set of scalar or integer instructions that can be identified by a PC address pointer for execution locally using a scalar unit of the PAAE 118. In some implementations, the central processor 104 is configured to generate specific memory mapping / allocation instructions with predefined opcode sequences for dynamically configuring load and store operations, physical address (PA) to virtual address (VA) mapping, and memory requests / allocations at the hardware I / O device.

[0087] The system 100 can include an example pluggable harness that is configured to implement fine-grained memory' allocations and thread-level resource sharing based on an instruction set that is fully supported by each of the scalar unit 232 and the scalar unit 320. The harness can be implemented via register sharing techniques that use integer values and sync flags to map and expose PAAE 118 to, for example, subset of memory pages 710 (e.g.,Attorney Docket No.: 56113-0796WO1 pagel, page2, page3) that are shared between the PAAE 118 and central processor 104 within the context of a processing thread (e.g., "thread l "). In some implementations, the harness is implemented using a semaphore programming construct that acts as a signal or a counter for controlling access to memory resources that are shared between multiple processes, such as processes of the PAAE 118 and the central processor 104.

[0088] Fig. 8 is a flow diagram of an example process 800 for integrating a hardware I / O device into a thread execution environment of a central processor. Process 800 is implemented or executed by a heterogeneous computing platform 100. The descriptions of process 800 will reference the above-mentioned heterogeneous computing platform 100. In some examples, the steps or actions of process 800 are enabled by programmed software instructions, firmware instructions, or both. Each type of instruction may be stored in a non- transitory machine-readable storage device and is executable by one or more of the processors or other resources described in this specification.

[0089] The heterogeneous computing platform 100 instantiates, at the central processor 104, a first processing thread that is used to execute a task of a machine-learning (AIL") workload (step 802). For example, the ML workload may include computations for training a neural network or computing an inference using a neural network. The central processor 104 has a streamlined instruction set that is configured for recognition by each hardware I / O device to facilitate thread-level integration and communication between the central processor and the hardware I / O devices. Such a streamlined instruction set includes at least a first instruction and, when the central processor is a central processing unit (CPU), a second instruction, as will be discussed in greater detail below, among other instructions.

[0090] The heterogeneous computing platform generates, by the central processing unit, a first instruction that allocates the first processing thread to the hardware I / O device (step 804). In some implementations, the hardware I / O device is a special-purpose processing unit. For example, the hardware I / O device can be an ML hardware accelerator configured to accelerate computations for the ML workload. As another example, the hardware I / O device can be a hardware ML tensor processing unit (“TPU”) configured to accelerate ML workload computations that use multi-dimensional tensors. As another example, the hardware I / O device can be a general-purpose graphics processing unit (GPU).

[0091] In some implementations, the first instruction is a thread integration instruction adapted for local execution at the hardware I / O device by using a scalar unit of the hardware I / O device that executes scalar instructions. Hence the first integration instruction can be a scalar instruction. When the first processing thread is allocated to the hardware I / O device.Attorney Docket No.: 56113-0796WO1 the hardware I / O device's instruction fetch and decode unit fetches and decodes the first instruction, and an AI / ML thread control manager of the heterogeneous computing platform schedules the first processing thread for execution in appropriate cycles, e.g., when the source operands of the instruction for the first processing thread become available. The source operands may include a constant, or may be fetched from the register files or memoiy. such as system memory or other memory locations. Then the instruction along with the source operands are sent to the hardware I / O device for execution.

[0092] The heterogeneous computing platform integrates the hardware I / O device into a thread execution environment of the central processing unit in response to having executed the first instruction at the hardware I / O device, e.g., by using the scalar unit of the hardware I / O device (step 806). In other words, at a result of the execution of the first instruction at the hardware I / O device, an operating system of the central processor will be able to recognize the hardware I / O device as a thread and rather than an individual hardware device (which would otherwise have to be interacted with through a doorbell interrupt mechanism).

[0093] In some implementations, the central processor 104 is a central processing unit (CPU) that has one or more processor cores, and the thread execution environment of the central processor 104 thus includes a processor core configured to execute multi -threaded operations. In these implementations, the central processor 104 can instantiate the first processing thread as a subordinate context of the multi -threaded operations that the processor core is configured to execute. Moreover, in these implementations, to integrate the hardware I / O device into the thread execution environment of the central processor 104, the heterogeneous computing platform 100 can dynamically configure the hardware I / O device to execute vector arithmetic operations within an instance of a CPU thread.

[0094] Further, in these implementations, the heterogeneous computing platform 100 can generate a second instruction that configures resource sharing at the central processor 104. The resources include at least a set of memory resources that are used to execute the multithreaded operations. In response to executing the second instruction, a thread-based shared memory mapping can be performed within the thread execution environment of the central processor 104 so as to expose an address space of the shared set of set resources to the hardware I / O device (e.g., AI / MU engine 318) concurrent with initiating the thread-based shared memoi ' mapping.

[0095] The heterogeneous computing platform performs, at the hardware I / O device, the task of the ML workload by executing vector multiplications using a matrix computing unit of the hardware I / O device (step 808). Notably, the heterogeneous computing platform 100Attorney Docket No.: 56113-0796WO1 executes the ML workload at the hardware I / O device but as an operation under the first processing thread (e.g., a CPU thread) and that occurs within the thread execution environment of the central processor 104 (e.g., a CPU).

[0096] As indicated above, in some implementations, the respective steps of process 800 are performed at a hardware integrated circuit as part of a larger compute operation to generate ML inference (and / or training) outputs, including an output for a neural network layer of a neural network that implements one or more ML models. For example, the output can be a portion of a computation for a ML task or inference workload to generate an image processing, image recognition, or generative speech output. A portion of the integrated circuit can include a special-purpose neural network processor or hardware ML accelerator configured to accelerate computations for generating and rendering different types of image, speech, or data processing outputs.

[0097] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of. data processing apparatus.

[0098] Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0099] The term “computing system” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.Attorney Docket No.: 56113-0796WO1

[0100] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0101] A computer program may. but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0102] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (General purpose graphics processing unit).

[0103] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random-access memory or both. Some elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0104] Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory7Atorney Docket No.: 56113-0796WO1 devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0105] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0106] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

[0107] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0108] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can alsoAttorney Docket No.: 56113-0796WO1 be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0109] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0110] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

Attorney Docket No.: 56113-0796WO1What is claimed is:

1. A method implemented using a central processor that communicates with a hardware Input / Output (“I / O”) device, the method comprising: instantiating, at the central processor, a first processing thread used to execute a task of a machine-learning (“ML”) workload; generating, by the central processor, a first instruction that allocates the first processing thread to the hardware I / O device; integrating the hardware I / O device into a thread execution environment of the central processor in response to executing the first instruction at the hardware I / O device; and executing, at the hardware I / O device, the task of the ML workload by performing vector multiplications using a matrix computing unit of the hardware I / O device.

2. The method of claim 1, wherein the central processor is a CPU and integrating the hardware I / O device into a thread execution environment of the central processor comprises: dynamically configuring the hardware I / O device to execute vector arithmetic operations within an instance of a CPU thread.

3. The method of claim 1, wherein the first instruction is a thread integration instruction adapted for local execution at the hardware I / O device using a scalar unit of the hardware I / O device.

4. The method of claim 2, wherein: i) the thread execution environment of the central processor comprises a first processor core configured to execute multi-threaded operations; and ii) the method further comprises, instantiating the first processing thread as a subordinate context of the multi-threaded operations that the first processor core is configured to execute.

5. The method of claim 4, further comprising: allocating, to the first processor core, a set of resources that are used to execute the multi-threaded operations; andAttorney Docket No.: 56113-0796WO1 configuring the set of resources as a shared resource set that is accessible to the first processor core and hardware I / O device.

6. The method of claim 5, wherein configuring the set of resources as a shared resource set of the central processor comprises: generating a second instruction that configures resource sharing at the central processor; in response to executing the second instruction, performing thread-based shared memory mapping within the thread execution environment of the central processor; and exposing an address space of the shared resource set to the hardware I / O device concurrent with initiating the thread-based shared memory mapping.

7. The method of claim 6, further comprising: executing, by the first processor core and the hardware I / O device, fined-grained memory access requests concurrently against the shared resource set of the central processor.

8. The method of claim 7, wherein the concurrent, fined-grained memory access requests are executed within the thread execution environment of the central processor.

9. The method of claim 6, further comprising: provisioning, using the shared resource set of the central processor, fined-grained memory access allocations concurrently to the first processor core and the hardw are I / O device.

10. The method of claim 1, wherein the hardw are I / O device is an ML hardware accelerator configured to accelerate computations for the ML workload.

11. The method of claim 1, wherein the hardware I / O device is a hardware ML tensor processing unit (“TPU”) configured to accelerate ML workload computations that use multidimensional tensors.

12. The method of claim 1, wherein the hardware I / O device is a general-purpose graphics processing unit (GPU).Attorney Docket No.: 56113-0796WO113. The method of claim 1, wherein the ML workload is an ML inference workload or an ML training workload.

14. A method implemented using a scalar unit of a hardware I / O device that communicates with a central processing unit (“CPU”), the method comprising: identifying a request to execute a machine-learning (“ML”) inference workload; receiving, by the hardware I / O device, an instruction indicating CPU allocation of a processor thread; based on the instruction, instantiating, by a scalar unit of the hardware I / O device, the allocated processor thread as a CPU thread implemented at the hardware I / O device; and executing the ML inference workload at the hardware I / O device but as an operation under the CPU thread and that occurs within a thread execution environment of the CPU.

15. A hardware compute engine of a hardware neural network (“NN”) processor, the hardware compute engine comprising: a scalar unit configured to: i) receive an instruction from a central processing unit (“CPU”); ii) execute the instruction to initiate CPU thread support at the NN processor; and iii) integrate the NN processor into a thread execution environment of the CPU in response to executing the instruction at the scalar unit; a vector unit configured to process vector instructions for executing a NN inference; and a matrix computing unit configured to perform vector multiplications to execute the NN inference based on the vector instructions.

Images