Dynamic voltage and frequency scaling management in artificial intelligence device

Abstract

An AI device may include a neural processing unit (NPU) that can execute neural networks. Agents may run on the AI device to ensure that the performance of the NPU meets quality of service (QoS) requirements in executions of neural networks. Jobs for executing one or more neural networks may be submitted by applications running on the AI device. The jobs may be grouped into queues based on their priorities. The jobs in the same queue may have the same priority. Different queues may be assigned to different agents. An agent may monitor the jobs in the queue that is assigned to the agent. The agent may also be subscribed to one or more other queues and receive activity data of the other queues. The agent may determine its residency on the NPU and use the residency and its QoS requirement to adjust the frequency of the NPU.

Description

PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCTDYNAMIC VOLTAGE AND FREQUENCY SCALING MANAGEMENT IN ARTIFICIAL INTELLIGENCE DEVICECross-Reference to Related Application

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 713,620, filed October 30, 2024, and titled "QUALITY OF SERVICE BASED DYNAMIC VOLTAGE AND FREQUENCY SCALING," which is incorporated by reference in its entirety.Technical Field

[0002] This disclosure relates generally to artificial intelligence (Al), and more specifically, dynamic voltage and frequency scaling (DVFS) management (e.g., priority aware DVFS management) in Al devices.Background

[0003] DVFS is a power management technique that can adjust the voltage or frequency of a device (e.g., a computing device) to reduce power consumption. DVFS is used in many modern computers to improve battery life, device longevity, and power savings. Quality of service (QoS) may be methods that prioritize computing jobs to ensure that important applications and services have the bandwidth, speed, and reliability they need. QoS can improve user experience by focusing on performance and availability. DVFS algorithms can be used to select desirable frequencies for the work being performed on processing units.Brief Description of the Drawings

[0004] Embodiments can be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0005] FIG. 1 is a block diagram of an Al device, in accordance with various embodiments.

[0006] FIG. 2 illustrates DVFS agents assigned to queues with different priorities, in accordance with various embodiments.

[0007] FIG. 3 illustrates execution of neural network jobs with single priority, in accordance with various embodiments.PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT

[0008] FIG. 4 illustrates execution of neural network jobs with two different priorities, in accordance with various embodiments.

[0009] FIG. 5 illustrates execution of neural network jobs with three different priorities, in accordance with various embodiments.

[0010] FIG. 6 illustrates a DVFS decision making process, in accordance with various embodiments.

[0011] FIG. 7 is a flowchart of a method of executing one or more neural networks, in accordance with various embodiments.

[0012] FIG. 8 is a block diagram of an example deep neural network (DNN) system, in accordance with various embodiments.

[0013] FIG. 9 is a block diagram of an example computing device, in accordance with various embodiments.Detailed DescriptionOverview

[0014] There are various ways to control the DVFS feature of a processing unit, such as a central processing unit (CPU) or graphics processing unit (GPU) across the stack. Typically, the application using the device, the Operating System managing it and any firmware running on it could play a role. For example, the application may require more performance from the GPU because it wants the best latency for its real-time workload. The Operating System may want to increase frequency on the GPU because the queues that it services are filling up. The firmware running on the GPU may increase the frequency because its utilization has increased beyond a certain threshold.

[0015] Typically, the further down the stack you go, the more difficult it is to make the correct decision for the user that is submitting the work. For example, increasing the frequency of a device because its queue of work is intermittently long would make sense when the latency on the workloads when the queue is long, compromises a user experience. It would be fine for the frequency to stay low when the user experience is not compromised. Any algorithm that spins up the frequency by monitoring the utilization may unnecessarily push the device to a less efficient operation.

[0016] The DVFS feature of neural processing units (NPUs) is often controlled by using the average residency of work on the NPU to apply frequency changes. The algorithm isPCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT required to be fine-tuned to meet real-time application latencies. That meant that any application with non-real-time workloads, such as Background, Idle, or any non-latency sensitive work would have the same frequency.

[0017] Embodiments of the present disclosure may improve on at least some of the challenges and issues described above by providing priority aware DVFS management for NPUs. The approach in the present disclosure can provide frequency boosts for higher priority work and limit frequency for lower priority work.

[0018] In various embodiments of the present disclosure, an Al device with a NPU may have DVFS agents running on another processing unit, which may be a NPU microcontroller or a CPU. The work submitted to the NPU, such as work for executing DNNs, may be allocated to queues labeled with different priority bands. The jobs in the same queue may have the same priority band. The priority bands may be determined by application (s) or Operating System(s) that run on the Al device and deploy the DNNs. The queues are assigned to the DVFS agents with each DVFS agent handling a particular priority band. The agent has access to the data of the corresponding queue. A DVFS agent may also be subscribed to data for one or more other queues, such as queues at higher priority bands than the queue of the agent itself. The agent may accumulate activity data across the priority bands it is subscribed to and determines a residency based on the activity data. The residency may indicate an amount of time that the NPU would be active for the agent, which may include the time needed to execute the jobs in the queue of the agent plus jobs in the queue(s) of all subscribed bands. Higher priority jobs can preempt a lower priority job. For instance, a lower priority job may be delayed or paused so that the NPU can perform a higher priority job first. After the higher priority job is done, the NPU can then start or resume the lower priority job.

[0019] The agent may use the residency to estimate a performance of the NPU and determine a frequency of the NPU based on the estimated performance and a QoS requirement for the queue. The frequency of the NPU may indicate the number of cycles that the NPU can complete per second. The higher the frequency, the better the performance. The QoS requirement may be a latency requirement, such as the amount of time it takes to perform the jobs in the queue by the NPU. A queue at a higher priority band may require less latency than a queue at a lower priority band. In an example, the queue with the lowest priority may have no latency requirement, while the queue with the highestPCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT priority may have the strictest latency requirement. The agent may increase the frequency after it determines that the estimated performance of the NPU does not meet the QoS requirement or decrease the frequency after it determines that the estimated performance of the NPU exceeds the QoS requirement for the agent. The agent may keep the frequency as is after it determines that the estimated performance of the NPU matches the QoS requirement. Different agents may adjust frequency independently. For instance, different agents may have differing criteria for what activity constitutes an increase or decrease in frequency, which may be based on the QoS requirements for the priority bands being monitored by the agents.

[0020] This disclosure addresses determining the frequency required on the NPU to meet the latency requirements of different applications submitting work requiring varying levels of service to the NPU at the same time. Bringing sensitivity to the traffic class being scheduled to the frequency selection can reduce the likelihood of the frequency being increased unnecessarily (at the cost of efficiency) or over eagerly reduced (at the cost of performance). With the approach in the present disclosure, power consumption on realtime Al workloads can be reduced. Also, it can provide lower latency for in-focus applications and facilitate unconditionally efficient execution for background work. The approach in this disclosure is more advantageous than currently available approaches, including utilization based DVFS management techniques.

[0021] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it can be apparent to one skilled in the art that the present disclosure may be practiced without the specific details or / and that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0022] Further, references are made to the accompanying drawings that form a part hereof, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0023] Various operations may be described as multiple discrete actions or operations in turn, in a mannerthat is most helpful in understanding the claimed subject matter.PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCTHowever, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0024] For the purposes of the present disclosure, the phrase "A or B" or the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase "A, B, or C" or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term "between," when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0025] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives "first," "second," and "third," etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0026] In the following detailed description, various aspects of the illustrative implementations are be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0027] The terms "substantially," "close," "approximately," "near," and "about," generally refer to being within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., "coplanar," "perpendicular," "orthogonal," "parallel," or any other angle between the elements, generally refer to being within + / - 5-20% of a target value as described herein or as known in the art.

[0028] In addition, the terms "comprise," "comprising," "include," "including," "have," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, device, or hardware accelerator that comprises a list ofPCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, device, or DNN accelerators. Also, the term "or" refers to an inclusive "or" and not to an exclusive "or."

[0029] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one or more implementations of the subject matter described in this specification are set forth in the description below and the accompanying drawings.

[0030] FIG. 1 is a block diagram of an Al device 100, in accordance with various embodiments. The Al device 100 can perform Al tasks, e.g., by using DNNs. The Al device 100 includes an interface module 110, an NPU 120, a DVFS manager 130, a CPU 140, and a memory 150. In other embodiments, alternative configurations, different or additional components may be included in the Al device 100. For instance, the Al device 100 may include multiple NPUs or CPUs. The Al device 100 may also include other types of processing units, such as GPU. Further, functionality attributed to a component of the Al device 100 may be accomplished by a different component included in the Al device 100 or a different device or system. In some embodiments, the Al device 100 may be an Al personal computer (PC). In other embodiments, the Al device 100 may be at least part of a server, such as a server in a data center.

[0031] The interface module 110 facilitates communication between the Al device 100 with other devices or systems. For example, the interface module 110 may receive information of DNNs to be executed by the NPU 120, such as input data, internal parameters of DNNs, hyperparameters of DNNs, and so on. The interface module 110 may also receive data to train DNNs. As another example, the interface module 110 may send out results of Al tasks performed by the Al device 100 to other devices or systems.

[0032] The NPU 120 is a processing unit that can execute DNNs. The NPU 120 may also be referred to as an Al processor. Even though not shown in FIG. 1, the NPU 120 may include one or more memories. In some embodiments, the NPU 120 includes processing elements (PEs) specifically designed for performing neural network operations, such as convolutions, layer normalization operations, attention operations, linear operations, and so on. An example processing element may include a multiply-accumulate (MAC) unit that can perform multiply-accumulate operations. The NPU 120 may have a different architecturePCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT from the CPU 140. An example of the NPU 120 may be a DNN accelerator, such as the DNN accelerator 802 in FIG. 8.

[0033] In some embodiments, the NPU 120 may operate at various frequencies. A frequency of the NPU 120 may refer to a clock speed of the NPU 120, which may be the number of cycles that the NPU 120 can execute per second. The frequency may indicate how fast the NPU 120 can process data. A higher frequency may mean faster processing power, and the NPU 120 can execute more instructions per second, resulting in better performance. In some embodiments, the NPU 120 may operate at multiple frequencies in the execution of a DNN. The NPU 120 may use the same frequency to perform multiple jobs in the execution of a DNN. Operating frequencies of the NPU 120 may be measured in gigahertz (GHz). In an example, the NPU 120 may operate at frequencies of approximately 1 GHz. Operating frequencies of the NPU 120 may be determined by the DVFS manager 130.

[0034] The DVFS manager 130 manages DVFS of the NPU 120. In some embodiments, the DVFS manager 130 can adjust the voltage or frequency of the NPU 120 to improve the performance of the NPU 120 during execution of DNNs. Execution of a DNN may be performed for training the DNN or deploying the DNN for performing an Al task. Examples of the Al tasks may include language processing, image classification, audio processing, autonomous driving, medical diagnosis, and so on. The execution of the DNN may include a plurality of jobs to be performed by the NPU 120. A job (also referred to as a neural network job or DNN job) may be the execution of at least part of a neural network operation in the DNN or execution of at least part of a layer in the DNN. The DVFS manager 130 may improve the performance of the NPU 120 in the execution of one or more DNNs by reducing its power consumption. For instance, the DVFS manager 130 may estimate the performance of the NPU executing the DNN jobs and adjust the frequency of the NPU 120, when needed, to ensure that the performance of the NPU 120 meets one or more QoS requirements. The QoS requirements may be predetermined, e.g., determined before the execution of the DNN(s). In some embodiments, the QoS requirements may be predetermined by the CPU 140, and the DVFS manager 130 may receive data indicating the QoS requirements from the CPU 140.

[0035] As shown in FIG. 1, the DVFS manager 130 includes a queue module 133, a plurality of DVFS agents 135 (individually referred to as "DVFS agent 135"), a virtual channel module 137, and a filtering module 139. In other embodiments, the DVFS manager 130 may includePCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT fewer, more, or different components. For instance, the DVFS manager 130 may include a microcontroller (not shown in FIG. 1) on which the queue module 133, DVFS agents 135, or virtual channel module 137 runs. In some embodiments, the DVFS manager 130 may be implemented at least partially through software.

[0036] The queue module 133 may group jobs into queues, also referred to as "job queues." Different DNN jobs may have different priorities. A job may be considered more important than another job. In an example, the latency requirements of the job may be stricter than the latency requirement of the other job. For instance, the job may be submitted by an infocus application. The in-focus application may be a foreground application, such as an active application that a user is interacting with at or near the time the job is submitted. In contrast, the other job may be submitted by a background application, such as an application that no user is interacting with at or near the time the job is submitted. In some embodiments, the queue module 133 may generate a queue for jobs having the same priority. The queue module 133 may label the queue with a priority band that indicates the priority of the jobs. The queue module 133 may generate multiple queues that are each at a different priority band. The queue module 133 may then assign the queues to the DVFS agents 135. For instance, the queue module 133 may assign a queue to a particular DVFS agent 135. A DVFS agent 135 may be allocated to one queue at a time.

[0037] In some embodiments, the queue that is assigned to a DVFS agent 135 is the primary queue of the DVFS agent. The DVFS agent 135 has subscription to the data of its primary queue by default. There is a 1:1 relationship between primary queues and DVFS agents 135. The virtual channel module 137 may set up virtual channels for the DVFS agents 135. A virtual channel may be a channel for a DVFS agent 135 to receive data of a queue that is not assigned to the DVFS agent 135. Such a queue is referred to as an external queue of the DVFS agent 135. Jobs in an external queue are referred to as external jobs. The virtual channel module 137 may identify one or more external queues for a DVFS agent 135 and generate a virtual channel between the DVFS agent 135 and each of the identified external queue(s). In some embodiments, the virtual channel module 137 may identify the external queue(s) from the queues for executing at least part of a single DNN. In other embodiments, the virtual channel module 137 may identify the external queue(s) from the queues for executing multiple DNNs, such as DNNs that are executed by the NPU 120 at the same time.PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT

[0038] In some embodiments, a virtual channel generated by the virtual channel module 137 may facilitate transmission of data of the entire external queue to the DVFS agent 135 so that the DVFS agent 135 may receive data of all the jobs in the external queue, despite whether any job in the external queue is outside the time window of the queue of the DVFS agent 135. In other embodiments, a virtual channel generated by the virtual channel module 137 may facilitate transmission of data of a part of the external queue to the DVFS agent 135. For instance, the virtual channel facilitates transmission of data of external job(s) that falls into the time window of the queue of the DVFS agent 135 but does not support transmission of data of external job(s) that falls outside the time window of the queue of the DVFS agent 135. An external job falls into the time window of the queue of the DVFS agent 135 when at least part of the time window of performing the external job is included in the time window of performing a job in the queue of the DVFS agent 135.

[0039] Through the virtual channel(s), the DVFS agent 135 can be subscribed to not only data of its own queue but also data of external queue(s). The data of the external queue(s) would be useful for the DVFS agent 135 to estimate performance of the NPU 120 during the residency of the DVFS agent 135 on the NPU 120. The residency of the DVFS agent 135 may indicate the amount of time that the NPU 120 is active for the queue that the DVFS agent 135 is assigned to. The residency is also referred to as agent residency. Different DVFS agents may have different residencies. As an external job associated with a virtual channel has a higher priority, a job in the queue of the DVFS agent 135 may be preempted by the external job. For instance, the performance of the job in the queue of the DVFS agent 135 may be paused or delayed for the NPU 120 to perform the external job. The performance of the job in the queue of the DVFS agent 135 may be started or resumed afterthe external job is completed. The DVFS agent 135 is considered active during the performance of the external job. With the data of the queue of the DVFS agent and data of external queue(s) at higher priority bands, the DVFS agent 135 can determine its total residency on the NPU 120. For instance, the DVFS agent 135 may amalgamate the times needed for performing all the jobs in the queue of the DVFS agent 135 and external jobs, the data of which is received through the virtual channel(s), to determine its total residency.

[0040] The time for performing a job may include a hardware preparation time and a job execution time. The hardware preparation time may be the time needed to get the NPU 120 ready, such as the time for storing data needed for the job into a memory in the NPU 120,PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT and so on. The job execution time may be the time that the NPU 120 performs computations in the job. In embodiments where the job is an external job that prompts a job in the queue of the DVFS agent 135, the time for performing the external job may also include a preemption time, which may be a time for doing the preemption. The preemption time may be before the hardware preparation time. The time for performing the preempted job (i.e., the job in the queue of the DVFS agent 135) may include a preemption time and a job execution time.

[0041] The DVFS agent 135 may estimate the performance of the NPU 120 based on its total residency. The DVFS agent may then determine whether the estimated performance of the NPU 120 meets the QoS requirement for the agent or for the queue. For instance, the DVFS agent 135 may compare the estimated performance of the NPU 120 with a QoS parameter for the agent. In an example, the DVFS agent 135 may compare the total residency with a latency threshold that indicates a latency requirement. The DVFS agent 135 may adjust a frequency of the NPU 120 based on the determination. When the estimated performance of the NPU 120 fails to meet the QoS requirement, the DVFS agent 135 may increase the frequency of the NPU 120. The increase of the frequency may improve the performance of the NPU 120. When the estimated performance of the NPU 120 matches the QoS requirement, the DVFS agent may keep the frequency of the NPU 120 the same. When the estimated performance of the NPU 120 exceeds the QoS requirement, the DVFS agent may decrease the frequency of the NPU 120, which can worsen the performance of the NPU 120.

[0042] The filtering module 139 may receive frequencies determined by the DVFS agents 135 and apply a QoS filter on the frequencies. In some embodiments, the filtering module 139 may obtain the highest end frequency by using the QoS filter. For instance, the QoS filter may filter out low end frequencies. The filtering module 139 may apply the highest end frequency to the NPU 120. The NPU 120 may operate at the highest end frequency when performing the jobs. In other embodiments, the NPU 120 may operate at a frequency determined by a DVFS agent 135 when performing the job(s) in the queue to which the DVFS agent is assigned to.

[0043] The CPU 140 may facilitate one or more Operating Systems running on the Al device. The CPU 140 facilitates applications running on the Al device 100. The applications may be run within the one or more operating systems. Examples of the applications may include word processing applications, presentation applications, web browsers, messaging ioPCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT applications, media player, streaming applications, image or video editing applications, antivirus applications, and so on. In some embodiments, one or more applications running on the Al device 100 may perform tasks based on DNNs. Such tasks may be Al tasks. For example, a word processing application may use a DNN (e.g., a large language model) to edit or generate words. As another example, a web browser may use a DNN to do an online search. An application with an Al task may submit jobs to the NPU 120. The jobs may be jobs for executing a DNN, such as jobs for performing the neural network operations in the DNN.

[0044] In some embodiments, one or more applications running on the Al device 100 may determine priorities of DNN jobs to be performed by the NPU 120. In an example, the priority of a DNN job for executing a DNN may be determined by the application that deploys the DNN. The applications may determine priorities of DNN jobs based on latent requirements. For instance, an in-focus application (e.g., a word processing application that the user is actively using) may have stricter latency requirements than a background application (e.g., an antivirus application running in the background). The jobs submitted by the in-focus application would have higher priorities than jobs submitted by the background application. Priorities of DNN jobs may be provided to the queue module 133 for setting up queues, as described above.

[0045] The memory 150 stores data associated with the Al device 100. The data may include data received, used, or generated by the Al device 100. The data may also include computer program instructions executed by the NPU 120 or CPU 140. In some embodiments, the data may include data generated by the DVFS manager 130, such as agent residencies, NPU frequencies, and so on. The memory 150 may include a dynamic random-access memory (DRAM) or static random-access memory (SRAM). Even though FIG. 1 show one memory, the Al device 100 may include multiple memories.

[0046] FIG. 2 illustrates DVFS agents 220A-220C assigned to queues 210A-210C at different priority bands, in accordance with various embodiments. For the purpose of illustration, FIG. 2 shows three queues 210A-210C (collectively referred to as "queues 210" or "queue 210") and three DVFS agents 220A-220C (collectively referred to as "DVFS agents 220" or "DVFS agent 220"). There may be a different number of queues 210 or DVFS agents 220 in other embodiments. The DVFS agents 220 may be examples of the DVFS agents 135 in FIG. 1.

[0047] For the purpose of illustration, the queue 210A has three jobs, the queue 210B has two jobs, and the queue 210C has four jobs. The jobs are represented by boxes with a dotPCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT pattern in FIG. 2. In some embodiments, the queues 210 at three different priority bands, meaning they have three different priorities, respectively. For instance, the queue 210A has a higher priority than the queue 210B, and the queue 210B has a higher priority than the queue 210C. The jobs in the same queue may have the same priority, i.e., the priority of the queue. For instance, all the three jobs in the queue 210A have a higher priority than the two jobs in the queue 210B, and the two jobs in the queue 210B have a higher priority than all the four jobs in the queue 210C.

[0048] The jobs in the same queue may have different schedules. For instance, the jobs may be executed at different times. Jobs in different queues may have time conflicts. For instance, the time for executing a job may partially or wholly overlap with the time for executing another job. A job with a higher priority is considered as a more important job and may be performed more timely. When there is a conflict between the job and a job with a lower priority (e.g., time conflict), the job with the higher priority may be executed before the job with the lower priority. In some embodiments, the hardware device may pause the execution of the job with the lower priority to execute the job with the higher priority and may not resume the execution of the job with the lower priority until it completes the job with the higher priority.

[0049] The DVFS agents 220 may facilitate DVFS management of the hardware device that executes the jobs in the queues 210. The hardware device may be an NPU, such as the NPU 120 in FIG. 1. In some embodiments, a DVFS agent 220 may have access to data of the queue 210 (e.g., data of the jobs in the queue) to which the DVFS agent 220 is assigned. For instance, the DVFS agent 220 may receive data of all jobs in the queue 210, which may be the start time of each job, the end time of each job, the amount of time needed for completing each job, data storage resource needed for completing each job, computational resources needed for completing each job (e.g., type(s) of data processing hardware, number of processing elements, etc.), other types of information about each job, or some combination thereof. As shown in FIG. 2, the queue 210A is assigned to the agent 220A and is the primary queue of the agent 220A, the queue 210B is assigned to the agent 220B and is the primary queue of the agent 220B, and the queue 210C is assigned to the agent 220C and is the primary queue of the agent 220C. Taking the DVFS agent 220C for example, the DVFS agent 220C may facilitate execution of the four jobs in the queue 210C. For instance, thePCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCTDVFS agent 220C may monitor or control the time when a job is executed, such as the start time, the end time, the amount of time needed for completing the job, and so on.

[0050] A DVFS agent 220 may be registered to receive data of one or more external queues. An external queue of the DVFS agent 220 is a queue that is assigned to a different DVFS agent 220. The DVFS agent 220 may receive data of an external queue through a virtual channel. Data received from the virtual channel may include data of one or more jobs in the external queue. In some embodiments, the virtual channel may be for the entire external queue. For instance, the DVFS agent 220 receives data of all the jobs in the external queue. In other embodiments, the virtual channel may be for one or more particular jobs in the external queue. For instance, the DVFS agent 220 receives data of the job(s) having time conf lict(s) with any of the jobs in the queue 210 but does not receive data of the other job(s) in the external queue.

[0051] In some embodiments, the DVFS agent 220 may be registered to receive data of one or more external queues at one or more higher priority bands. For example, the DVFS agent 220C may be registered to receive data of the queue 210A and queue 210B. As the external queue is at a higher priority band than the queue 210C, a job in the external queue, may be executed between two jobs in the queue 210C or be executed in the middle of a job in the queue 210C. The DVFS agent 220C may use data of the queue 210C plus data received through the virtual channel(s) to estimate the performance of the NPU. The DVFS agent 220C may then determine whether the estimated performance meets a QoS requirement for the priority band of the queue 210C. In an example where the DVFS agent 220C determines that the estimated performance exceeds the QoS requirement, the DVFS agent 220C may decrease the frequency of the NPU, which can reduce power consumption. In an example where the DVFS agent 220C determines that the estimated performance matches the QoS requirement, the DVFS agent 220C may keep the frequency of the NPU as is. In an example where the DVFS agent 220C determines that the estimated performance fails to meet the QoS requirement, the DVFS agent 220C may increase the frequency of the NPU. In some embodiments, the NPU may execute the jobs in the queue 210C at the frequency determined by the DVFS agent 220C.

[0052] In other embodiments, the NPU may execute the jobs in all the three queues 210 at the frequency that is selected from frequencies determined by the three DVFS agents 220. For instance, the DVFS agent 220B may determine a frequency based on data of the queuePCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT210B and data of at least part of the queue 210A. The DVFS agent 220A may determine a frequency based on data of the queue 210A. The frequencies determined by the three DVFS agents 220 may be different. The highest end frequency out of the frequencies may be selected as the operating frequency of the NPU for executing the jobs in the three queues 210.

[0053] FIG. 3 illustrates execution of neural network jobs with single priority, in accordance with various embodiments. In the embodiments of FIG. 3, there is one queue Pl that is assigned to one agent, which is denoted as Pl_agent. The queue has four jobs: Pl_0, Pl_l, Pl_2, and Pl_3, which have the same priority. The four jobs may be jobs for running a DNN and may be submitted by an application in which the output of the DNN may be used to perform an Al task. The application submits the first three jobs (i.e., Pl_0, Pl_l, and Pl_2) concurrently. The agent residency, which is shown as "Pl_tO" in FIG. 3, may be measured as the duration of time between the first job (i.e., P l_0) being queue by the firmware to the point where the result for the last outstanding job (i.e., Pl_2) is returned. The agent residency may indicate the amount of time that the NPU is active for Pl. The agent residency Pl_tO includes a first hardware preparation time 301, a Pl_0 job execution time 302, a second hardware preparation time 303, a Pl_l job execution time 304, a third hardware preparation time 305, and a Pl_2 job execution time 306.

[0054] After the first three jobs are completed, the application makes a single submission and submits the fourth job Pl_3. The agent residency for P 1_3 is shown as Pl_tl in FIG. 3, which includes a hardware preparation time 307 and a Pl_3 job execution time 308. The total agent residency of Pl_agent for Pl is Pl_tO + Pl_tl.

[0055] FIG. 4 illustrates execution of neural network jobs with two different priorities, in accordance with various embodiments. For the purpose of illustration and simplicity, FIG. 4 shows the queue Pl in FIG. 3 plus another queue P17. P17 is at a higher priority band than Pl and is assigned to another agent P17_agent. P17 has two jobs: P17_0 and P17_l, which are submitted separately by the application. P17_agent receives data of its own queue P17 but has no access to data of Pl. The total agent residency of P17_agent is P17_t0 + P17_tl, in which P17_t0 is the agent residency for the first job in the queue P17 and P17_t 1 is the agent residency for the second job in the queue P17. The agent residency of P17_agent is shown as two dashed arrows in FIG. 4.PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT

[0056] P17_t0 includes a preemption time 401, a hardware preparation time 402, and a job execution time 403. During the preemption time, the job P17_0 preempts the job Pl_l, which causes the execution of the job Pl_l to be stopped. The NPU may prepare for executing the job P17_0 during the hardware preparation time 402. For instance, data needed to execute the job P17_0 may be loaded into a memory of the NPU. Other types of preparations may also be performed during the hardware preparation time 402. During the job execution time 403, the NPU executes the job P17_0. The job execution time 403 is followed by another preemption time 404, during which the job Pl_l may be resumed. After the preemption time 404, execution of the job Pl_l is completed during a job execution time 405. The job execution time 405 is followed by a hardware preparation time 406 and a job execution time 407, which are for executing the job Pl_2. FIG. 4 shows additional hardware preparation times, which are shown by the black shapes in FIG. 4. Each hardware preparation time is followed by a job execution time in which the corresponding job is executed by the NPU.

[0057] Pl_agent is subscribed to P17 and receives data of both Pl and P17. The total agent residency of Pl_agent is P l_tO+P l_t 1+P 17_t 1, in which P l_t0 is the agent residency for the first three jobs in the queue Pl, Pl_tl is the agent residency for the fourth job in the queue Pl, and P17_tl is the agent residency for the second job in the queue P17. The agent residency of Pl_agent is shown as three solid arrows in FIG. 4. As shown in FIG. 4, P17_t0 is part of P l_t0. The NPU pauses the job Pl_l to execute the job P17_0 and resumes the job Pl_l after P17_0 is complete. The job P17_0 preempts the job Pl_l as P17 has a higher priority than Pl.

[0058] FIG. 5 illustrates execution of neural network jobs with three different priorities, in accordance with various embodiments. In the embodiments of FIG. 5, three queues Pl, P17, and P24 are active. Pl has one job Pl_0. P17 has four jobs: P17_0, P17_l, P17_2, and P17_3, which are submitted in two batches with the first batch including P17_0, P17_l, and P17_2 and the second batch including P17_3. P24 has two jobs P24_0 and P24_l. Pl, P17, and P24 are at three different priority bands, respectively. P24 has a higher priority than P17, which has a higher priority than Pl. The three queues Pl, P17, and P24 are assigned to three agents: Pl_agent, P17_agent, and P24_agent, respectively. Pl_agent is registered to gather residency data on Pl, P17, and P24; P17_agent is registered to P17 and P24; while P24_agent is registered to P24. This configuration would provide the ability for real-timePCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT work (P17 and 24) to have dedicated frequency increases to hit real-time QoS, while lower priority work would also account for its lost execution time by being preempted. Each job preemption may be associated with two preemption times. The preemption times are shown by unfilled shapes in FIG. 5. A hardware preparation time (which is shown by black shapes in FIG. 5) and a job execution time (which is shown by shapes with dots, grids, or diagonal lines in FIG. 5) are between the two preemption times.

[0059] In FIG. 5, the total agent residency of P24_agent is P24_t0 + P24_tl, in which P24_t0 is the agent residency for the first job in the queue P24 and P24_t 1 is the agent residency for the second job in the queue P24. The agent residency of P24_agent is shown as two dotted arrows in FIG. 5. The total agent residency of P17_agent is P17_t0+P17_tl+P24_tl, in which P17_t0 is the agent residency for the first three jobs in the queue P17, P17_t 1 is the agent residency for the fourth job in the queue P17, and P24_t 1 is the agent residency for the second job in the queue P24. The agent residency of P17_agent is shown as three dashed arrows in FIG. 5. The total agent residency of Pl_agent is longer than the total residency of P17_agent by the amount of the time needed for the job Pl_0. The agent residency of Pl_agent is shown as two solid arrows in FIG. 5.

[0060] For the purpose of illustration, each virtual channel between an agent and its external queue in FIGS. 4 and 5 is for the entire external queue. For instance, P17_agent in FIG. 5 receives data of the job P24_0 and the job P24_l. In other embodiments, a virtual channel between an agent and its external queue may be for the one or more jobs in the external queue that prompt one or more jobs in the agent's queue. For instance, P17_agent in FIG. 5 may receive data of the job P24_0, which preempts the job P17_l, but receives no data of the job P24_l, which does not preempt any job in P17.

[0061] FIG. 6 illustrates a DVFS process 600, in accordance with various embodiments. The DVFS process 600 may be performed, e.g., by the DVFS manager 130 in FIG. 1. As shown in FIG. 6, the DVFS process 600 includes three steps. The first step includes residency capturing 610, in which the agents receive the data based on their band subscription(s) and capture their residency. The second step includes agent QoS checking 620, in which each agent independently determines whether the captured residency meets the QoS requirement for the agent and makes an agent decision 625. An agent decision 625 may be a decision of increasing or decreasing the NPU frequency. Increasing the NPU frequency may shorten the execution time on the virtual channel, while decreasing the NPU frequency may lengthenPCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT the execution time on the virtual channel. The agent may alternatively decide to keep the frequency the same, e.g., when the agent determines that its residency matches the QoS requirement. For the purpose of illustration and simplicity, FIG. 6 shows five agents that perform the second step. In other embodiments, there may be fewer or more agents. The third step includes QoS filtering 630, which may be done by applying a final QoS filter. The QoS filter may select the highest frequency from the frequencies indicated by the agent decisions as a final frequency 635 before applying it to the NPU. The NPU may operate at the final frequency 635 to perform the jobs.

[0062] FIG. 7 is a flowchart of a method 700 of executing one or more DNNs, in accordance with various embodiments. The method 700 may be performed by the Al device 100 in FIG.1. Although the method 700 is described with reference to the flowchart illustrated in FIG.7, many other methods of executing DNNs may alternatively be used. For example, the order of execution of the steps in FIG. 7 may be changed. As another example, some of the steps may be changed, eliminated, or combined.

[0063] The Al device 100 assigns 710 a first job queue to an agent. The first job queue comprises one or more jobs to be performed by a NPUs for executing the one or more neural networks. In some embodiment, the agent is an example of the DVFS agents 135 in FIG. 1.

[0064] The Al device 100 provides 720 access to data of a second job queue to the agent. The second job queue is assigned to another agent and comprises one or more other jobs to be performed by the NPUs for executing the one or more neural networks. In some embodiments, the first job queue and the second job queue are executed by the NPUs with different priorities. The different priorities correspond to different QoS requirements for the agent and the another agent. The different priorities are determined by a computer program running on a CPU associated with the NPUs.

[0065] In some embodiments, the second job queue further comprises an additional job. A time of the additional job is outside a time window of the first job queue, and a time of the one or more other jobs is inside the time window of the first job queue. The Al device 100 provides data of the one or more other jobs to the agent. The Al device 100 bypasses providing data of the additional job to the agent.

[0066] The Al device 100 determines 730, by the agent, a frequency of the NPUs based on data of the first job queue, the data of the second job queue, and a QoS requirement for thePCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT first job queue. In some embodiments, the Al device 100 determines an amount of time during which the NPUs is active for the first job queue and the second job queue based on the data of the first job queue and the data of the second job queue. The Al device 100 estimates a performance of the NPUs executing the first job queue and the second job queue based on the amount of time. The Al device 100 adjusts an initial frequency of the NPUs by determining whether the estimated performance of the NPUs meets the QoS requirement.

[0067] In some embodiments, in response to determining that the estimated performance of the neural unit exceeds the QoS requirement, the Al device 100 decreases the initial frequency. In some embodiments, in response to determining that the estimated performance of the neural unit fails to meet the QoS requirement, the Al device 100 increases the initial frequency.

[0068] The Al device 100 executes 740, by the NPUs, at least part of the one or more neural networks based on the frequency. In some embodiments, the Al device 100 determines, by the another agent, another frequency of the NPUs based on the data of the second job queue and a QoS requirement for a priority level of the second job queue. The NPUs executes at least part of the one or more neural networks further based on the another frequency. In some embodiments, the Al device 100 selects a frequency from a group of frequencies that comprise the frequency and the another frequency. The Al device 100 executes, by the NPUs, the one or more neural networks at the selected frequency.

[0069] In some embodiments, the Al device 100 provides access to data of one or more other job queues to the agent. The one or more other job queues are assigned to one or more other agents and comprise additional jobs to be performed by the NPUs for executing the one or more neural networks. The frequency of the NPUs is determined by the agent further based on the data of the one or more other job queues.

[0070] FIG. 8 is a block diagram of a DNN system 800, in accordance with various embodiments. The whole DNN system 800 or a part of the DNN system 800 may be implemented in one or more computing devices, such as the Al device 100 in FIG. 1 or the computing device 900 in FIG. 9. The DNN system 800 can generate and execute DNNs. As shown in FIG. 8, the DNN system 800 includes a DNN module 801 and a DNN accelerator 802. In other embodiments, alternative configurations, different or additional components may be included in the DNN system 800. For instance, the DNN system 800 may includePCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT multiple DNN modules or multiple DNN accelerators. Further, functionality attributed to a component of the DNN system 800 may be accomplished by a different component included in the DNN system 800 or a different system. In some embodiments, the DNN module 801 and DNN accelerator 802 may include different types of processing units. In an example, the DNN module 801 may be implemented by one or more CPUs, such as the CPU 140. The DNN accelerator 802 may be an NPU, which may also be referred to as an Al accelerator or an Al processor. The DNN module 801 and DNN accelerator 802 may be implemented in the same chip or separate chips.

[0071] The DNN module 801 facilitates generation and deployment of DNNs. In some embodiments, the DNN module 801 may generate and train DNNs. For instance, the DNN module 801 can define the layered architecture of a DNN. The DNN module 801 can also determine the internal parameters of the DNN through a DNN training process. The DNN module 801 may also determine one or more hyperparameters that define how the DNN is trained. An example hyperparameter is a sparsity ratio that defines the sparsity level of one or more deep learning tensors for the DNN. The DNN module 801 may also compress DNNs, e.g., during or after training. In some embodiments, the DNN module 801 may prune internal parameters (e.g., weights) of one or more layers of a DNN by changing nonzero parameters to zeros. The DNN module 801 may prune internal parameters based on a target sparsity ratio. A sparsity ratio may be the ratio of the number of zero parameters to the total number of parameters. In an example where the DNN module 801 prunes parameters during DNN training, the DNN module 801 may prune parameters of a layer to achieve a target sparsity ratio after one or more epochs. The DNN module 801 may prevent the pruned parameters from changing values during the rest of the training process.Alternatively, the DNN module 801 may allow the pruned parameters to change values so that a pruned, zero parameter may have a nonzero value after further training. The DNN module 801 may prune parameters of the layer again after one or more additional epochs.

[0072] The DNN module 801 may deploy trained, compressed, or validated DNNs for use in neural network applications. In some embodiments, the DNN module 801 may distribute trained, compressed, or validated DNNs to devices or systems which may use the DNNs to perform tasks (e.g., image classification, motion planning, etc.) for which the DNNs were trained. In other embodiments, the DNN module 801 may facilitate deployment of the DNNs using the DNN accelerator 802. For instance, the DNN module 801 may receive data from aPCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT device or system coupled with the DNN system 800 and input the received data (or data generated by the DNN module 801, e.g., based on the received data) into a DNN. The DNN module 801 may generate instructions (e.g., computer program instructions) that can be executed by the DNN accelerator 802 for DNN execution. The DNN module 801 may receive an output of the DNN from the DNN accelerator 802. The DNN module 801 may transmit the output of the DNN (or a result of processing the output of the DNN by the DNN module 801) to the device or system. In some embodiments, the DNN module 801 may control execution processes of trained, compressed, or validated DNNs. The DNN module 801 may function as a compiler for DNNs executed by the DNN accelerator 802. The DNN module 801 may perform compilation of DNNs and generate compilation descriptors, based on which the DNNs may be executed.

[0073] The DNN module 801 may compress neural network operations to optimize or improve the performance of the DNN accelerator 802. For instance, the DNN module 801 may modify internal parameters of a neural network operation to reduce the memory and compute requirements for carrying out the neural network operation. In an example, the DNN module 801 may compress a weight tensor of the neural network operation on a block level and may use different compression ratios for different blocks while ensuring subblocks within the same block have the same compression ratio. Through the compression, the DNN module 801 may obtain a compressed weight tensor that has mixed formats. In an example, some weights may not be modified and have the original precision, while other weights may be modified to have a lower precision. A weight with the lower precision may have less bits and therefore, would requires less storage and compute resources. In another example, some weights may not be modified and have their original values, while other weights may be modified to have a value of zero. The zero weights may be skipped from storing in memory or being used in computations. The DNN module DNN module 801 may provide the compressed weight tensor to the DNN accelerator 802, and the latter may carry out the neural network operation using the compressed weight tensor.

[0074] The DNN accelerator 802 executes DNNs provided by the DNN module 801. For instance, the DNN accelerator 802 can execute a DNN by carrying out neural network operations in the DNN. The process of carrying out a neural network operation is also referred to as a process of executing the neural network operation or performing the neural network operation. The execution of the DNN may be for training the DNN or for using thePCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCTDNN to perform Al tasks. As shown in FIG. 8, the DNN accelerator 802 includes a memory 810, a direct memory access (DMA) engine 820, and DPUs 830 (individually referred to as "DPU 830"). In other embodiments, alternative configurations, different or additional components may be included in the DNN accelerator 802. For example, the DNN accelerator 802 may include more than one memory 810 or DMA engine 820. As another example, the DNN accelerator 802 may include a single DPU 830. Further, functionality attributed to a component of the DNN accelerator 802 may be accomplished by a different component included in the DNN accelerator 802 or by a different system. A component of the DNN accelerator 802 may be implemented in hardware, software, firmware, or some combination thereof.

[0075] The memory 810 stores data associated with neural network operations performed by the DNN accelerator 802. In some embodiments, the memory 810 may store data to be used by the DPUs 830 for executing neural network operations. The memory 810 may store input activations. The memory 810 may also store weights, such as weights in kernels of convolutions, which are determined by training DNNs. In some embodiments, the memory 810 may store weights having mixed formats. The memory 810 may further store outputs of neural network operations, such as output activations. In some embodiments, the memory 810 includes one or more DRAMs. In some embodiments (e.g., embodiments in which the memory 810 is implemented on a different chip from the DPUs 830), the memory 810 may function as an off-chip memory from the perspective of the DPUs 830.

[0076] The DMA engine 820 facilitates data transfer between the memory 810 and local memories of the DPUs 830. For example, the DMA engine 820 can read data from the memory 810 and write data into a local memory of a DPU 830. As another example, the DMA engine 820 can read data from a local memory of a DPU 830 and write data into the memory 810. For instance, the DMA engine 820 may read input activations and weights of convolution from the memory 810 and load the input activations and weights to one or more DPUs 830. The DMA engine 820 may also write output activations of convolutions computed by one or more DPUs 830 to the memory 810. The DMA engine 820 provides a DMA feature that allows the DPU 830 to initiate data transfer between the memory 810 and the local memories of the DPUs 830 and to perform other operations while the data transfer is being conducted. In some embodiments, the DMA engine 820 may read tensors from thePCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT memory 810, modify the tensors in a way that is optimized for the DPU 830 before it writes the tensors into the local memories of the DPUs 830.

[0077] The DPUs 830 perform neural network operations in DNNs. For instance, a DPU 830 may execute a DNN layer by running one or more deep learning operations in the DNN layer. A DPU 830 may execute a layer, or a portion of a layer, at a time. In some embodiments, the operations of the DNN layers may be run by multiple DPUs 830 in parallel. For instance, multiple DPUs 830 may each perform a portion of a workload for a neural network operation. Data may be shared between the DPUs 830. A DPU 830 may also be referred to as a NPUs, a compute block, or a compute tile.

[0078] The DPUs 830 may be capable of running various types of neural network operations, such as convolution (including depthwise convolutions), layer normalization, SoftMax operation, pooling, elementwise operation, linear operation, nonlinear operation, and so on. N=Neural network operations performed by the DPUs 830 include tensor operations, i.e., operations whose inputs are tensors or operations whose outputs are tensors. In an example, the DPU 830 receives an input tensor and one or more convolutional kernels and performs a convolution with the input tensor and convolutional kernels. The result of the convolution may be an output tensor, which can be further computed, e.g., by the DPU 830 or another DPU 830.

[0079] In the embodiments of FIG. 8, each DPU 830 includes a local memory 840, a control module 850, a processing engine 860, and a post-processing engine 870. The processing engine 860 and post-processing engine 870 may constitute a compute engine of the DPU 830. Some or all the components of the DPU 830 can be implemented on the same chip. In other embodiments, alternative configurations, different or additional components may be included in the DPU 830. Further, functionality attributed to a component of the DPU 830 may be accomplished by a different component included in the DPU 830, a different DPU 830, another component of the DNN accelerator 802, or a different system. A component of the DPU 830 may be implemented in hardware, software, firmware, or some combination thereof.

[0080] The local memory 840 is local to the corresponding DPU 830. In the embodiments of FIG. 8, the local memory 840 is inside the DPU 830. In other embodiments, the local memory 840 may be outside the DPU 830. Data in the local memory 840 may be transferred to or from the memory 810, e.g., through the DMA engine 820. In some embodiments, dataPCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT in the local memory 840 may be transferred to or from the local memory of another DPU 830. The local memory 840 may store data received, used, or generated by the control module 850, the processing engine 860, or the post-processing engine 870. Examples of the data may include input activations, weights, output activations, sparsity bitmaps, keys, values, and so on.

[0081] In some embodiments, the local memory 840 may store tensors to be processed by the processing engine 860 or the post-processing engine 870. The tensors may be input tensors of deep learning operations. The local memory 840 may store weights of mixed weights. The local memory 840 may also store tensors generated by the processing engine 860 or the post-processing engine 870. The tensors may be output tensors of deep learning operations. The layout of data points of a tensor in the local memory 840 may depend on the format in which the tensor is stored. In some embodiments, the local memory 840 may store tensors in various formats, including Z-major (e.g., ZXY or ZYX) format, X-major (e.g., XYZ or XZY) format, and Y-major (e.g., YXZ or YZX) format. For a tensor with Z-major format, the local memory 840 may store data points having the same (x, y) coordinate contiguously. For instance, the data points having the same (x, y) coordinate may be stored at a sequence of memory addresses in the local memory 840. For a tensor with the ZXY format or ZYX format, the local memory 840 may store data points having the same (x, y) coordinate contiguously. For instance, the data points having the same (x, y) coordinate may be stored at a sequence of memory addresses in the local memory 840. For a tensor with X-major format, the local memory 840 may store data points having the same (y, z) coordinate contiguously. For a tensor with Y-major format, the local memory 840 may store data points having the same (x, z) coordinate contiguously.

[0082] In some embodiments, the local memory 840 includes one or more SRAMs. The local memory 840 may be byte-addressable, and each memory address identifies a single byte (eight bits) of storage. In some embodiments, the local memory 840 may include memory banks (e.g., SRAM banks). The number of memory banks in the local memory 840 may be 16, 64, 128, 856, 512, 1024, 2048, or other numbers. A memory bank may include a plurality of storage units. In an example, a memory bank may include 8, 16, 64, or a different number of storage units. A memory bank or a storage unit in a memory bank may have a memory address. In an example, a storage unit may store a single byte, and data larger than a single byte may be stored in storage units with consecutive memory addresses, i.e.,PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT adjacent storage units. For instance, a storage unit can store an integer number in the INT8 format, versus two storage units may be needed to store a number in the FP16 or BF16 format, which has 16 bits. In some embodiments, 16 bits can be transferred from the local memory 840 in a single read cycle. In other embodiments, 16 bits can be transferred from the local memory 840 in multiple read cycles, such as two cycles.

[0083] The control module 850 controls data transfer between the local memory 840 and the processing engine 860 or between the local memory 840 and the post-processing engine 870. The control module 850 may load data (e.g., input activations, weights, etc.) from the local memory 840 into the processing engine 860 or post-processing engine 870. The control module 850 may also drain data (e.g., output activations, etc.) from the processing engine 860 or post-processing engine 870 into the local memory 840. In some embodiments, the control module 850 may transfer data from the local memory 840 into one or more storage units in the processing engine 860 or post-processing engine 870. A storage unit in the processing engine 860 or post-processing engine 870 may include one or more register files for storing data to be used for computations in deep learning operations. The control module 850 can transfer data generated by the processing engine 860 or post-processing engine 870 into the local memory 840. The data may be results of deep learning operations performed by the processing engine 860 or post-processing engine 870.

[0084] In some embodiments, the control module 850 may generate data transfer requests or manage the generation of data transfer requests by the processing engine 860 or postprocessing engine 870. A data transfer request may be a read request to read data from the local memory 840, such as data that the processing engine 860 or post-processing engine 870 may use to perform a deep learning operation. Additionally or alternatively, a data transfer request may be a write request to write data computed by the processing engine 860 or post-processing engine 870 into the local memory 840. The control module 850 may also facilitate transmission of response to data transfer requests from the local memory 840 to the processing engine 860 or post-processing engine 870. In some embodiments, the control module 850 may facilitate transmission of data transfer requests to data ports associated with the local memory 840.

[0085] The processing engine 860 performs operations in DNNs. The processing engine 860 may include one or more processing cells. In some embodiments, the processing cells may be arranged in one or more rows and one or more columns in the processing engine 860.PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCTEach processing cell may include PEs that may be arranged in an array that includes rows and columns. All the PEs in the processing engine 860 may constitute a bigger array that includes more rows and columns. An example PE may be or may include one or more MAC units that can perform MAC operations. In some embodiments (e.g., embodiments where the DPU 830 executes a convolutional layer), a computation in an MAC unit may be an MAC operation on an activation operand and a weight operand. The activation operand may be an activation tensor that may include one or more activations in the input tensor of the convolution. Different activations may be in different input channels. The weight operand may be a weight tensor that may include one or more weights in the filter of the convolution. The values of the weights are determined through training the DNN or compressing the neural network operation after training. The weights in the weight operand may be in different input channels. In some embodiments, the activation operand or weight operand is a vector along the input channel (IC) dimension.

[0086] In some embodiments, an MAC unit includes one or more multipliers for performing multiplications. An MAC unit may also include one or more accumulators ("adders") for performing accumulations. A column of MAC units is referred to as an MAC column. An MAC column may be associated with one or more MAC lanes. A MAC lane is a path for loading data e.g., by the control module 850, into an MAC column. A MAC lane may be also referred to as a data transmission lane or data loading lane. An MAC column may have multiple MAC lanes. The loading bandwidth of the MAC column is an aggregation of the loading bandwidths of all the MAC lanes associated with the MAC column. With a certain number of MAC lanes, data can be fed into the same number of independent MAC units simultaneously. In some embodiments where an MAC column has four MAC lanes for feeding activations or weights into the MAC column and each MAC lane may have a bandwidth of 16 bytes, the four MAC lanes can have a total loading bandwidth of 64 bytes.

[0087] In some embodiments, the processing engine 860 may be capable of depthwise convolution, standard convolution, or both. In a depthwise convolution, an MAC unit may perform an MAC operation that includes a sequence of multiplications for an input operand and a weight operand. Each multiplication in the sequence (also referred to as a cycle) is a multiplication of a different activation in the input operand with a different weight in the weight operand. The activation and weight in the same cycle may correspond to the same channel. The sequence of multiplication produces a product operand that includes aPCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT sequence of products. The MAC operation may also include accumulations in which multiple product operands are accumulated to produce an output operand of the MAC unit. The processing engine 860 may output multiple output operands at a time, each of which is generated by a different MAC unit. In a standard convolution, MAC operations may include accumulations across the channels. For instance, as opposed to generating an output operand, a MAC unit may accumulate products across different channels to generate a single output point.

[0088] In some embodiments, the processing engine 860 may include acceleration logic for accelerating neural network operations. For instance, each processing cell in the processing engine 860 may include one or more acceleration modules. In an example, each MAC column or each MAC row may have a corresponding acceleration module that accelerates MAC operations in the MAC column or MAC row. In some embodiments, an acceleration module accelerates computations in the processing engine 860 based on sparsity in input data. The acceleration module may include a storage unit that stores a sparsity map (e.g., a sparsity bitmap), which may be loaded to the storage unit by the control module 850. The sparsity map may indicate the sparsity pattern of a weight block and shows the positions of zero data elements and positions of nonzero data elements in a tensor, such as an input activation tensor or weight tensor.

[0089] Taking a weight tensor for example, a sparsity map of the weight tensor may have the same number of elements as the weight block and these elements are arranged in a data structure having the same shape as the weight tensor. An element in the sparsity map may indicate whether the corresponding element (i.e., weight) in the weight tensor is zero or not. For instance, a zero element in the sparsity map may indicate that the corresponding weight is zero, while a one-valued element in the sparsity map may indicate that the corresponding weight is nonzero. The acceleration module may use the sparsity map to identify activations and weights to be used in MAC operations by the MAC units and to transfer the identified activations and weights to the storage units (e.g., register files) of the MAC units. The other activations and weights may be skipped from computation to accelerate the neural network operation.

[0090] The post-processing engine 870 also performs computations in neural network operations. In some embodiments, the post-processing engine 870 processes outputs of the processing engine 860. The post-processing engine 870 may include one or more post-PEs.PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCTIn some embodiments, the post-PEs in the post-processing engine 870 may be arranged in an array that has rows and columns. In some embodiments, the post-processing engine 870 computes activation functions. The post-processing engine 870 may receive outputs of the processing engine 860 as inputs to the activation functions. In addition or alternative to activation functions, the post-processing engine 870 may perform other types of post processing on outputs of the processing engine 860. For instance, the post-processing engine 870 may apply a bias on an output of the processing engine 860. In some embodiments, the post-processing engine 870 may be bypassed for certain neural network operations.

[0091] FIG. 9 is a block diagram of an example computing device 900, in accordance with various embodiments. In some embodiments, the computing device 900 can be used as at least part of the Al device 100. A number of components are illustrated in FIG. 9 as included in the computing device 900, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 900 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 900 may not include one or more of the components illustrated in FIG. 9, but the computing device 900 may include interface circuitry for coupling to the one or more components. For example, the computing device 900 may not include a display device 906, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 906 may be coupled. In another set of examples, the computing device 900 may not include an audio input device 918 or an audio output device 908 but may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which an audio input device 918 or audio output device 908 may be coupled.

[0092] The computing device 900 may include a processing device 902 (e.g., one or more processing devices). The processing device 902 processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. The computing device 900 may include a memory 904, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid state memory, and / or a hard drive. In some embodiments, the memory 904PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT may include memory that shares a die with the processing device 902. In some embodiments, the memory 904 includes one or more non-transitory computer-readable media storing instructions executable to perform operations for executing DNNs (e.g., the method 700 described in conjunction with FIG. 7) or some operations performed by one or more components of the Al device 100. The instructions stored in the one or more non- transitory computer-readable media may be executed by the processing device 902.

[0093] In some embodiments, the computing device 900 may include a communication chip 912 (e.g., one or more communication chips). For example, the communication chip 912 may be configured for managing wireless communications for the transfer of data to and from the computing device 900. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data through the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not.

[0094] The communication chip 912 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication chip 912 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E- HSPA), or LTE network. The communication chip 912 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 912 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced CordlessTelecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, asPCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication chip 912 may operate in accordance with other wireless protocols in other embodiments. The computing device 900 may include an antenna 922 to facilitate wireless communications and / or to receive other wireless communications (such as AM or FM radio transmissions).

[0095] In some embodiments, the communication chip 912 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, the communication chip 912 may include multiple communication chips. For instance, a first communication chip 912 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication chip 912 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication chip 912 may be dedicated to wireless communications, and a second communication chip 912 may be dedicated to wired communications.

[0096] The computing device 900 may include battery / power circuitry 914. The battery / power circuitry 914 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 900 to an energy source separate from the computing device 900 (e.g., AC line power).

[0097] The computing device 900 may include a display device 906 (or corresponding interface circuitry, as discussed above). The display device 906 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.

[0098] The computing device 900 may include an audio output device 908 (or corresponding interface circuitry, as discussed above). The audio output device 908 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0099] The computing device 900 may include an audio input device 918 (or corresponding interface circuitry, as discussed above). The audio input device 918 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays,PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0100] The computing device 900 may include a GPS device 916 (or corresponding interface circuitry, as discussed above). The GPS device 916 may be in communication with a satellitebased system and may receive a location of the computing device 900, as known in the art.

[0101] The computing device 900 may include another output device 910 (or corresponding interface circuitry, as discussed above). Examples of the other output device 910 may include an audio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, or an additional storage device.

[0102] The computing device 900 may include another input device 920 (or corresponding interface circuitry, as discussed above). Examples of the other input device 920 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0103] The computing device 900 may have any desired form factor, such as a handheld or mobile computer system (e.g., a cell phone, a smart phone, a mobile internet device, a music player, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultramobile PC, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, or a wearable computer system. In some embodiments, the computing device 900 may be any other electronic device that processes data.

[0104] The following paragraphs provide various examples of the embodiments disclosed herein.

[0105] Example 1 provides a method for executing one or more neural networks, the method including assigning a first job queue to an agent, the first job queue including one or more jobs to be performed by a NPUs for executing the one or more neural networks; providing access to data of a second job queue to the agent, in which the second job queue is assigned to another agent and includes one or more other jobs to be performed by the NPUs for executing the one or more neural networks; determining, by the agent, a frequency of the NPUs based on data of the first job queue, the data of the second jobPCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT queue, and a QoS requirement forthe first job queue; and executing, by the NPUs, at least part of the one or more neural networks based on the frequency.

[0106] Example 2 provides the method of example 1, in which determining the frequency of the NPUs includes determining an amount of time during which the NPUs is active for the first job queue and the second job queue based on the data of the first job queue and the data of the second job queue; estimating a performance of the NPUs executing the first job queue and the second job queue based on the amount of time; and adjusting an initial frequency of the NPUs by determining whether the estimated performance of the NPUs meets the QoS requirement.

[0107] Example 3 provides the method of example 2, in which adjusting the initial frequency of the NPUs includes in response to determining that the estimated performance of the neural unit exceeds the QoS requirement, decreasing the initial frequency; and in response to determining that the estimated performance of the neural unit fails to meet the QoS requirement, increasing the initial frequency.

[0108] Example 4 provides the method of example 1, further including determining, by the another agent, another frequency of the NPUs based on the data of the second job queue and a QoS requirement for a priority level of the second job queue, in which the NPUs executes at least part of the one or more neural networks further based on the another frequency.

[0109] Example 5 provides the method of example 4, in which executing the one or more neural networks includes selecting a frequency from a group of frequencies that include the frequency and the another frequency; and executing, by the NPUs, the one or more neural networks at the selected frequency.

[0110] Example 6 provides the method of example 1, in which the first job queue and the second job queue are executed by the NPUs with different priorities, the different priorities corresponding to different QoS requirements for the agent and the another agent.

[0111] Example 7 provides the method of example 6, in which the different priorities are determined by a computer program running on a CPU associated with the NPUs.

[0112] Example 8 provides the method of example 1, in which the second job queue further includes an additional job, a time of the additional job is outside a time window of the first job queue, and a time of the one or more other jobs is inside the time window of the first job queue.PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT

[0113] Example 9 provides the method of example 8, in which providing access to data of a second job queue to the agent includes providing data of the one or more other jobs to the agent; and bypassing providing data of the additional job to the agent.

[0114] Example 10 provides the method of example 1, further including providing access to data of one or more other job queues to the agent, in which the one or more other job queues are assigned to one or more other agents and include additional jobs to be performed by the NPUs for executing the one or more neural networks, in which the frequency of the NPUs is determined by the agent further based on the data of the one or more other job queues.

[0115] Example 11 provides one or more non-transitory computer-readable media storing instructions executable to perform operations for executing one or more neural networks, the operations including assigning a first job queue to an agent, the first job queue including one or more jobs to be performed by a NPUs for executing the one or more neural networks; providing access to data of a second job queue to the agent, in which the second job queue is assigned to another agent and includes one or more other jobs to be performed by the NPUs for executing the one or more neural networks; determining, by the agent, a frequency of the NPUs based on data of the first job queue, the data of the second job queue, and a QoS requirement for the first job queue; and executing, by the NPUs, at least part of the one or more neural networks based on the frequency.

[0116] Example 12 provides the one or more non-transitory computer-readable media of example 11, in which determining the frequency of the NPUs includes determining an amount of time during which the NPUs is active for the first job queue and the second job queue based on the data of the first job queue and the data of the second job queue; estimating a performance of the NPUs executing the first job queue and the second job queue based on the amount of time; and adjusting an initial frequency of the NPUs by determining whether the estimated performance of the NPUs meets the QoS requirement.

[0117] Example 13 provides the one or more non-transitory computer-readable media of example 12, in which adjusting the initial frequency of the NPUs includes in response to determining that the estimated performance of the neural unit exceeds the QoS requirement, decreasing the initial frequency; and in response to determining that the estimated performance of the neural unit fails to meet the QoS requirement, increasing the initial frequency.PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT

[0118] Example 14 provides the one or more non-transitory computer-readable media of any one of examples 11-13, in which the operations further include determining, by the another agent, another frequency of the NPUs based on the data of the second job queue and a QoS requirement for a priority level of the second job queue, in which the NPUs executes at least part of the one or more neural networks further based on the another frequency.

[0119] Example 15 provides the one or more non-transitory computer-readable media of example 14, in which executing the one or more neural networks includes selecting a frequency from a group of frequencies that include the frequency and the another frequency; and executing, by the NPUs, the one or more neural networks at the selected frequency.

[0120] Example 16 provides the one or more non-transitory computer-readable media of any one of examples 11-15, in which the first job queue and the second job queue are executed by the NPUs with different priorities, the different priorities corresponding to different QoS requirements for the agent and the another agent.

[0121] Example 17 provides the one or more non-transitory computer-readable media of example 16, in which the different priorities are determined by a computer program running on a CPU associated with the NPUs.

[0122] Example 18 provides the one or more non-transitory computer-readable media of any one of examples 11-17, in which the second job queue further includes an additional job, a time of the additional job is outside a time window of the first job queue, and a time of the one or more other jobs is inside the time window of the first job queue.

[0123] Example 19 provides the one or more non-transitory computer-readable media of example 18, in which providing access to data of a second job queue to the agent includes providing data of the one or more other jobs to the agent; and bypassing providing data of the additional job to the agent.

[0124] Example 20 provides the one or more non-transitory computer-readable media of any one of examples 11-19, in which the operations further include providing access to data of one or more other job queues to the agent, in which the one or more other job queues are assigned to one or more other agents and include additional jobs to be performed by the NPUs for executing the one or more neural networks, in which the frequency of thePCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCTNPUs is determined by the agent further based on the data of the one or more other job queues.

[0125] Example 21 provides a computer system, including one or more processing units for executing computer program instructions; and one or more non-transitory computer- readable media storing computer program instructions executable by the one or more processing units to perform operations for executing one or more neural networks, the operations including assigning a first job queue to an agent, the first job queue including one or more jobs to be performed by a NPUs for executing the one or more neural networks, providing access to data of a second job queue to the agent, in which the second job queue is assigned to another agent and includes one or more other jobs to be performed by the NPUs for executing the one or more neural networks, determining, by the agent, a frequency of the NPUs based on data of the first job queue, the data of the second job queue, and a QoS requirement forthe first job queue, and executing, by the NPUs, at least part of the one or more neural networks based on the frequency.

[0126] Example 22 provides the computer system of example 21, in which determining the frequency of the NPUs includes determining an amount of time during which the NPUs is active for the first job queue and the second job queue based on the data of the first job queue and the data of the second job queue; estimating a performance of the NPUs executing the first job queue and the second job queue based on the amount of time; and adjusting an initial frequency of the NPUs by determining whether the estimated performance of the NPUs meets the QoS requirement.

[0127] Example 23 provides the computer system of example 21 or 22, in which the operations further include determining, by the another agent, another frequency of the NPUs based on the data of the second job queue and a QoS requirement for a priority level of the second job queue, in which executing the one or more neural networks includes selecting a frequency from a group of frequencies that include the frequency and the another frequency, and executing, by the NPUs, the one or more neural networks at the selected frequency.

[0128] Example 24 provides the computer system of any one of examples 21-23, in which the second job queue further includes an additional job, a time of the additional job is outside a time window of the first job queue, and a time of the one or more other jobs is inside the time window of the first job queue, in which providing access to data of a secondPCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT job queue to the agent includes providing data of the one or more other jobs to the agent; and bypassing providing data of the additional job to the agent.

[0129] Example 25 provides the computer system of any one of examples 21-24, in which the operations further include providing access to data of one or more other job queues to the agent, in which the one or more other job queues are assigned to one or more other agents and include additional jobs to be performed by the NPUs for executing the one or more neural networks, in which the frequency of the NPUs is determined by the agent further based on the data of the one or more other job queues.

[0130] The above description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art can recognize. These modifications may be made to the disclosure in light of the above detailed description.

Claims

PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCTClaims1. A method for executing one or more neural networks, the method comprising: assigning a first job queue to an agent, the first job queue comprising one or more jobs to be performed by a neural processing unit for executing the one or more neural networks; providing access to data of a second job queue to the agent, wherein the second job queue is assigned to another agent and comprises one or more other jobs to be performed by the neural processing unit for executing the one or more neural networks; determining, by the agent, a frequency of the neural processing unit based on data of the first job queue, the data of the second job queue, and a quality of service requirement for the first job queue; and executing, by the neural processing unit, at least part of the one or more neural networks based on the frequency.

2. The method of claim 1, wherein determining the frequency of the neural processing unit comprises: determining an amount of time during which the neural processing unit is active for the first job queue and the second job queue based on the data of the first job queue and the data of the second job queue; estimating a performance of the neural processing unit executing the first job queue and the second job queue based on the amount of time; and adjusting an initial frequency of the neural processing unit by determining whether the estimated performance of the neural processing unit meets the quality of service requirement.

3. The method of claim 2, wherein adjusting the initial frequency of the neural processing unit comprises: in response to determining that the estimated performance of the neural unit exceeds the quality of service requirement, decreasing the initial frequency; and in response to determining that the estimated performance of the neural unit fails to meet the quality of service requirement, increasing the initial frequency.PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT4. The method of any one of claims 1-3, further comprising: determining, by the another agent, another frequency of the neural processing unit based on the data of the second job queue and a quality of service requirement for a priority level of the second job queue, wherein the neural processing unit executes at least part of the one or more neural networks further based on the another frequency.

5. The method of claim 4, wherein executing the one or more neural networks comprises: selecting a frequency from a group of frequencies that comprise the frequency and the another frequency; and executing, by the neural processing unit, the one or more neural networks at the selected frequency.

6. The method of any one of claims 1-5, wherein the first job queue and the second job queue are executed by the neural processing unit with different priorities, the different priorities corresponding to different quality of service requirements for the agent and the another agent.

7. The method of claim 6, wherein the different priorities are determined by a computer program running on a central processing unit associated with the neural processing unit.

8. The method of any one of claims 1-7, wherein the second job queue further comprises an additional job, a time of the additional job is outside a time window of the first job queue, and a time of the one or more other jobs is inside the time window of the first job queue.

9. The method of claim 8, wherein providing access to data of a second job queue to the agent comprises: providing data of the one or more other jobs to the agent; and bypassing providing data of the additional job to the agent.

10. The method of any one of claims 1-9, further comprising:PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT providing access to data of one or more other job queues to the agent, wherein the one or more other job queues are assigned to one or more other agents and comprise additional jobs to be performed by the neural processing unit for executing the one or more neural networks, wherein the frequency of the neural processing unit is determined by the agent further based on the data of the one or more other job queues.

11. One or more non-transitory computer-readable media storing instructions executable to perform operations for executing one or more neural networks, the operations comprising: assigning a first job queue to an agent, the first job queue comprising one or more jobs to be performed by a neural processing unit for executing the one or more neural networks; providing access to data of a second job queue to the agent, wherein the second job queue is assigned to another agent and comprises one or more other jobs to be performed by the neural processing unit for executing the one or more neural networks; determining, by the agent, a frequency of the neural processing unit based on data of the first job queue, the data of the second job queue, and a quality of service requirement for the first job queue; and executing, by the neural processing unit, at least part of the one or more neural networks based on the frequency.

12. The one or more non-transitory computer-readable media of claim 11, wherein determining the frequency of the neural processing unit comprises: determining an amount of time during which the neural processing unit is active for the first job queue and the second job queue based on the data of the first job queue and the data of the second job queue; estimating a performance of the neural processing unit executing the first job queue and the second job queue based on the amount of time; and adjusting an initial frequency of the neural processing unit by determining whether the estimated performance of the neural processing unit meets the quality of service requirement.PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT13. The one or more non-transitory computer-readable media of claim 12, wherein adjusting the initial frequency of the neural processing unit comprises: in response to determining that the estimated performance of the neural unit exceeds the quality of service requirement, decreasing the initial frequency; and in response to determining that the estimated performance of the neural unit fails to meet the quality of service requirement, increasing the initial frequency.

14. The one or more non-transitory computer-readable media of any one of claims 11-13, wherein the operations further comprise: determining, by the another agent, another frequency of the neural processing unit based on the data of the second job queue and a quality of service requirement for a priority level of the second job queue, wherein the neural processing unit executes at least part of the one or more neural networks further based on the another frequency.

15. The one or more non-transitory computer-readable media of claim 14, wherein executing the one or more neural networks comprises: selecting a frequency from a group of frequencies that comprise the frequency and the another frequency; and executing, by the neural processing unit, the one or more neural networks at the selected frequency.

16. The one or more non-transitory computer-readable media of any one of claims 11-15, wherein the first job queue and the second job queue are executed by the neural processing unit with different priorities, the different priorities corresponding to different quality of service requirements for the agent and the another agent.

17. The one or more non-transitory computer-readable media of claim 16, wherein the different priorities are determined by a computer program running on a central processing unit associated with the neural processing unit.PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT18. The one or more non-transitory computer-readable media of any one of claims 11-17, wherein the second job queue further comprises an additional job, a time of the additional job is outside a time window of the first job queue, and a time of the one or more other jobs is inside the time window of the first job queue.

19. The one or more non-transitory computer-readable media of claim 18, wherein providing access to data of a second job queue to the agent comprises: providing data of the one or more other jobs to the agent; and bypassing providing data of the additional job to the agent.

20. The one or more non-transitory computer-readable media of any one of claims 11-19, wherein the operations further comprise: providing access to data of one or more other job queues to the agent, wherein the one or more other job queues are assigned to one or more other agents and comprise additional jobs to be performed by the neural processing unit for executing the one or more neural networks, wherein the frequency of the neural processing unit is determined by the agent further based on the data of the one or more other job queues.

21. A computer system, comprising: one or more processing units for executing computer program instructions; and one or more non-transitory computer-readable media storing computer program instructions executable by the one or more processing units to perform operations for executing one or more neural networks, the operations comprising: assigning a first job queue to an agent, the first job queue comprising one or more jobs to be performed by a neural processing unit for executing the one or more neural networks, providing access to data of a second job queue to the agent, wherein the second job queue is assigned to another agent and comprises one or more other jobs to be performed by the neural processing unit for executing the one or more neural networks,PCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT determining, by the agent, a frequency of the neural processing unit based on data of the first job queue, the data of the second job queue, and a quality of service requirement for the first job queue, and executing, by the neural processing unit, at least part of the one or more neural networks based on the frequency.

22. The computer system of claim 21, wherein determining the frequency of the neural processing unit comprises: determining an amount of time during which the neural processing unit is active for the first job queue and the second job queue based on the data of the first job queue and the data of the second job queue; estimating a performance of the neural processing unit executing the first job queue and the second job queue based on the amount of time; and adjusting an initial frequency of the neural processing unit by determining whether the estimated performance of the neural processing unit meets the quality of service requirement.

23. The computer system of claim 21 or 22, wherein the operations further comprise: determining, by the another agent, another frequency of the neural processing unit based on the data of the second job queue and a quality of service requirement for a priority level of the second job queue, wherein executing the one or more neural networks comprises: selecting a frequency from a group of frequencies that comprise the frequency and the another frequency, and executing, by the neural processing unit, the one or more neural networks at the selected frequency.

24. The computer system of any one of claims 21-23, wherein the second job queue further comprises an additional job, a time of the additional job is outside a time window of the first job queue, and a time of the one or more other jobs is inside the time window of the first job queue, wherein providing access to data of a second job queue to the agent comprises: providing data of the one or more other jobs to the agent; andPCT / US25 / 19167 10 March 2025 (10.03.2025)AG3648-PCT bypassing providing data of the additional job to the agent.

25. The computer system of any one of claims 21-24, wherein the operations further comprise: providing access to data of one or more other job queues to the agent, wherein the one or more other job queues are assigned to one or more other agents and comprise additional jobs to be performed by the neural processing unit for executing the one or more neural networks, wherein the frequency of the neural processing unit is determined by the agent further based on the data of the one or more other job queues.

Images