Provisioned power optimization for data centers

Abstract

This disclosure is directed to optimizing the amount of power provisioned to a data center. The system can obtain, at a first time window, electric waveform data measured for a rack cabinet in the data center, the rack cabinet comprising shelves, each shelf comprising processing unit(s). The system can identify, for the first time window, an amount of power provisioned to the data center and used by the rack cabinets. The system can predict, via executing an artificial intelligence model, a first amount of power for a processing operation by the processing unit(s) of the rack cabinet and a second amount of power for processing operations by additional rack(s). The system can determine a third amount of power for provisioning the processing operation at the second time window. The system can adjust, based on the third amount of power, parameter(s) associated with the rack cabinet at the second time window.

Description

Docket No. 102555-0261PROVISIONED POWER OPTIMIZATION FOR DATA CENTERS CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to and the benefit of U.S. Provisional Application Number 63 / 773,072, filed March 17, 2025, U.S. Provisional Application Number 63 / 824,725, filed June 16, 2025, and U.S. Provisional Application Number 63 / 845,589 filed July 17, 2025. Each of the foregoing applications are incorporated herein by reference in their entireties for all purposes.FIELD OF THE DISCLOSURE

[0002] This disclosure relates generally to systems and methods for provisioned power optimization.BACKGROUND

[0003] Utility distribution grids can generate and distribute electric power to various devices on the edge of the distribution grid, including to electric consumption entities. The utility distribution grids can supply power via transmission or distribution lines to various loads at the edge of the grid, such as consumer electric devices or residential charging infrastructures. The utility distribution grids can use meters to observe or measure utility delivery or consumption in the grid.BRIEF SUMMARY OF THE DISCLOSURE

[0004] Data centers, and the servers, processors, memory, network, and cooling systems housed therein, can consume significant amounts of power, both to perform computing tasks as well as to control the climate of the data center. A certain amount of power can be provisioned for use by the data center. For example, a maximum amount of power can be allocated by the utility service provider for use by the data center, including to power the servers as well as maintain the climate within the data center. However, it can be technically challenging for data centers to maximize or otherwise optimize the use of the power that has been provisioned to the data center in an efficient, effective, and reliable manner, thereby resulting in reduced computing capacity for the data center.14913-1418-6318.1Docket No. 102555-0261

[0005] The power usage effectiveness (PUE) can refer to the ratio of the total amount of energy used by a data center facility relative to the energy delivered to computing equipment located within the data center facility. For example, a PUE of 1.0 can indicate that all of the energy delivered to the data center facility is used by the computing equipment. However, this may not be possible due to the energy used to maintain the climate within the data center facility, for example. Further, in some cases, additional power may be provisioned to the data center but not used by the data center. The additional provisioned power (e.g., unused provisioned power) may be unaccounted for or may not be measured when using the PUE. Failure to utilize the power that has been provisioned or allocated to the data center may unnecessarily limit or reduce the computing capacity of the data center, thereby reducing the performance of the data center.

[0006] Aspects of the technical solutions disclosed herein can manage and optimize power utilization across a facility (e.g., data center), to facilitate the increased, optimal, or full utilization of provisioned power for compute operations. This technology can facilitate increasing or maximizing the use of the total amount of power, which can refer to the interconnected power, that is used for computation at a data center facility, thereby improving the use of resources that have already been allocated to the data center.

[0007] In some cases, an escalating demand for computational power, such as driven by artificial intelligence (Al) workloads (e.g., growth of Al and machine learning applications), may pose challenges to data centers due to a surge in demands. The increasing prevalence of Large Language Models (LLMs) and graphics processing unit (GPU)-intensive workloads (or other computing workloads) may introduce challenges in power management and strain existing infrastructure, resulting in potential restrictions in power provisioning and utilization. Power consumption may be a bottleneck in the continued expansion of data centers. Certain power management strategies or techniques may focus on static provisioning and reliability, which may result in underutilization of available power capacity.

[0008] Compared to other systems (e.g., central processing unit (CPU)-based systems), GPUs can exhibit rapid and excessive power fluctuations, potentially rendering traditional static power provisioning techniques ineffective. The unpredictable load patterns of GPU tasks may necessitate adaptive and dynamic power management techniques. Additionally, millisecond-level responsiveness may be desired for effective management, exceeding the capabilities of conventional CPU-centric systems, for example. Certain techniques for power24913-1418-6318.1Docket No. 102555-0261management in data centers, such as CPU-based oversubscription, may be inadequate for handling the dynamic demands of GPU-intensive (e.g., GPU-based) applications. For instance, GPUs may lack the detailed power telemetry and control mechanisms found in CPUs. In-band power management methods for GPUs may have limited resolution and documentation, while out-of-band mechanisms may be relatively slow (e.g., unresponsive or delay) and unreliable, hindering safe power throttling. Aspects of the technical solutions disclosed herein provide techniques for optimizing data center power utilization, with AL driven methodologies for monitoring and controlling power distribution.

[0009] Certain metrics, such as PUE, may focus on the proportion of power used for computation which may potentially overlook aspects of unused provisioned power.Addressing the gap introduced by such metrics can be desired, to minimize or avoid power provisioning being a constraint in data center construction and operation. Aspects of the technical solutions can provide a distribution controls for power systems optimization approach to maximize the utilization of interconnected power resources. Aspects of the technical solutions can include Al-driven methodologies for monitoring and optimizing power utilization within data center environments. By leveraging real-time data analysis, dynamic adjustments of operational parameters of GPU (among other components), and predictive modeling, power efficiency can be improved and resource allocation can be achieved. Aspects of the technical solutions can provide distributed control and machine learning techniques, including Kubemetes scheduling, to maximize the use of provisioned power, improving metrics such as the PUE.

[0010] Aspects of the technical solutions discussed herein can provide a data processing system for optimizing power utilization, such as by a data center. The systems and methods can include various components or functions, such as at least one or a combination of system-wide objective functions, edge-based measurement and control nodes, or a hierarchical communication system. The optimization of the power utilization can allow for more compute operations to be performed at the data center. For example, the systems and methods can involve the development of the system-wide objective function that, when optimized, allows for an overall increase in the overall compute utilization. The systems and methods can involve the edge-based measurement and control nodes that can implement system controls, for instance, to achieve a desired optimization objective by managing compute power-consuming loads or other types of power consuming loads, such as compute hardware.34913-1418-6318.1Docket No. 102555-0261The systems and methods can involve the hierarchical or fully-connected communication system that can allow for the subdivision of the system-wide objective into relatively smaller optimizations, which can be implemented by the edge nodes.

[0011] The systems and methods can utilize machine learning to adapt control responses to local conditions, optimize the system, or make predictions for system optimization. With the implementation of the features or functionalities discussed herein, data centers or other infrastructures can at least one of achieve higher average power utilization, reduce load variability, or balance power between loads and energy storage systems. Optimizing the utilization of provisioned power can improve the compute capacity and operational efficiency of data centers.

[0012] The systems and methods can utilize an Al-driven approach to optimize data center power utilization, for instance, utilizing the data processing system. The data processing system can include a custom GPU which allows for real-time analysis and forecasting leveraging the underlying GPU architecture. The data processing system can include features or functionalities to perform operations, such as real-time waveform analysis, dynamic parameter adjustment, predictive modeling, distributed control approach, or Kubemetes scheduling, among others. The systems and methods can utilize the data processing system to reduce load variability, allow for power balancing between loads, and optimize the total load with the potential inclusion of energy storage. The systems and methods can employ machine learning techniques to adapt control responses to local conditions, optimize the system, and perform predictions for system optimization.

[0013] In an aspect, this disclosure is directed to a method for monitoring and optimizing power consumption. The method can include analyzing electric waveform data in real-time using artificial intelligence (Al)-based models. The electric waveform data can include information about characteristics of electricity (e.g., voltage or current) at a power supply, rack level, cluster level, data center, or on the distribution grid. The method can include dynamically adjusting operational parameters of a processing unit including frequency and power levels based on the analyzed electric waveform data using the Al-based models. The method can include iteratively training the Al-based models and implementing real-time control actions to improve power efficiency.44913-1418-6318.1Docket No. 102555-0261

[0014] The processing unit can be a graphics processing unit (GPU) or a central processing unit (CPU). The method can include coordinating power optimization strategies across multiple servers, racks, or physical computing complexes within a data center. The method can include employing the processing unit to perform advanced waveform analysis to derive actionable insights and optimize power usage.

[0015] The processing unit performing the advanced waveform analysis can be integrated internally within computing systems or externally attached to power infrastructure components, including one or more of at least power distribution units (PDU), uninterruptible power supplies (UPS), generators, automatic transfer switches (ATS), or circuit breakers. The method can include analyzing data flows directly received by computing at least the processing unit to enhance accuracy and effectiveness of power consumption monitoring and optimization.

[0016] The Al-based models can incorporate data from cooling systems, workloads, and environmental conditions to enhance accuracy in power consumption predictions and minimize latency. The method can include monitoring voltage and current across a plurality of processing units. The method can include facilitating data exchange through one or more high-speed communication interfaces. The one or more high-speed communication interfaces can use or be configured with a computer networking communication standard that facilitates high-performance, high throughput, and low latency. The high-speed communication interface (e.g., Infiniband) can use a direct or switched interconnect between servers and storage systems, an interconnect between storage systems, or use a switched fabric network topology, while being scalable.

[0017] In some aspects, the disclosure is directed to a system, a method, or a non-transitory computer-readable medium, comprising one or more processors to provision power used by rack cabinets in a data center. The one or more processors can obtain, at a first time window, electric waveform data measured for a rack cabinet in the data center, the rack cabinet comprising a plurality of shelves, each of the plurality of shelves comprising a respective one or more processing units. The one or more processors can identify, for the first time window, an amount of power provisioned to the data center and used by the rack cabinets. The one or more processors can predict, via executing an artificial intelligence model using (i) at least one characteristic of operation executed on the one or more processing units, (ii) the electric waveform data, and (iii) the amount of power, for a second54913-1418-6318.1Docket No. 102555-0261time window subsequent to the first time window, a first amount of power for a processing operation by the one or more processing units of the rack cabinet and a second amount of power for processing operations by one or more additional racks of the data center. The one or more processors can determine, based on the first amount of power and the second amount of power from the artificial intelligence model, a third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet at the second time window. The one or more processors can adjust, based on the third amount of power, one or more parameters associated with the rack cabinet at the second time window.

[0018] The one or more processors can monitor, via one or more measurement circuits disposed at a power distribution unit electrically coupled to the rack cabinet, voltage waveform data or current waveform data associated with electric power delivered to the plurality of shelves of the rack cabinet. The one or more processors can monitor, via one or more measurement circuits disposed at a power supply unit electrically coupled to the rack cabinet, voltage waveform data or current waveform data associated with electric power delivered to the plurality of shelves of the rack cabinet.

[0019] The one or more processors can convert the electric waveform data to at least one of a quantitative value representing an amount of power consumed at the rack cabinet, a load variability, or an output from a fast Fourier transform (FFT) at the first time window. The one or more processors can monitor the amount of power at a metering device electrically coupled with the rack cabinets used in the data center, wherein the amount of power represents a total power available for use by processing units of the rack cabinets for the processing operations, and wherein the amount of power is affected by at least one of time of day, weather condition, operating conditions at an electricity distribution grid, or power allocation limits supported by the data center.

[0020] The at least one characteristic of operation can comprise priorities associated with the respective rack cabinets. The one or more processors can predict, via executing the artificial intelligence model using (i) the priorities associated with the respective rack cabinets, (ii) the electric waveform data, and (iii) the amount of power, for the second time window, the first amount of power for the processing operation by the one or more processing units of the rack cabinet and the second amount of power for processing operations by the one or more additional racks of the data center.64913-1418-6318.1Docket No. 102555-0261

[0021] The at least one characteristic of operation can comprise a Service Level Agreement (SLA) associated with at least the one or more processing units, the SLA representing a respective allocation of compute resources supported by the respective one or more processing units. The one or more processors can predict, via executing the artificial intelligence model using (i) the SLA associated with at least the one or more processing units, (ii) the electric waveform data, and (iii) the amount of power, for the second time window, the first amount of power for the processing operation by the one or more processing units of the rack cabinet and the second amount of power for processing operations by the one or more additional racks of the data center.

[0022] The at least one characteristic of operation can comprise a data flow directly received by the one or more processing units, the data flow indicative of computation resources to be used by the one or more processing units for the processing operation in the second time window. The one or more processors can detect that the first amount of power exceeds a subset of the amount of power provisioned to the rack cabinet of the data center. The one or more processors can determine, responsive to detection of the first amount of power exceeds the subset of the amount of power, via executing the artificial intelligence model using the first amount of power and the second amount of power, the third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet such that the first amount of power predicted for the processing operation by the one or more processing units of the rack cabinet is reduced below the subset of the amount of power at the second time window.

[0023] The one or more processors can detect that the second amount of power exceeds a subset of the amount of power provisioned to the one or more additional racks of the data center. The one or more processors can determine, responsive to detection of the second amount of power exceeds the subset of the amount of power, via executing the artificial intelligence model using the first amount of power and the second amount of power, the third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet such that the second amount of power predicted for the processing operations by the one or more additional racks of the data center is reduced below the subset of the amount of power at the second time window.

[0024] The one or more processors can increase a current amount of power provisioned to the one or more processing units of the rack cabinet for the second time window. The one74913-1418-6318.1Docket No. 102555-0261or more processors can increase an allocation of compute resources for the processing operation by the one or more processing units for the second time window. The one or more processors can increase at least one of a frequency of the one or more processing units, a voltage of the one or more processing units, a power limit of the one or more processing units, a frequency of a memory in electrical communication with the one or more processing units, or a capacity of the memory.

[0025] The one or more processors can decrease an allocation of compute resources for the processing operations by the one or more additional racks of the data center for the second time window. The one or more processors can adjust, based on the third amount of power, scheduling of workload across the rack cabinets for the one or more processing units of the rack cabinet to operate at the third amount of power.

[0026] The one or more processors can determine, based on the first amount of power and the second amount of power from the artificial intelligence model, and the at least one characteristic of operation comprising a Service Level Agreement (SLA) associated with at least the one or more processing units, the third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet at the second time window. The one or more processors can adjust, at the second time window, the one or more parameters to increase a current amount of power for the rack cabinet to the third amount of power for supporting a higher allocation of compute resources relative to the one or more additional racks of the data center based on the SLA.

[0027] The one or more processing units can comprise at least one of graphics processing units (GPUs) or central processing units (CPUs) for the processing operation. The one or more processors can obtain, at the second time window, second electric waveform data measured for the rack cabinet. The one or more processors can update the artificial intelligence model using at least the first amount of power and the second amount of power predicted for the second time window, the second electric waveform data obtained at the second time window, and the one or more parameters associated with the rack cabinet adjusted at the second time window. The one or more processors can deploy the artificial intelligence model subsequent to updating for executing a prediction for a third time window subsequent to the second time window.84913-1418-6318.1Docket No. 102555-0261

[0028] In some aspects, the disclosure is directed to a system, a method, or a non-transitory computer-readable medium, comprising one or more processors to manage controller state. The one or more processors can monitor, via one or more measurement circuits, electric waveform data for electric power delivered to a server on a shelf in a rack cabinet. The one or more processors can aggregate the electric waveform data to estimate a total power for a control iteration. The one or more processors can identify a power envelope value for the control iteration. The one or more processors can determine, based on the total power and the power envelope value, one or more state variables comprising a headroom value. The one or more processors can select a controller state based on the one or more state variables and the headroom value. The one or more processors can compute a second power limit to request for one or more processing units of the server based on the controller state, the total power, the power envelope value, and a current power limit of the one or more processing units. The one or more processors can generate an instruction to update the current power limit of the one or more processing units based on the second power limit.

[0029] The one or more processors can determine the headroom value based on a difference between the power envelope value and the total power, the headroom value indicating whether the total power satisfies the power envelope value or is predicted to exceed the power envelope value. The one or more processors can determine the one or more state variables comprising at least one of a data integrity flag, a moving average of the total power, a rate of change of the total power, or a time to excursion.

[0030] The one or more processors can select the controller state from at least one of an idle state, a ramp up state, a regulate state, a ramp down state, or an excursion state. The controller state can be a ramp up state, and the one or more processors can determine, based on the ramp up state and the current power limit of the one or more processing units of the server, a rate of increase from the current power limit to the second power limit to request for the one or more processing units.

[0031] The controller state can be one of a regulate state, a ramp down state, or an excursion state, and the one or more processors can determine, based on the controller state and the current power limit of the one or more processing units of the server, the second power limit using a proportional derivative control based on a measured power, a margin, and the power envelope value. The one or more processors can clip the second power limit to a range. The one or more processors can apply a cooldown between commands to adjust power94913-1418-6318.1Docket No. 102555-0261limits, wherein the cooldown is between transmission of the instruction and at least one subsequent instruction.

[0032] The one or more processors can cache a latest power sample from each of a plurality of power sources associated with the server. Responsive to receipt of updated samples from the plurality of power sources, the one or more processors can (i) estimate the total power as a sum of latest power samples of the plurality of power sources, and (ii) initiate the control iteration.

[0033] The one or more processors can quantize the second power limit to discrete increments prior to generation of the instruction to update the current power limit. The one or more processors can apply a cooldown period between instructions to update the current power limit. Responsive to the controller state being one of an excursion state or a ramp down state, the one or more processors can generate the instruction during the cooldown period.

[0034] The one or more processors can apply a cooldown between commands to adjust power limits. The one or more processors can select a second controller state in a second control iteration based on the one or more state variables and the headroom value, the second controller state is one of a ramp down state or an excursion state. The one or more processors can generate, during the cooldown, a second instruction to update the second power limit of the one or more processing units based on a third power limit compute for the second control iteration.

[0035] The controller state can be one of a ramp down state or an excursion state, and the one or more processors can generate the instruction to reduce the current power limit of the one or more processing units to the second power limit such that a measured power returns to the power envelope value minus a margin within a predefined time window.

[0036] In some aspects, the disclosure is directed to a system, a method, or a non-transitory computer-readable medium, comprising one or more processors to bridge power across a power chain of a data center and power of one or more processing units. The one or more processors can receive first electric waveform data measured at a plurality of devices distributed across a power chain of a data center. The one or more processors can receive second electric waveform data measured for a rack cabinet of rack cabinets in the data center, the rack cabinet comprising a plurality of shelves, each of the plurality of shelves comprising 104913-1418-6318.1Docket No. 102555-0261a respective one or more processing units, wherein the second electric waveform data is indicative of power consumed for processing operations by the respective one or more processing units across the plurality of shelves of the rack cabinet. The one or more processors can correlate the first electric waveform data measured at the plurality of devices and the second electric waveform data measured for the rack cabinet. The one or more processors can detect at least one power condition based on correlation of the first electric waveform data and the second electric waveform data. The one or more processors can initiate one or more actions for at least one of the plurality of devices or at least one of the one or more processing units according to the at least one power condition.

[0037] The first electric waveform data can comprise alternating current (AC) data and the second electric waveform data can comprise direct current (DC) data. The plurality of devices distributed across the power chain can comprise at least one of a utility feed, an automatic transfer switch (ATS), an uninterruptible power supply (UPS), a generator, a circuit breaker, a power distribution unit (PDU), a power supply unit (PSU), or one or more of the rack cabinets.

[0038] The one or more processors can compute at least one root mean square (RMS) value using one or more functions on the second electric waveform data measured for the rack cabinet. The one or more processors can determine, based on the at least one RMS value, the power consumed for the processing operations by the respective one or more processing units. The second electric waveform data can be measured at one or more of (i) one or more power conversion stages of a power supply unit (PSU) supplying power to the rack cabinet, or (ii) at least one power input port associated with the one or more processing units.

[0039] The one or more processors can align timestamps between the first electric waveform data and the second electric waveform data. The one or more processors can determine a relationship between changes in the first electric waveform data and changes in the second electric waveform data in a same time window to correlate the first electric waveform data and the second electric waveform data.

[0040] The one or more processors can detect that changes in the first electric waveform data are caused by changes in the second electric waveform data. The one or more processors can adjust, responsive to the detection, one or more parameters associated with at least one of114913-1418-6318.1Docket No. 102555-0261the one or more processing units, the one or more parameters comprising at least one of a frequency of the one or more processing units, a voltage of the one or more processing units, a power limit of the one or more processing units, or a scheduling of compute resources for the processing operations of the one or more processing units.

[0041] The one or more processors can detect that changes in the first electric waveform data are independent from changes in the second electric waveform data. Responsive to the detection, The one or more processors can one of: adjust first one or more parameters associated with at least one of the plurality of devices based on the at least one power condition detected from the first electric waveform data; or adjust second one or more parameters associated with at least one of the one or more processing units based on the at least one power condition detected from the second electric waveform data.

[0042] The at least one power condition can comprise at least one of: (i) a load variability being greater than or equal to a threshold, the load variability corresponding to a difference between a maximum power and an average power over a time window, (ii) a fluctuation in power at or above a magnitude threshold or rate-of-change threshold, (iii) an amount of power available for consumption is less than a threshold based on a difference between a first amount of power provisioned for the rack cabinet and a second amount of power consumed for the processing operations by the one or more processing units of the rack cabinet, or (iv) an indication that an amount of power being consumed has increased over a threshold relative to an amount of power provisioned for at least one of the rack cabinets of the data center, a subset of the rack cabinets of the data center, the rack cabinet, at least one of the plurality of shelves, or the one or more processing units of at least one of the plurality of shelves.

[0043] The one or more processors can select the at least one of the plurality of devices. The one or more processors can generate a signal comprising one or more parameters associated with the at least one of the plurality of devices according to the at least one power condition. The one or more processors can transmit the signal to the at least one of the plurality of devices to adjust the one or more parameters. The one or more parameters can comprise: (i) a target power level for power delivery by a power distribution unit (PDU) to one or more loads, (ii) an output power limit for one or more output stages of the PDU, (iii) an enable state, a disable state, or a switching state for at least one output stage of the PDU, (iv) a configuration to control an allocation of available power from the PDU among a124913-1418-6318.1Docket No. 102555-0261plurality of loads powered via the PDU, or (v) a setpoint of power level for one or more cooling systems of the data center.

[0044] The one or more processors can generate a signal comprising one or more parameters associated with the one or more processing units according to the at least one power condition. The one or more processors can transmit the signal to the one or more processing units to adjust the one or more parameters, the one or more parameters comprising at least one of a frequency of a processing unit, a frequency of a memory in electrical communication with the processing unit, a voltage of the processing unit, a power limit of the processing unit, a capacity of the memory, or a scheduling of compute resources for the processing operations of the one or more processing units.

[0045] The one or more processors can initiate the one or more actions responsive to the at least one power condition satisfying at least one trigger criterion to trigger initiation of the one or more actions. The at least one trigger criterion can comprise at least one of: (i) a first amount of power consumed across the power chain of the data center by the rack cabinets being greater than or equal to a first threshold, the first threshold based on an amount of power provisioned to the data center, (ii) a second amount of power consumed for the processing operations by the respective one or more processing units of the rack cabinet being greater than or equal to a second threshold, the second amount of power computed from the second electric waveform data, and the second threshold based on an amount of power provisioned to the rack cabinet, (iii) a first amount of provisioned power available for consumption being less than or equal to a third threshold, or (iv) a second amount of provisioned power available for consumption being greater than or equal to a fourth threshold.

[0046] The one or more processors can receive feedback signals from at least one of the plurality of devices or at least one of the one or more processing units subsequent to initiating the one or more actions. The one or more processors can dynamically adjust the one or more actions according to the feedback signals.

[0047] In some aspects, the disclosure is directed to a system, a method, or a non-transitory computer-readable medium, comprising one or more processors to forecast power consumption. The one or more processors can obtain electric waveform data for electric power delivered to a server on a shelf in a rack cabinet. The one or more processors can134913-1418-6318.1Docket No. 102555-0261compute, from the electric waveform data, metrics over a plurality of time windows. The one or more processors can execute a first neural network using the metrics over the plurality of time windows as input. The one or more processors can select a first subset of the metrics corresponding to a first time window of the plurality of time windows. The one or more processors can execute a second neural network using the selected first subset of the metrics over the first time window of the plurality of time windows as input. The one or more processors can execute a third neural network using a first output from the first neural network and a second output from the second neural network as input. The one or more processors can generate, based on a third output from the third neural network, forecast values representing power consumption by one or more processing units of the server over a second time window subsequent to the plurality of time windows. The one or more processors can adjust, for the second time window, one or more parameters of the one or more processing units according to the forecast values.

[0048] The electric waveform data can comprise alternating current waveform data. The rack cabinet can comprise a plurality of shelves including the shelf, each of the plurality of shelves associated with a respective server. Each of the plurality of time windows can comprise a window length corresponding to at least one cycle of the electric waveform data or a portion of a cycle of the electric waveform data.

[0049] The plurality of time windows can have a first predefined length and the first time window have a second predefined length, the first predefined length corresponds to multiples of the second predefined length. The second time window can have the second predefined length. The metrics can comprise at least one of root mean square (RMS) values or power values computed from the electric waveform data.

[0050] The first time window can correspond to a latest one of the plurality of time windows. The first neural network can comprise a convolutional neural network (CNN). The second neural network can comprise a first linear neural network. The third neural network can comprise a second linear neural network.

[0051] The first neural network can process the metrics over the plurality of time windows using a one-dimensional convolution technique to generate the first output comprising a first vector representing first one or more characteristics of power consumption over the first time window. The second neural network can process the first subset of the144913-1418-6318.1Docket No. 102555-0261metrics using linear transformations to generate the second output comprising a second vector representing second one or more characteristics of power consumption over the plurality of time windows. The third neural network can process the first output and the second output using linear transformations to generate the third output comprising a vector representing a combination of first one or more characteristics of power consumption over the first time window and second one or more characteristics of power consumption over the plurality of time windows.

[0052] The one or more processors can execute a fourth neural network using a third output from the third neural network as input, the fourth neural network comprising a decoder neural network. The one or more processors can generate, responsive to execution of the fourth neural network, the forecast values of power consumption as a fourth output from the fourth neural network.

[0053] The one or more processors can compute, from the electric waveform data, second metrics over the second time window. The one or more processors can train at least one of the first neural network, the second neural network, or the third neural network using the second metrics and the forecast values from the second time window, wherein one or more weights are applied to differences between the second metrics and the forecast values based on (i) respective magnitude of the differences and (ii) respective forecast values being greater than or less than the second metrics. The metrics over the plurality of time windows can be stored in the memory. The one or more processors can compute, from the electrical waveform data, second metrics over the second time window. The one or more processors can store the second metrics of the second time window in the memory. The one or more processors can discard a portion of the metrics associated with an earliest one of the plurality of time windows such that the second time window is a latest one of the plurality of time windows.

[0054] In some aspects, the disclosure is directed to a system, a method, or a non-transitory computer-readable medium, comprising one or more processors to manage power across rack cabinets. The one or more processors can monitor, via one or more measurement circuits, a plurality of electric waveform data measured for the rack cabinets in the data center. The one or more processors can identify a first subset of power provisioned to a first rack cabinet and a second subset of power provisioned to a second rack cabinet, wherein the first subset of power and the second subset of power correspond to at least a portion of an amount of power provisioned to the data center and used by the rack cabinets. The one or154913-1418-6318.1Docket No. 102555-0261more processors can determine, based on first electric waveform data of the plurality of electric waveform data measured for the first rack cabinet and the first subset of power, a first amount of power available for consumption at the first rack cabinet. The one or more processors can determine, based on second electric waveform data of the plurality of electric waveform data measured for the second rack cabinet and the second subset of power, a second amount of power available for consumption at the second rack cabinet. The one or more processors can adjust one or more parameters associated with the first rack cabinet or the second rack cabinet based on a difference between the first amount of power that is available and the second amount of power that is available being greater than or equal to a threshold.

[0055] The first rack cabinet can comprise a first plurality of shelves, each of the first plurality of shelves comprising a respective first one or more processing units. The second rack cabinet can comprise a second plurality of shelves, each of the second plurality of shelves comprising a respective second one or more processing units.

[0056] The one or more processors can obtain, via the one or more measurement circuits disposed at a power distribution unit electrically coupled to the rack cabinets, voltage waveform data or current waveform data associated with electric power delivered to a plurality of shelves of the rack cabinets. The one or more processors can obtain, via the one or more measurement circuits disposed at power supply units electrically coupled to the rack cabinets, voltage waveform data or current waveform data associated with electric power delivered to a plurality of shelves of the rack cabinets.

[0057] The one or more processors can determine, based on the first electric waveform data, a third amount of power consumed for a first processing operation of the first one or more processing units. The one or more processors can determine, based on the second electric waveform data, a fourth amount of power consumed for a second processing operation of the second one or more processing units. The one or more processors can determine the first amount of power available for consumption at the first rack cabinet based on a difference between the first subset of power and the third amount of power. The one or more processors can determine the second amount of power available for consumption at the second rack cabinet based on a difference between the second subset of power and the fourth amount of power.164913-1418-6318.1Docket No. 102555-0261

[0058] The one or more processors can generate a signal comprising an indication to update a power limit of one or more processing units of the first rack or the second rack based on the difference between the first amount of power that is available and the second amount of power that is available being greater than or equal to the threshold. The one or more processors can, based on (i) the difference being greater than or equal to the threshold, and (ii) the first amount of power available for consumption being greater than the second amount of power available for consumption, increase at least one of a frequency of one or more processing units of the first rack cabinet, a voltage of the one or more processing units, a power limit of the one or more processing units, a frequency of a memory in electrical communication with the one or more processing units, or a capacity of the memory.

[0059] The one or more processors can, based on (i) the difference being greater than or equal to the threshold, and (ii) the first amount of power available for consumption being greater than the second amount of power available for consumption, decrease at least one of a frequency of one or more processing units of the second rack cabinet, a voltage of the one or more processing units, a power limit of the one or more processing units, a frequency of a memory in electrical communication with the one or more processing units, or a capacity of the memory. The one or more processors can, based on (i) the difference being greater than or equal to the threshold, and (ii) the first amount of power available for consumption being less than the second amount of power available for consumption, decrease at least one of a frequency of one or more processing units of the first rack cabinet, a voltage of the one or more processing units, a power limit of the one or more processing units, a frequency of a memory in electrical communication with the one or more processing units, or a capacity of the memory.

[0060] The one or more processors can, based on (i) the difference being greater than or equal to the threshold, and (ii) the first amount of power available for consumption being less than the second amount of power available for consumption, increase at least one of a frequency of one or more processing units of the second rack cabinet, a voltage of the one or more processing units, a power limit of the one or more processing units, a frequency of a memory in electrical communication with the one or more processing units, or a capacity of the memory.

[0061] The one or more processors can reallocate, based on (i) the difference being greater than or equal to the threshold, and (ii) the first amount of power available for174913-1418-6318.1Docket No. 102555-0261consumption being greater than the second amount of power available for consumption, compute resources from the second rack cabinet to the first rack cabinet for a processing operation by one or more processing units of the first rack cabinet. The one or more processors can reallocate, based on (i) the difference being greater than or equal to the threshold, and (ii) the second amount of power available for consumption being greater than the first amount of power available for consumption, compute resources from the first rack cabinet to the second rack cabinet for a processing operation by one or more processing units of the second rack cabinet.

[0062] The one or more processors can adjust the one or more parameters associated with the first rack cabinet or the second rack cabinet based on (i) the difference between the first amount of power that is available and the second amount of power that is available being greater than or equal to the threshold, and (ii) a third amount of power consumed for a first processing operation of the first rack cabinet and a fourth amount of power consumed for a second processing operation of the second rack cabinet being greater than or equal to a second threshold.

[0063] The one or more processors can detect, at a second time window after adjustment of the one or more parameters in the first time window, that the difference between the first amount of power that is available and the second amount of power that is available is below the threshold. The one or more processors can maintain the one or more parameters during the second time window. The one or more processors can detect, at a third time window subsequent to the second time window, that the difference between the first amount of power that is available and the second amount of power that is available is greater than or equal to the threshold. The one or more processors can adjust, at the third time window, the one or more parameters associated with the first rack cabinet or the second rack cabinet based on the difference being greater than or equal to the threshold.

[0064] The first electric waveform data and the second electric waveform data can be obtained at a first time window. The one or more processors can predict, for a second time window based on at least the first electric waveform data and the second electric waveform data, a third amount of power consumed for a first processing operation by first one or more processing units of the first rack cabinet and a fourth amount of power consumed for a second processing operation by second one or more processing units of the second rack cabinet. The one or more processors can determine, based on the third amount of power and the first184913-1418-6318.1Docket No. 102555-0261subset of power, the first amount of power available for consumption at the first rack cabinet for the second time window. The one or more processors can determine, based on the fourth amount of power and the second subset of power, the second amount of power available for consumption at the second rack cabinet for the second time window. The one or more processors can adjust, for the second time window, the one or more parameters associated with the first rack cabinet or the second rack cabinet based on the difference between the first amount of power that is available and the second amount of power that is available being greater than or equal to the threshold.

[0065] The one or more processors can identify a third subset of power provisioned to a third rack cabinet of the amount of power, the third subset of power corresponding to another portion of the amount of power provisioned to the data center and used by the rack cabinets. The one or more processors can determine, based on third electric waveform data of the plurality of electric waveform data measured for the third rack cabinet and the third subset of power, a third amount of power available for consumption at the third rack cabinet. The one or more processors can determine a second difference between the third amount of power that is available and each of the first amount of power that is available and the second amount of power that is available. The one or more processors can adjust the one or more parameters associated with the third cabinet and at least one the first rack cabinet or the second rack cabinet based on the second difference being greater than or equal to the threshold.

[0066] The one or more processors can maintain the one or more parameters after adjustment for a predefined time window or until satisfying a criterion. The one or more processors can revert the one or more parameters to one or more values before adjustment subsequent to the predefined time window or satisfying the criterion.

[0067] The one or more processors can adjust a second one or more parameters associated with an energy storage device based on the first amount of power that is available and the second amount of power that is available being greater than or equal to a second threshold or less than or equal to a third threshold. The energy storage device can (i) supply a third amount of power for consumption at the first rack cabinet or the second rack cabinet based on the first amount of power that is available and the second amount of power that is available being less than or equal to the second threshold, or (ii) receive a fourth amount of power, that is unused at the first rack cabinet and the second rack cabinet, for storage during a194913-1418-6318.1Docket No. 102555-0261time window when the first amount of power that is available and the second amount of power that is available being greater than or equal to the third threshold.

[0068] These and other aspects and implementations are discussed in detail below. The foregoing information and the following detailed description include illustrative examples of various aspects and implementations, and provide an overview or framework for understanding the nature and character of the claimed aspects and implementations. The drawings provide illustration and a further understanding of the various aspects and implementations, and are incorporated in and constitute a part of this specification.BRIEF DESCRIPTION OF THE FIGURES

[0069] The accompanying drawings are not intended to be drawn to scale. Like reference numbers and designations in the various drawings indicate like elements having similar structure or functionality. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:

[0070] FIG. l is a block diagram depicting an illustrative utility grid, in accordance with an implementation;

[0071] FIG. 2A is a block diagram illustrating an example system for provisioned power optimization, in accordance with an implementation;

[0072] FIG. 2B is a block diagram illustrating example components and functionalities of a data processing system, in accordance with an implementation;

[0073] FIG. 3 illustrates an example diagram of changes in power utilization with increased optimization, in accordance with an implementation;

[0074] FIGS. 4-5 illustrate block diagrams of example data center power distribution systems, in accordance with an implementation;

[0075] FIG. 6 illustrates a block diagram of an example power distribution to the data center, in accordance with an implementation;

[0076] FIG. 7 illustrates a bar graph of an example change in data center microgrid efficiency, in accordance with an implementation;204913-1418-6318.1Docket No. 102555-0261

[0077] FIG. 8 illustrates a block diagram of an example power flow from the data center for controlling one or more components, in accordance with an implementation;

[0078] FIG. 9 illustrates a block diagram of example components of a power supply controlled using measured electrical data, in accordance with an implementation;

[0079] FIG. 10 illustrates a block diagram of an example power distribution unit (PDU), in accordance with an implementation;

[0080] FIG. 11 illustrates a block diagram of an example processing unit compute, in accordance with an implementation;

[0081] FIG. 12 illustrates a graph of an example mapping of alternating current (AC) power consumption and total GPU consumption, in accordance with an implementation;

[0082] FIG. 13 illustrates a graph of an example estimated data center capacity demand, in accordance with an implementation;

[0083] FIG. 14 illustrates a graph of an example compute between different power profiles, in accordance with an implementation;

[0084] FIG. 15 illustrates an example diagram of end-to-end processing resolution, in accordance with an implementation;

[0085] FIG. 16 illustrates edge artificial intelligence (Al) platform, in accordance with an implementation;

[0086] FIG. 17 illustrates an example diagram of open source integration for power-aware features, in accordance with an implementation;

[0087] FIG. 18 illustrates an example architecture for power envelope control, in accordance with an implementation;

[0088] FIG. 19 illustrates a graph of an example pinball loss, in accordance with an implementation;

[0089] FIG. 20 illustrates a graph of an example input and output data for neural networks, in accordance with an implementation;214913-1418-6318.1Docket No. 102555-0261

[0090] FIG. 21 illustrates a graph of an example forecasting results, in accordance with an implementation;

[0091] FIG. 22 is a flow diagram of an example method for provisioning power used by rack cabinets in a data center, in accordance with an implementation;

[0092] FIG. 23 is a flow diagram of an example method for managing controller state, in accordance with an implementation;

[0093] FIG. 24 is a flow diagram of an example method for bridging power across a power chain of a data center and power of one or more processing units, in accordance with an implementation;

[0094] FIG. 25 is a flow diagram of an example method for forecasting power consumption, in accordance with an implementation;

[0095] FIG. 26 is a flow diagram of an example method for managing power across rack cabinets, in accordance with an implementation; and

[0096] FIG. 27 is a block diagram illustrating an architecture for a computer system that can be employed to implement elements of the systems and methods described and illustrated herein, including, for example, aspects of the utility grid depicted in FIG. 1, the system of FIGS. 2A-B, the systems and operations associated with FIGS. 3-21, and methods associated with FIGS. 22-26.

[0097] The features and advantages of the present solution will become more apparent from the detailed description set forth below when taken in conjunction with the drawings, in which like reference characters identify corresponding elements throughout. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.DETAILED DESCRIPTION

[0098] Following below are more detailed descriptions of various concepts related to, and implementations of, methods, apparatuses, and systems of provisioned power optimization.224913-1418-6318.1Docket No. 102555-0261The various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways.

[0099] In certain systems, unused provisioned power may not be managed, potentially hindering data center performance, including when power provisioning limits construction and expansion. The systems and methods discussed herein can provide a data processing system for optimizing power utilization (e.g., usage of provisioned power) across facilities, including data centers. The systems and methods can incorporate or provide system-wide objective functions, edge-based measurement and control nodes, and a hierarchical communication system. The systems and methods can manage power-consuming loads and subdivide system -wide objectives into smaller optimizations implemented by edge nodes to optimize power utilization, thereby allowing for more compute operations. The systems and methods can utilize machine learning to adapt control responses, optimize the system, and perform predictions, for instance, to achieve a relatively higher average power utilization, reduced load variability, or balanced power between loads and energy storage systems, improving compute capacity and operational efficiency. The systems and methods can provide Al-driven approach(es) to optimize data center power utilization, with the data processing system capable of real-time analysis and forecasting leveraging a GPU architecture.

[0100] The systems and methods described herein can provide a technological improvement to computer and power distribution systems by, for example, providing real-time enforcement of power-envelope constraints using measured electrical waveform data and closed-loop control of physical computing hardware. The techniques described herein can process high-resolution alternating-current and direct-current waveform measurements, compute state variables indicative of imminent power excursions, and generate control instructions that modify operational parameters of processing units, power distribution units, or power supply units within defined control intervals. The data processing system can use predictive models to generate forecast values that are consumed by deterministic control logic, including finite-state control transitions and rate-limited power-limit adjustments, such that the overall operation remains rooted in physical measurements, hardware actuation, and real-time feedback control. Accordingly, the techniques described herein can improve the functioning, safety, and utilization efficiency of data center power infrastructure and computing devices by reducing load variability,234913-1418-6318.1Docket No. 102555-0261preventing power excursions, and enabling increased compute throughput within fixed provisioned-power constraints.

[0101] FIG. 1 depicts an example utility distribution environment. The utility distribution environment can include a utility grid 100. The utility grid 100 can include an electricity distribution grid with one or more devices, assets, or digital computational devices and systems, such as a data processing system 150 (e.g., DPS). In brief overview, the utility grid 100 includes a power source 101 that can be connected via a subsystem transmission bus 102 and / or via substation transformer 104 to a voltage regulating transformer 106a. The voltage regulating transformer 106a can be controlled by voltage controller 108 with regulator interface 110. Voltage regulating transformer 106a can be optionally coupled on primary distribution circuit 112 via optional distribution transformer 114 to secondary utilization circuits 116 and to one or more electrical or electronic devices 119. Voltage regulating transformer 106a can include multiple tap outputs 106b with each tap output 106b supplying electricity with a different voltage level. The utility grid 100 can include monitoring devices 118a-l 18n that can be coupled through optional potential transformers 120a-120n to secondary utilization circuits 116. The monitoring or metering devices 118a-l 18n can detect (e.g., continuously, periodically, based on a time interval, responsive to an event or trigger) measurements and continuous voltage signals of electricity supplied to one or more electrical devices 119 connected to circuit 112 or 116 from a power source 101 coupled to bus 102. These metering devices 118a-l 18n, among other components within utility distribution grids, can collect samples of power delivery or consumption, such as voltage information, at a predetermined sample rate. A voltage controller 108 can receive, via a communication media 122, measurements obtained by the metering devices 118a-l 18n, and use the measurements to make a determination regarding a voltage tap settings, and provide an indication to regulator interface 110. The regulator interface can communicate with voltage regulating transformer 106a to adjust an output tap level 106b.

[0102] In FIG. 1, in further detail, the utility grid 100 includes a power source 101. The power source 101 can include a power plant such as an installation configured to generate electrical power for distribution. The power source 101 can include an engine or other apparatus that generates electrical power. The power source 101 can create electrical power by converting power or energy from one state to another state. In some embodiments, the power source 101 can be referred to or include a power plant, power station, generating244913-1418-6318.1Docket No. 102555-0261station, powerhouse or generating plant. In some embodiments, the power source 101 can include a generator, such as a rotating machine that converts mechanical power into electrical power by creating relative motion between a magnetic field and a conductor. The power source 101 can use one or more energy source to turn the generator including, e.g., fossil fuels such as coal, oil, and natural gas, nuclear power, or cleaner renewable sources such as solar, wind, wave and hydroelectric.

[0103] In some embodiments, the utility grid 100 includes one or more substation transmission bus 102. The substation transmission bus 102 can include or refer to transmission tower, such as a structure (e.g., a steel lattice tower, concrete, wood, etc.), that supports an overhead power line used to distribute electricity from a power source 101 to a substation 104 or distribution point 114. Transmission towers 102 can be used in high-voltage AC and DC systems, and come in a wide variety of shapes and sizes. In an illustrative example, a transmission tower can range in height from 15 to 55 meters or more.Transmission towers 102 can be of various types including, e.g., suspension, terminal, tension, and transposition. In some embodiments, the utility grid 100 can include underground power lines in addition to or instead of transmission towers 102.

[0104] In some embodiments, the utility grid 100 includes a substation 104 or electrical substation 104 or substation transformer 104. A substation can be part of an electrical generation, transmission, and distribution system. In some embodiments, the substation 104 transform voltage from high to low, or the reverse, or performs any of several other functions to facilitate the distribution of electricity. In some embodiments, the utility grid 100 can include several substations 104 between the power plant 101 and the consumer electoral devices 119 with electric power flowing through them at different voltage levels.

[0105] The substations 104 can be remotely operated, supervised and controlled (e.g., via a supervisory control and data acquisition system or data processing system 150). A substation can include one or more transformers to change voltage levels between high transmission voltages and lower distribution voltages, or at the interconnection of two different transmission voltages.

[0106] The regulating transformer 106 can include: (1) a multi-tap autotransformer (single or three phase), which are used for distribution; or (2) on-load tap changer (three phase transformer), which can be integrated into a substation transformer 104 and used for254913-1418-6318.1Docket No. 102555-0261both transmission and distribution. The illustrated system described herein can be implemented as either a single-phase or three-phase distribution system. The utility grid 100 can include an alternating current (AC) power distribution system and the term voltage can refer to an “RMS Voltage”, in some embodiments.

[0107] The utility grid 100 can include a distribution point 114 or distribution transformer 114, which can refer to an electric power distribution system. In some embodiments, the distribution point 114 can be a final or near final stage in the delivery of electric power. For example, the distribution point 114 can carry electricity from the transmission system (which can include one or more transmission towers 102) to individual consumers 119. In some embodiments, the distribution system can include the substations 104 and connect to the transmission system to lower the transmission voltage to medium voltage ranging between 2 kV and 35 kV with the use of transformers, for example. Primary distribution lines or circuit 112 carry this medium voltage power to distribution transformers located near the customer's premises 119. Distribution transformers can further lower the voltage to the utilization voltage of appliances and can feed several customers 119 through secondary distribution lines or circuits 116 at this voltage. Commercial and residential customers 119 can be connected to the secondary distribution lines through service drops. In some embodiments, customers demanding high load can be connected directly at the primary distribution level or the subtransmission level.

[0108] The utility grid 100 can include or couple to one or more consumer sites 119. Consumer sites 119 can include, for example, a building, house, shopping mall, factory, office building, residential building, commercial building, stadium, movie theater, etc. The consumer sites 119 can be configured to receive electricity from the distribution point 114 via a power line (above ground or underground). A consumer site 119 can be coupled to the distribution point 114 via a power line. The consumer site 119 can be further coupled to a site meter 118a-n or advanced metering infrastructure (“AMI”). The site meter 118a-n can be associated with a controllable primary circuit segment 112. The association can be stored as a pointer, link, field, data record, or other indicator in a data file in a database.

[0109] The utility grid 100 can include site meters 118a-n or AMI. Site meters 118a-n can measure, collect, and analyze energy usage, and communicate with metering devices such as electricity meters, gas meters, heat meters, and water meters, either on request or on a schedule. Site meters 118a-n can include hardware, software, communications, consumer264913-1418-6318.1Docket No. 102555-0261energy displays and controllers, customer associated systems, Meter Data Management (MDM) software, or supplier business systems. In some embodiments, the site meters 118a-n can obtain samples of electricity usage in real time or based on a time interval, and convey, transmit or otherwise provide the information. In some embodiments, the information collected by the site meter can be referred to as meter observations or metering observations and can include the samples of electricity usage. In some embodiments, the site meter 118a-n can convey the metering observations along with additional information such as a unique identifier of the site meter 118a-n, unique identifier of the consumer, a time stamp, date stamp, temperature reading, humidity reading, ambient temperature reading, etc. In some embodiments, each consumer site 119 (or electronic device) can include or be coupled to a corresponding site meter or monitoring device 118a-l 18n.

[0110] Monitoring devices 118a-l 18n can be coupled through communications media 122a-122n to voltage controller 108. Voltage controller 108 can compute (e.g., discrete-time, continuously or based on a time interval or responsive to a condition / event) values for electricity that facilitates regulating or controlling electricity supplied or provided via the utility grid. For example, the voltage controller 108 can compute estimated deviant voltage levels that the supplied electricity (e.g., supplied from power source 101) will not drop below or exceed as a result of varying electrical consumption by the one or more electrical devices 119. The deviant voltage levels can be computed based on a predetermined confidence level and the detected measurements. Voltage controller 108 can include a voltage signal processing circuit 126 that receives sampled signals from metering devices 118a-l 18n.Metering devices 118a-l 18n can process and sample the voltage signals such that the sampled voltage signals are sampled as a time series (e.g., uniform time series free of spectral aliases or non-uniform time series).

[0111] Voltage signal processing circuit 126 can receive signals via communications media 122a-n from metering devices 118a-n, process the signals, and feed them to voltage adjustment decision processor circuit 128. Although the term “circuit” is used in this description, the term is not meant to limit this disclosure to a particular type of hardware or design, and other terms known generally known such as the term “element”, “hardware”, “device” or “apparatus” could be used synonymously with or in place of term “circuit” and can perform the same function. For example, in some embodiments the functionality can be carried out using one or more digital processors, e.g., implementing one or more digital signal274913-1418-6318.1Docket No. 102555-0261processing algorithms. Adjustment decision processor circuit 128 can determine a voltage location with respect to a defined decision boundary and set the tap position and settings in response to the determined location. For example, the adjustment decision processing circuit 128 in voltage controller 108 can compute a deviant voltage level that is used to adjust the voltage level output of electricity supplied to the electrical device. Thus, one of the multiple tap settings of regulating transformer 106 can be continuously selected by voltage controller 108 via regulator interface 110 to supply electricity to the one or more electrical devices based on the computed deviant voltage level. The voltage controller 108 can also receive information about voltage regulator transformer 106a or output tap settings 106b via the regulator interface 110. Regulator interface 110 can include a processor controlled circuit for selecting one of the multiple tap settings in voltage regulating transformer 106 in response to an indication signal from voltage controller 108. As the computed deviant voltage level changes, other tap settings 106b (or settings) of regulating transformer 106a are selected by voltage controller 108 to change the voltage level of the electricity supplied to the one or more electrical devices 119.

[0112] The network 140 can be connected via wired or wireless links. Wired links can include Digital Subscriber Line (DSL), coaxial cable lines, or optical fiber lines. The wireless links can include BLUETOOTH, Wi-Fi, Worldwide Interoperability for Microwave Access (WiMAX), an infrared channel or satellite band. The wireless links can also include any cellular network standards used to communicate among mobile devices, including standards that qualify as 1G, 2G, 3G, or 4G. The network standards can qualify as one or more generation of mobile telecommunication standards by fulfilling a specification or standards such as the specifications maintained by International Telecommunication Union. The 3G standards, for example, can correspond to the International Mobile Telecommunications-2000 (IMT-2000) specification, and the 4G standards can correspond to the International Mobile Telecommunications Advanced (IMT- Advanced) specification. Examples of cellular network standards include AMPS, GSM, GPRS, UMTS, LTE, LTE Advanced, Mobile WiMAX, and WiMAX- Advanced. Cellular network standards can use various channel access methods e.g. FDMA, TDMA, CDMA, or SDMA. In some embodiments, different types of data can be transmitted via different links and standards. In other embodiments, the same types of data can be transmitted via different links and standards.284913-1418-6318.1Docket No. 102555-0261

[0113] The network 140 can be any type and / or form of network. The geographical scope of the network 140 can vary widely and the network 140 can be a body area network (BAN), a personal area network (PAN), a local-area network (LAN), e.g. Intranet, a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of the network 140 can be of any form and can include, e.g., any of the following: point-to-point, bus, star, ring, mesh, or tree. The network 140 can be an overlay network which is virtual and sits on top of one or more layers of other networks 140. The network 140 can be of any such network topology as known to those ordinarily skilled in the art capable of supporting the operations described herein. The network 140 can utilize different techniques and layers or stacks of protocols, including, e.g., the Ethernet protocol, the internet protocol suite (TCP / IP), the ATM (Asynchronous Transfer Mode) technique, the SONET (Synchronous Optical Networking) protocol, or the SDH (Synchronous Digital Hierarchy) protocol. The TCP / IP internet protocol suite can include application layer, transport layer, internet layer (including, e.g., IPv6), or the link layer. The network 140 can be a type of a broadcast network, a telecommunications network, a data communication network, or a computer network.

[0114] The network 140 can include computer networks such as the internet, local, wide, near field communication, metro or other area networks, as well as satellite networks or other computer networks such as voice or data mobile phone communications networks, and combinations thereof. The network 140 can include a point-to-point network, broadcast network, telecommunications network, asynchronous transfer mode network, synchronous optical network, or a synchronous digital hierarchy network, for example. The network 140 can include at least one wireless link such as an infrared channel or satellite band. The topology of the network 140 can include a bus, star, or ring network topology. The network 140 can include mobile telephone or data networks using any protocol or protocols to communicate among vehicles or other devices, including advanced mobile protocols, time or code division multiple access protocols, global system for mobile communication protocols, general packet radio services protocols, or universal mobile telecommunication system protocols, and the same types of data can be transmitted via different protocols.

[0115] One or more components, assets, or devices of utility grid 100 can communicate via network 140. The utility grid 100 can use one or more networks, such as public or private networks. The utility grid 100 can communicate or interface with a data processing system 150 designed and constructed to communicate, interface or control the utility grid 100 via294913-1418-6318.1Docket No. 102555-0261network 140. Each asset, device, or component of utility grid 100 can include one or more computing devices 1800 or a portion of computing device 1800 or some or all functionality of computing device 1800.

[0116] The data processing system 150 can reside on a computing device of the utility grid 100, or on a computing device or server that is external or remote from the utility grid 100. The data processing system 150 can reside or execute in a cloud computing environment or distributed computing environment. The data processing system 150 can reside on or execute on multiple local computing devices located throughout the utility grid 100. For example, the utility grid 100 can include multiple local computing devices each configured with one or more components or functionality of the data processing system 150. The data processing system 150 can reside on a computing device in a facility, such as but not limited to a data center. The data processing system 150 can reside on a computing device external from the facility.

[0117] Each of the components of the data processing system 150 can be implemented using hardware or a combination of software and hardware. Each component of the data processing system 150 can include logical circuity (e.g., a central processing unit or CPU) that responses to and processes instructions fetched from a memory unit (e.g., memory 2715 or storage device 2725). Each component of the data processing system 150 can include or use a microprocessor or a multi-core processor. A multi-core processor can include two or more processing units on a single computing component. Each component of the data processing system 150 can be based on any of these processors, or any other processor capable of operating as described herein. Each processor can utilize instruction level parallelism, thread level parallelism, different levels of cache, etc. For example, the data processing system 150 can include at least one logic device such as a computing device or server having at least one processor to communicate via the network 140.

[0118] The components and elements of the data processing system 150 can be separate components, a single component, or part of the data processing system 150. For example, individual components or elements of the data processing system 150 can operate concurrently to perform at least one feature or function discussed herein. In another example, components of the data processing system 150 can execute individual instructions or tasks. The components of the data processing system 150 can be connected or communicatively coupled to one another. The connection between the various components of the data304913-1418-6318.1Docket No. 102555-0261processing system 150 can be wired or wireless, or any combination thereof. Counterpart systems or components can be hosted on other computing devices.

[0119] The data processing system 150 can communicate with one or more metering devices 118 via the network 140. In some cases, the data processing system 150 can include features or functionalities of the metering devices 118. In some other cases, the data processing system 150 can be a part of the metering device 118, such that the metering device 118 can perform certain features or functionalities of the data processing system 150. In some configurations, the data processing system 150 can be electrically or communicatively coupled to the one or more components at the grid edge.

[0120] The data processing system 150 can obtain measurements (e.g., raw or processed data or electric waveforms) from the one or more metering devices 118 within the utility grid 100. The data processing system 150 can receive or obtain the measurements from the metering devices 118 in response to each metering device 118 performing the measurement. The data processing system 150 may receive an aggregate of the measurements from the metering devices 118. In this case, each metering device 118 may store the measurements in a local memory and send the data in response to at least one of a predetermined time interval for a scheduled transmission, receiving a request for data from the data processing system 150, or a predetermined duration, size, or amount of data samples is collected. In some cases, the metering devices 118 can process the data prior to transmitting the data to the data processing system 150. The metering device 118 can be a part of or connected to a facility at the grid edge. The data processing system 150 can obtain measurements from one or more components at the grid edge (e.g., data center). The data processing system 150 can obtain measurements at various electrical connections between components, input or output ports of one or more components, etc., associated with the grid edge. The data processing system 150 can include one or more sensors or measuring devices to obtain the measurements.

[0121] FIG. 2A depicts a block diagram illustrating an example system 200 for provisioned power optimization. The system 200 can include, interface with, access, or otherwise communicate with at least one utility grid 100, at least one data processing system 150 (e.g., one or more of data processing system 150A-C), and at least one data center 201 (or one or more components of the data center 201). The data center 201 can include one or more components (e.g., one or more processing units, batteries, power supplies, servers,314913-1418-6318.1Docket No. 102555-0261databases, or other devices) configured to operate with each other, for instance, to manage, store, process, and disseminate data.

[0122] The data center 201 can provide computing resources for devices within the network 140, such as for cloud services, enterprise applications, or internet services, to name a few. The data center 201 can support servers and storage systems that handle data-related tasks, support high-performance computing, or ensure data availability and security. The data center 201 can include cooling systems to maintain optimal operating temperatures and power management systems to ensure efficient energy use and uninterrupted operation. The components of the data center 201 can be communicatively coupled with the data processing system 150A or interface with the data processing system 150A to communicate data (e.g., electrical data) for processing. The components of the data center 201 can operate with multiple data processing systems 150A-C such as those local or remote from the data center 201.

[0123] As illustrated in FIG. 2A, the data processing system 150A-C can be located in various environments. For example, at least one data processing system 150A can be located in or local to the data center 201. At least one data processing system 150B can be integrated into the utility grid 100. At least one data processing system 150C can be deployed remotely from the data center 201 and the utility grid 100. It should be noted that the system 200 can include other types of facilities at the grid edge, not limited to the data center 201. One or more (or all) of the data processing systems 150A-C can include one or more components or functionalities of the data processing system 150 as shown in and described in conjunction with FIG. 2B.

[0124] In some configurations, the data processing system 150A may be implemented in a computing device within the data center 201. In further examples, the data processing system 150A can be situated in, mounted on, or part of a rack cabinet (sometimes referred to as a rack or a node). The data processing system 150A can perform operations in conjunction with components of the rack cabinet, for instance, the data processing system 150A can orchestrate at least one operation for processing units or servers of the rack cabinet.

[0125] The data processing system 150A may be a hardware component, a software component, or both hardware and software components on a server within a shelf of a rack cabinet, e.g., deployed within or hosted by a server computing environment. The data324913-1418-6318.1Docket No. 102555-0261processing system 150A can be a part of one of a plurality of shelves in a rack cabinet. In this case, the data processing system 150A can operate in conjunction with the server, e.g., one or more processing units or components of the server. The data processing system 150A being on a server can refer to the data processing system 150A being co-located with the hardware of the server. The data processing system 150A can utilize the server as an execution or hosting environment. In some cases, the data processing system 150A may operate with multiple racks. The data processing system 150A can be located, embedded, or disposed in any suitable location in the data center 201.

[0126] The data center 201 can include multiple data processing systems 150A that function as orchestrators. The data processing systems 150A can be distributed across any suitable location within the data center 201 to orchestrate one or more operations discussed herein. For example, at least one data processing system 150A can receive data from one or more devices or components of the data center 201 for processing. The data processing system 150A can provide one or more commands or instructions to one or more devices or components of the data center 201. For example, the data processing system 150A can issue at least one command to one or more servers or one or more processing units of a rack cabinet. In another example, the data processing system 150 A can issue multiple commands to servers across multiple rack cabinets. The data processing system 150A can coordinate operations between rack cabinets.

[0127] In some configurations, the data processing system 150A can orchestrate operations for a row of rack cabinets. In some other configurations, the data processing system 150A can orchestrate operations for multiple rows of rack cabinets. In certain cases, the data processing system 150A can orchestrate operations for all rack cabinets of the data center 201. The data processing system 150A can be an orchestrator for one or more other data processing systems within the data center 201. For example, the data processing system 150A can issue commands or instructions to one or more other data processing systems local to one or more respective servers. These one or more other data processing systems can operate in accordance with the instructions from the orchestrator (e.g., the data processing system 150A). It should be noted that the data processing system 150A can be in other locations or operate with different components of the data center 201, not limited to those discussed herein.334913-1418-6318.1Docket No. 102555-0261

[0128] The data processing system 150B can be integrated into the utility grid 100 to perform monitoring, control, or orchestration operations for various grid components. The data processing system 150B can be deployed within or adjacent to one or more components of the utility grid 100 such as a metering device 118, a voltage regulator 108, etc. The data processing system 150B can be located at a site or grid edge. The data processingsystem 150B can receive electrical data, e.g., high-voltage transmission data, voltage measurements, and operational status signals from grid devices such as one or more metering devices 118 or voltage controllers 126. The data processing system 150B can process these inputs to, for instance, adjust grid parameters, manage load balancing strategies, initiate fault recovery procedures, etc. The grid parameters can include at least one of, but not limited to, voltage level, voltage regulator settings, current level, frequency, power factor, load shedding thresholds, or protection relay settings. The data processing system 150B can operate in real time such as operating at sub-millisecond speed or frequency. The data processing system 150B can be provided at any suitable locations, devices, or components in the utility grid 100 or at the edge of the utility grid 100.

[0129] The data processing system 150C can be deployed at a location that is remote from both the utility grid 100 and the data center 201. The data processing system 150C may reside at a third-party facility, a cloud computing environment, or an edge site not directly tied to the utility grid 100 or the data center 201. The data processing system 150C can communicate with one or more components of the utility grid 100 or one or more components of the data center 201 via the network 140. The data processing system 150C can receive or transmit operational data, measurement values, and status reports from or to distributed components within those systems. The data processing system 150C can perform processing, analytics, or orchestration functions for component(s) of the utility grid 100 or the data center 201, for instance, without being physically embedded in the utility grid 100 or co-located with the data center components. For example, the data processing system 150C can act as a supervisory orchestrator that integrates both the utility grid 100 and the data center 201 operational states to at least one of optimize electrical resource allocation, perform prediction of power consumption, faults, or electrical spikes (or dips), or manage crosssystem workflows. The one or more devices, components, or systems of the utility grid 100 or the system 200 can be composed of hardware, software, or a combination of hardware and software components.344913-1418-6318.1Docket No. 102555-0261

[0130] FIG. 2B depicts a block diagram illustrating example components and functionalities of the data processing system 150. The one or more components of the data processing system 150 can include or be composed of hardware, software, or a combination of hardware and software components. The data processing system 150 can execute nonlimiting features or functionalities discussed herein. For instance, the data processing system 150 can execute features for one or more components of the utility grid 100. The data processing system 150 can execute features for one or more components of the data center 201. In various configurations, the data processing system 150 can operate in conjunction with the components of the data center 201 to at least optimize provisioned power (e.g., adjust power envelope), control data center components (e.g., schedule or adjust compute resources), adjust performance of one or more processing units, provide visibility into processing unit workload (e.g., at least GPU workload or CPU workload), train neural networks, or forecast power consumption, to name a few.

[0131] The data processing system 150 can be a grid data processing system of the utility grid 100. For instance, the data processing system 150 can be part of a metering device 118, a local device at a site or grid edge, or a centralized device on the utility grid 100 for sending commands to other data processing systems. In another example, the data processing system 150 can be a local data processing system of the data center 201. In this case, the data processing system 150 can include or be a part of a device in the data center 201, on at least one rack cabinet (e.g., per server or shelf, per rack, or per device on each shelf), or remote from rack cabinets of the data center 201.

[0132] The data processing system 150 can optimize the utilization of provisioned power at the grid edge. The data processing system 150 can transmit or receive data to or from one or more components of the data center 201 or other facilities of the system 200 via the network 140. The utility grid 100, the network 140, and the data processing system 150 can be referred to in conjunction with FIG. 1. The data processing system 150 can perform the features or operations discussed herein to optimize power usage at least at the data center 201.

[0133] For example, the data processing system 150 can optimize data center power utilization, allowing for relatively more compute hardware to be utilized and relatively more compute operations to be performed at least at the data center 201. In some cases, the data processing system 150 can optimize data center power allocation, for instance, by354913-1418-6318.1Docket No. 102555-0261(dynamically) adjusting power provisioned to one or more rack cabinets or the power envelope depending on compute resource utilization.

[0134] The data processing system 150 can perform features or functionalities discussed herein, for instance, as part of a plurality of elements or components to achieve the optimal power utilization. The plurality of elements or components can interface or communicate with one another to facilitate optimal power utilization. For example, the elements can include at least a system -wide objective function that, when optimized, allows for an increase in the overall compute utilization, a specification of edge-based measurement and control nodes that implement system controls to achieve the optimization objective by managing compute (or other) power consuming loads, and a specification of a hierarchical or fully-connected communication system that allows for the system -wide objective to be subdivided into smaller optimizations which can be implemented by the edge nodes.

[0135] The systems and methods can involve the system-wide optimization objective based on load-balancing to demonstrate the potential for achieving significant increases in utilization. The systems and methods can involve different types of hierarchical system structure that break down the optimization to individual load controls, and their associated communication structures. The systems and methods can involve a variety of implementations of edge nodes based on a platform (e.g., data processing system 150) for electricity measurement, optimization, and controls. The non-limiting features of the systems and methods can be executed by the data processing system 150. The non-limiting components of the systems and methods can include, correspond to, or be a part of the data processing system 150.

[0136] The features or functionalities of the data processing system 150 can be applied to data center 201. The one or more operations or features associated with the data processing system 150 can be implemented using or performed by the data processing system 150 or other suitable computing devices communicatively coupled to one or more components of the data center 201. For example, certain systems may not connect the power distribution system of certain facilitate or data centers to compute load management. The systems and methods of the technical solution can connect the power distribution system of at least the data center 201 to at least the data processing system 150 for compute load management, thereby allowing provisioned power optimization.364913-1418-6318.1Docket No. 102555-0261

[0137] The system-wide optimization objective can utilize a distribution-controls-for-power-systems-optimization approach. The system-wide optimization objective can involve maximizing the use of the total provisioned (interconnected) power being used for computation, e.g., improve usage of existing or provisioned electrical resources. For example, contrary to certain metrics, such as PUE, the systems and methods (e.g., the data processing system 150) can effectively utilize the unused provisioned power. As power provisioning may be a gating factor in data center construction, directly increasing the utilization of the provisioned resources can have an impact on various operational efficiencies of the data center 201. In some implementations, control performance can be evaluated using one or more metrics such as RMS excess power, maximum excess power, or unused capacity, to quantify (i) compliance with thermal or steady-state regimes, (ii) excursion severity relative to protection limits, or (iii) utilization efficiency relative to provisioned capacity.

[0138] In some implementations, the power distribution system within the data center 201 may at least in part be similar to or include various parallels to an electrical grid distribution system (e.g., utility grid 100). Techniques applied or executed for the utility grid 100 may apply to or translate for the power distribution system of the data center 201. For example, information or measurements from the power distribution system can be utilized to reduce load variability, allowing for a relatively higher average-power-utilization and reduction of reserved power. In some cases, the measurements can be utilized for power balancing between loads (e.g., load balancing), allowing for compute spikes to be offset (e.g., minimize or mitigate a sudden increase in power demand to prevent overload or excessive resource consumption) while maintaining overall utilization. In some other cases, the measurements can be utilized for power balancing of the total load with energy storage using peak-shaving and load-leveling approaches.

[0139] The distributed-control approach can utilize existing control affordances over power-consuming and power-producing devices to increase overall system power utilization. The data processing system 150 can be configured or include authorization to control one or more components of the power distribution system of the data center 201 (or other devices). For example, the data processing system 150 can manage compute load via dynamic voltage frequency scaling (DVFS) or power-storage devices. These control inputs (e.g., the DVFS or power-storage devices) can (directly) relate to power usage. With at least one of the374913-1418-6318.1Docket No. 102555-0261objectives being on the power domain, the system design or configuration can be minimized and implemented with or additionally to existing installations (e.g., of the data center 201).

[0140] The data processing system 150 can use machine learning or Al-based models to adapt control responses to local conditions, optimize the system, and perform predictions as inputs into system optimization, such as described in conjunction with at least FIG. 3. The system structure and a high-level roadmap of distributed and hierarchical-control approaches of increasing functionalities and potential for benefit can be provided herein. A series of system models can be provided or implemented for analysis.

[0141] The data processing system 150 can include one or more components configured for data processing, power envelope management, forecasting power consumption, training and deploying artificial intelligence model, controlling components of the data center 201, or executing one or more actions. The one or more components of the data processing system 150 can include, but are not limited to, at least one interface 202, at least one data collector 204, at least one power envelope identifier 206, at least one data processor 208, at least one model manager 210, at least one power predictor 212, at least one action manager 214, at least one controller state manager 216, at least one power condition detector 218, at least one controller 220, and at least one data repository 222.

[0142] Each of the components (e.g., interface 202, data collector 204, power envelope identifier 206, data processor 208, model manager 210, power predictor 212, action manager 214, controller state manager 216, power condition detector 218, controller 220, or data repository 222) of the data processing system 150 can be implemented using hardware or a combination of software and hardware. In some configurations, the one or more components of the data processing system 150 can be deployed as services, such as a power enforcer service executing a control loop that consumes real-time power telemetry from a metrology aggregation service and issues commands to adjust power-cap (or power limit) for processing unit(s) via a control service interfacing with a management API of the processing unit(s).

[0143] Each component of the data processing system 150 can include logical circuity (e.g., a central processing unit or graphics processing unit) that responds to and processes instructions fetched from a memory unit (e.g., memory 2715 or storage device 2725). Each component of the data processing system 150 can include or use a microprocessor or a multicore processor. A multi-core processor can include two or more processing units on a single384913-1418-6318.1Docket No. 102555-0261computing component. Each component of the data processing system 150 can be based on any of these processors, or any other processor capable of operating as described herein. Each processor can utilize instruction level parallelism, thread level parallelism, different levels of cache, etc. For example, the data processing system 150 can include at least one logic device such as a computing device or server having at least one processor to communicate via the network 140.

[0144] The components and elements (e.g., interface 202, data collector 204, power envelope identifier 206, data processor 208, model manager 210, power predictor 212, action manager 214, controller state manager 216, power condition detector 218, controller 220, or data repository 222) of the data processing system 150 can be separate components, a single component, or part of the data processing system 150. For example, individual components or elements of the data processing system 150 can operate concurrently to perform at least one feature or function discussed herein. In another example, components of the data processing system 150 can execute individual instructions or tasks. In yet another example, the components of the data processing system 150 can be a single component to perform one or more features or functions discussed herein. The components of the data processing system 150 can be connected or communicatively coupled to one another, such as via the interface 202. The components of the data processing system 150 can share input, output, or processed data with each other for executing one or more operations of the data processing system 150. The connection between the various components of the data processing system 150 can be wired or wireless, or any combination thereof. Counterpart systems or components can be hosted on other computing devices.

[0145] The interface 202 can interface with the network 140, devices within the system 200 (e.g., one or more components of the data center 201 or utility grid 100), or components of the data processing system 150. The interface 202 can include features and functionalities similar to the communication interface of one or more metering devices 118 to interface with the aforementioned components, such as in conjunction with FIG. 1. For example, the interface 202 can include standard telephone lines LAN or WAN links (e.g., 802.11, Tl, T3, Gigabit Ethernet, Infiniband), broadband connections (e.g., ISDN, Frame Relay, ATM, Gigabit Ethernet, Ethemet-over-SONET, ADSL, VDSL, BPON, GPON, fiber optical including FiOS), wireless connections, or some combination of any or all of the above.Connections can be established using a variety of communication protocols (e.g., TCP / IP,394913-1418-6318.1Docket No. 102555-0261Ethernet, ARCNET, SONET, SDH, Fiber Distributed Data Interface (FDDI), IEEE 802.11a / b / g / n / ac CDMA, GSM, WiMax and direct asynchronous connections). The interface 202 can include at least a built-in network adapter, network interface card, PCMCIA network card, EXPRESSCARD network card, card bus network adapter, wireless network adapter, USB network adapter, modem, or any other device suitable for interfacing one or more devices within the system 200 to any type of network capable of communication.

[0146] The interface 202 can communicate with one or more aforementioned components to receive data from or transmit data to at least one of the utility grid 100, the data center 201, or one or more metering devices 118. For example, the interface 202 can receive or transmit data representative of electricity distribution to the data center 201, to one or more rack cabinets of the data center 201, or to one or more servers of the one or more rack cabinets, electrical data (e.g., electric waveform data) measured at one or more devices across a power chain of the data center 201, data representative of electricity distribution to one or more metering devices 118 within the utility grid 100, processed data from other devices or components, or instructions from client devices in communication with the data processing system 150. The interface 202 can communicate other types of signals or data, not limited to those discussed herein.

[0147] In some configurations, the interface 202 can include one or more sensors for measuring, obtaining, or otherwise receiving electric waveform data at a location within the data center 201. For example, the data processing system 150 can be embedded in a rack cabinet or located at any suitable location within the power chain of the data center 201 to monitor electrical data. The interface 202 can interface with one or more components of the data center 201. The interface 202 can communicate with other data processing systems 150 local to or remote from the data center 201. Other types of data, additional or alternative to those herein, communicated to, from, or otherwise within the data processing system 150 can be routed or exchanged via the interface 202.

[0148] The data collector 204 can collect data received from the interface 202. The data collector 204 can obtain or collect electrical data of the data center 201, at location(s) across the power chain of the data center 201, or one or more components of the data center 201. The electrical data can include data samples of an electrical waveform corresponding to electricity (e.g., electrical signals) distributed at or to the location of the data processing system 150 within the data center 201 (or other locations depending on the location of the404913-1418-6318.1Docket No. 102555-0261data processing system 150 or device(s) providing the electrical data). The data collector 204 can receive relatively high-resolution data, such as at least 1 kHz, 7 kHz, 10 kHz, or 32 kHz of voltage data or current data.

[0149] In some cases, the data collector 204 can receive pre-processed data from other devices. For example, other devices can apply at least one of filtering operations, aggregation operations, or normalization operations to the data samples before transmission to the data processing system 150. The data collector 204 can store collected data in the data repository 222. For purposes of examples herein, the electrical data discussed herein can be voltage data or current data, although other types of electrical data or metrics.

[0150] The data collector 204 can obtain electric waveform data representative of power delivered at one or more locations within the data center 201 or across the power chain of the data center 201. The power chain of the data center 201 can refer to a series of electrical components or connections that distribute electrical power from a power source to one or more loads of the data center 201. For instance, the power source can include (or correspond to) the power source 101 of the utility grid 100 or at least one electrical circuit distributing electricity to the data center 201. In some cases, the power source can include (or correspond to) a generator, a power storage, or a power distribution unit of the data center 201. Other types of power sources supplying electricity to one or more loads of the data center 201 can be included. The one or more loads can include, but are not limited to, one or more processing units, memory devices, network equipment, storage systems, cooling systems, or power distribution components that consume electrical power during operation. The electric waveform data can include data samples corresponding to electricity distributed to or consumed by one or more components of the data center 201 depending on a location at which the data processing system 150 is situated or a location at which one or more measurement devices supplying the electric waveform data are situated.

[0151] The data collector 204 can obtain electric waveform data from one or more measurement circuits disposed at a power distribution unit (PDU) electrically coupled to one or more rack cabinets. The data collector 204 can obtain electric waveform data from one or more measurement circuits disposed at a power supply unit (PSU) electrically coupled to one or more rack cabinets. The data collector 204 can obtain electric waveform data from one or more measurement circuits disposed at one or more power input ports associated with one or more processing units of a rack cabinet.414913-1418-6318.1Docket No. 102555-0261

[0152] In some configurations, the data collector 204 can obtain electric waveform data measured at devices distributed across the power chain of the data center 201. For example, the devices distributed across the power chain can include at least one of a utility feed, an automatic transfer switch, an uninterruptible power supply, a generator, a circuit breaker, a power distribution unit, a power supply unit, or one or more of the rack cabinets, among others. The data collector 204 can obtain electric waveform data comprising alternating current (AC) data or direct current (DC) data.

[0153] The data collector 204 can obtain electric waveform data measured at one or more power conversion stages of a power supply unit supplying power to at least one rack cabinet or at least one power input port associated with one or more processing units. The one or more power conversion stages can include at least one of a rectification stage, a filtering stage, or a voltage regulation stage, among others. The processing unit discussed herein can include graphics processing unit (GPU), central processing unit (CPU), or other types of processing units. The data collector 204 can cache or store electrical data from one or more power sources associated with a server.

[0154] The data collector 204 can receive feedback signal(s) from at least one device or at least one processing unit subsequent to initiation actions, such as adjusting one or more parameters of devices or processing units. For example, the feedback signals can include status updates, performance metrics, or operational states from the devices or processing units to which control signals were provided. The feedback signals can include electric waveform data obtained after performing at least one action, e.g., in a subsequent operational cycle of the data processing system 150. The feedback signals may include an acknowledgement that the device receives the instructions or commands. In some cases, the feedback signals can indicate whether the one or more actions initiated by the data processing system 150 resulted in a desired change in power consumption, operational parameter adjustments, or other targeted outcomes. The feedback signals can include measurements of power consumption or other data after adjustment of the one or more parameters, such as voltage waveform data or current waveform data captured by measurement circuits coupled to the devices or processing units. In some implementations, and as used herein, a parameter of a processing unit can, for example, a power limit, operating frequency, voltage, memory frequency, memory capacity, workload scheduling parameter, or any other configurable settings or behaviors associated424913-1418-6318.1Docket No. 102555-0261with the processing unit, where a parameter of another device can be any configurable settings associated with the device.

[0155] The data collector 204 can receive other types of data such as predefined data or metadata. For example, the data collector 204 can receive configuration parameters that indicate sampling rates, thresholds, or identifiers of sources from which to collect electric waveform data. The data collector 204 can obtain hardware specification of the processing units, software versions of the processing units, configuration settings of the processing units, etc. The data collector 204 can obtain layout information of the data center 201 such as a number of rack cabinets, a number of shelves per rack cabinet, a number of processing units per shelf, operations assigned to the processing units, server configuration (e.g., processing units assigned to server operation, at least one shelf assigned to the server, or a type of server), etc. The metadata can include timestamps, source identifiers associated with measurement circuits, or labels indicating the type of electrical signal captured, such as voltage data or current data. The data collector 204 can provide collected data for processing, for instance, by the data processor 208. The data collector 204 can collect other types of data, not limited to those discussed herein.

[0156] The power envelope identifier 206 can identify an amount of power provisioned to the data center 201 or one or more components within the data center 201. The amount of power provisioned can refer to a power envelope or other corresponding terms. Provisioned power (or power envelope) can refer an amount of electrical power allocated or made available to a data center, rack cabinet, or processing units for use during operation. The provisioned power can be independent of an actual power consumed by components of the data center 201. For instance, a portion of the power envelope may be reserved such as for contingencies, including transient load surges, redundancy requirements, or equipment fault conditions, as described in conjunction with at least FIG. 3. The power envelope identifier 206 can receive information from a utility grid 100 indicating a total amount of power allocated for use by the data center 201 during a time window. In some implementations, the power envelope identifier 206 can access a power allocation record stored in the data repository 222 that specifies a maximum power capacity provisioned to the data center 201 by an external power source.

[0157] For example, the power allocation record can specify a peak power limit (e.g., negotiated with a utility provider) or a predefined maximum power corresponding to at least a434913-1418-6318.1Docket No. 102555-0261combined capacity of all electrical circuits feeding the data center 201. In some cases, the power envelope identifier 206 can monitor electric waveform data or electrical power supplied to the data center 201, e.g., from the utility grid 100, to determine the amount of power provisioned to the data center 201. For instance, the power envelope identifier 206 can receive electric waveform data from one or more metering devices 118 coupled to components of the data center 201, a power transformer supplying electricity from the utility grid 100 to the data center 201, at least one outlet of the data center 201 electrically coupled to rack cabinets, or other non-limiting locations.

[0158] The power envelope identifier 206 can identify a subset of the power provisioned to the data center 201 such as a portion allocated for a subset of components. For example, the power envelope identifier 206 can identify a total amount of power provisioned to the data center 201 to be used by at least the rack cabinets, cooling systems, and other components of the data center 201. The power envelope identifier 206 can identify a subset of the total provisioned power (e.g., a portion of a power envelope) for at least one rack cabinet or at least one row of the rack cabinets. In some cases, the power envelope identifier 206 can identify a subset of the provisioned power allocated or provided for one or more processors on a respective shelf of the rack cabinet.

[0159] In some configurations, the power envelope for the data center 201 or one or more components of the data center 201 can be predefined. In this case, the power envelope identifier 206 can retrieve the predefined information regarding the power envelope from the data repository 222 or other storage device. The power envelope identifier 206 may obtain the predefined power envelope information from respective components or devices in the data center 201. The power envelope may be based on the capability of the component to handle an amount of electrical power. The power envelope identifier 206 can perform other types of identification to identify an amount of power provisioned to the data center 201 or components of the data center 201.

[0160] The data processor 208 can perform one or more processing operations on collected or stored data such as electric waveform data (e.g., from the data collector 204), power envelope (e.g., from the power envelope identifier 206), prediction data (e.g., from at least a power predictor 212), or other types of data. For example, the data processor 208 can convert the electric waveform data into quantitative metrics representing power consumption at one or more locations within the data center 201. For example, the data processor 208 can444913-1418-6318.1Docket No. 102555-0261apply a root mean square (RMS) calculation to alternating current (AC) waveform samples to determine an RMS power value for a time window. The time window can be a predefined time period or based on one or more cycles of the waveform. The time window can sometimes be referred to as a time interval, a time frame, or a time duration, among other expressions.

[0161] In some configurations, the data processor 208 can compute load variability by determining a difference between a maximum power value and an average power value over a predefined time window. The data processor 208 can perform a fast Fourier transform (FFT) on the electric waveform data to extract frequency-domain characteristics of the power signal. For example, the data processor 208 can apply an FFT to direct current (DC) waveform data obtained from a power supply unit (PSU) to identify harmonic components or transient power events associated with processing unit operations. Harmonic components can refer to frequency components of a signal that are integer multiples of a fundamental frequency. Transient power events can refer to changes in power level or voltage that occur over relatively short time intervals (e.g., less than 100 milliseconds or less than 10 milliseconds). Each transient power event can indicate changes in load characteristics or potential faults in a power delivery path. The identified harmonic components can be provided to one or more other components of the data processing system 150 (e.g., the power predictor 212 or the power condition detector 218) to detect power conditions indicative of processing unit load changes or pending power excursions, perform forecasting, or determine at least one action to execute, for example.

[0162] In some configurations, the data processor 208 can aggregate power measurements from a plurality of sources to estimate a total power consumed by a rack cabinet or a plurality of rack cabinets. In some implementations, the data processor 208 can align timestamps associated with electric waveform data samples received from different measurement circuits to facilitate correlation of power consumption across the data center 201, e.g., correlate power consumption at various points in a power chain of the data center 201 and power consumption by a server or one or more processing units. The data processor 208 can compute a rate of change of power consumption by determining a difference between consecutive power samples and dividing by a time interval between the samples, for example. The data processor 208 can provide processed metrics to the model manager 210 or the power predictor 212 for use in forecasting or optimization operations. The data processor 208454913-1418-6318.1Docket No. 102555-0261can process other types of data received, stored, or otherwise obtained by the data processing system 150.

[0163] The model manager 210 can train, validate, deploy, or otherwise manage artificial intelligence (Al) models used by the data processing system 150. The Al models can include or be referred to as machine learning models, neural network models, predictive models, computational models, statistical models, or deep learning models, among others. In some cases, the model manager 210 can manage Al models for other data processing systems or devices such as for power forecasting and optimization within the data center 201.

[0164] The model manager 210 can receive electric waveform data, processed data, metadata, or other data from other components of the data processing system 150. Metadata can include information describing characteristics, properties, attributes, or context associated with data items, files, documents, or digital content. Metadata can include compute resources scheduled or to be executed for operation(s) of the one or more processors. The model manager 210 can generate training datasets for the Al models using the received data. For example, the model manager 210 can collect historical power consumption metrics (e.g., power level or RMS values), workload characteristics, or operational parameters from the data repository 222, as examples. The model manager 210 can organize the data into feature vectors suitable for training an Al model (or neural network). In some configurations, the model manager 210 can iteratively train one or more neural networks using supervised learning techniques based on labeled training examples that associate input features with target power consumption values, for example. The model manager 210 can utilize other nonlimiting techniques to train the neural network(s) such as unsupervised learning techniques, semi-supervised learning techniques, reinforcement learning techniques, or transfer learning techniques, among others.

[0165] The model manager 210 can deploy the Al model to perform forecasting or prediction. In some cases, the model manager 210 can provide the Al model to other components of the data processing system 150 for deployment, e.g., the power predictor 212 can execute at least one Al model to perform power consumption prediction or forecasting. The model manager 210 can utilize the results from prediction and feedback data (e.g., measured power consumption) to iteratively train the Al model. For instance, the model manager 210 can execute a training procedure to compare predicted data for a time window and the actual data measured at the time window, such as by computing residuals between464913-1418-6318.1Docket No. 102555-0261forecasted power consumption values generated by a neural network and corresponding actual power measurements obtained from electric waveform data captured during that same time window, among others. The model manager 210 can train the model using other types of operations or procedures.

[0166] The model manager 210 can partition training data into training sets and validation sets and evaluate model performance on the validation sets to determine whether the model achieves an accuracy threshold. In some implementations, the model manager 210 can update model parameters stored in the data repository 222 subsequent to detecting that a validation accuracy exceeds a predefined threshold, for example. The model manager 210 can deploy a trained model by providing model parameters to the power predictor 212 or the controller 220 for use in real-time power forecasting or control operations. The model manager 210 can deploy the trained model to other components of the data processing system 150 or other external devices. The model manager 210 can perform validation on the model prior to deployment. The model manager 210 can monitor model performance by comparing predicted values (e.g., predicted power consumption) with measured values (e.g., measured or computed power consumption). The model manager 210 can retrain the model when a performance metric degrades below a predefined threshold, e.g., the difference between the predicted values and measured values is greater than or equal to the predefined threshold.

[0167] In some cases, the model manager 210 may retrain the model subsequent to the performance metric degrading below the predefined threshold for a predefined number of predictions, the performance metric degrading below the predefined threshold for a number of predictions within a predefined time window, or after receiving an instruction from an external device to execute the retraining procedure. In some other cases, the model manager 210 can obtain or receive trained model provided by other device(s) such as another data processing system or a client device. The model manager 210 can provide trained model to other device(s). The model manager 210 can store and retrieve model(s) from the data repository 222 or from an external storage device.

[0168] The power predictor 212 can execute one or more Al models to forecast power consumption, for instance, by processing units or rack cabinets over a future or subsequent time window from a current time. The power predictor 212 can receive input data from other component s) of the data processing system 150, e.g., electric waveform data from the data collector 204, at least one power envelope from the power envelope identifier 206, processed474913-1418-6318.1Docket No. 102555-0261electric waveform data from the data processor 208, etc. The power predictor 212 can retrieve one or more models (or neural networks) from the data repository 222 and / or model parameters from the data repository 222 to perform a prediction operation. The model parameters can include weights of neural network layers, bias values, activation functions, or hyperparameters, to name a few.

[0169] For example, the power predictor 212 can obtain historical metrics over a plurality of time windows. The power predictor 212 can execute a model to identify a pattern according to the historical metrics. The model can generate an output vector representing characteristics of power consumption over at least one subsequent time window. The output vector can include multiple values representing predicted changes in power consumption level over the at least one time window.

[0170] In some configurations, the power predictor 212 can deploy multiple models or neural networks to perform a prediction. For example, the power predictor 212 can obtain (historical) metrics computed over a plurality of time windows and provide the metrics as input to a first neural network to generate an output vector. This output vector can represent characteristics of power consumption over the time windows. The metrics over the time windows can represent relatively long-term power samples. The power predictor 212 can select a subset of the metrics corresponding to a most recent time window and provide the subset as input to a second neural network to generate a second output vector representing characteristics of power consumption over the most recent time window. The subset of the metrics over the most recent time window can represent relatively short-term power samples. The power predictor 212 can provide the first output vector and the second output vector as input to a third neural network to generate a third output vector representing a combined representation of short-term and long-term power consumption patterns. For example, the power predictor 212 can execute a decoder neural network that receives the third output vector as input and generates forecast values representing predicted power consumption by one or more processing units over a predicted (e.g., second) time window subsequent to the plurality of time windows (e.g., historical time windows). The power predictor 212 can generate forecast values representing power consumption. The power predictor 212 can generate the forecast values at a resolution corresponding to a window length equal to at least one cycle of an AC waveform or a portion of a cycle of an AC waveform.484913-1418-6318.1Docket No. 102555-0261

[0171] In some implementations, the power predictor 212 can generate a prediction interval by computing an upper percentile value and a lower percentile value of a predicted power distribution to indicate a range within which actual power consumption may occur with a specified probability. The power predictor 212 can provide the forecast values to the action manager 214 to determine one or more actions to be performed such as parameter adjustment. The power predictor 212 can provide the forecast values to other component(s) of the data processing system 150 to perform their respective functionalities.

[0172] The action manager 214 can determine one or more actions to perform based on power consumption forecasts, power envelope, detected power conditions, optimization objectives, or a controller state. The action manager 214 can receive forecast values from the power predictor 212 and compare the forecast values with a power envelope value associated with a rack cabinet or at least one processing unit. For example, the action manager 214 can determine that a first amount of power predicted for a processing operation by one or more processing units of a rack cabinet exceeds a subset of provisioned power allocated to the rack cabinet and select an action to reduce power consumption at the rack cabinet or adjust a power cap (e.g., power limit) of the one or more processors. The action manager 214 can adjust the amount of power provisioned to the rack cabinet based on the predicted amount of power for the processing operation of the one or more processing units.

[0173] The action manager 214 can generate a signal comprising one or more parameters to adjust for at least one processing unit, a PDU, a PSU, or other components of the data center 201. For example, the action manager 214 can generate a signal to adjust at least one of a frequency of a processing unit, a voltage of a processing unit, a power limit of a processing unit, a frequency of a memory in electrical communication with the processing unit, or a capacity of the memory, to name a few. The action manager 214 can select an action to reallocate compute resources from one rack cabinet to at least one other rack cabinet, for instance, based on a determination that a difference between a first amount of power available for consumption at the rack cabinet and a second amount of power available for consumption at the other rack cabinet exceeds a predefined threshold. In some implementations, the action manager 214 can select an action to adjust a scheduling of workload across rack cabinets to distribute processing operations such that power consumption at individual rack cabinets remains within respective power envelopes. The494913-1418-6318.1Docket No. 102555-0261action manager 214 can provide the selected action to the controller 220 for execution. The action manager 214 can select other actions not limited to those discussed herein.

[0174] The controller state manager 216 can manage a controller state associated with a power envelope enforcement operation performed by the controller 220. The controller state can represent a current operational mode of the controller 220, wherein the controller 220 can execute a feedback control loop that adjusts power limits for one or more processing units to maintain total power consumption within a power envelope value. In this case, managing the controller state can include at least one of, but not limited to, maintaining, updating, or transitioning the controller state. In some cases, the controller state can represent the state or current operational mode of one or more processing units themselves, such that the controller 220 can adjust one or more parameters of the processing units according to satisfy or meeting the desired controller state selected by the controller state manager 216. The controller state manager 216 can determine one or more state variables based on total power estimated for a control iteration and a power envelope value identified for the control iteration. For example, the controller state manager 216 can determine a headroom value by computing a difference between a power envelope value and the total power consumed by one or more components of the data center 201. The headroom value can indicate whether the total power satisfies the power envelope value or may exceed the power envelope value. The headroom value can refer to a remaining amount of provisioned power or an unused amount of power of the power envelope, for example.

[0175] In some implementations, the controller state manager 216 can determine state variables comprising at least one of a data integrity flag, a moving average of total power, a rate of change of total power, or a time to excursion. The controller state manager 216 can select a controller state from a plurality of controller states based on the one or more state variables and the headroom value. For example, the controller state manager 216 can select an idle state based on a data integrity flag indicating that (new) electric waveform data may be missing or invalid. The controller state manager 216 can select a regulate state based on the headroom value falling below a positive threshold while remaining above a negative threshold. The controller state manager 216 can select a ramp-down state based on a time to excursion falling below a predefined threshold. The controller state manager 216 can select an excursion state based on the headroom value falling below a negative threshold. The controller state manager 216 can select a ramp-up state for other scenarios such as based on at504913-1418-6318.1Docket No. 102555-0261least one of the headroom value exceeding a positive threshold or a moving average of total power exceeding a predefined threshold. In some implementations, the controller state manager 216 can apply a cooldown period between state transitions to prevent rapid oscillation between controller states. The controller state manager 216 can provide the selected controller state to the controller 220 to inform a power limit adjustment operation. The controller state manager 216 can perform other non-limiting operations for managing the controller state.

[0176] The power condition detector 218 can monitor electric waveform data to detect one or more power conditions based on an analysis of the electric waveform data. The power condition can refer to or define specific electrical states or events that can impact data center operation or processing operations of the one or more processing units. The power condition detector 218 can utilize electric waveform data from devices distributed across a power chain of the data center 201 and electric waveform data measured for at least one of the rack cabinets to detect one or more power conditions.

[0177] For example, power condition detector 218 can receive first electric waveform data measured at a plurality of devices distributed across a power chain of the data center 201. The power condition detector 218 can receive second electric waveform data measured for at least one rack cabinet of rack cabinets in the data center 201. The first electric waveform data can comprise AC data such as from a PDU. The second electric waveform data can comprise DC data from a power input port associated with one or more processing units, for example. The power condition detector 218 can correlate the first electric waveform data and the second electric waveform data by aligning timestamps between the first electric waveform data and the second electric waveform data. With the timestamps aligned between the first electric waveform data from the devices across the power chain and the second electric waveform data power measured at the one or more processors, the condition detector 218 can determine a relationship between changes in the first electric waveform data and changes in the second electric waveform data in a same time window to correlate the first electric waveform data and the second electric waveform data.

[0178] The power condition detector 218 can detect at least one power condition based on the correlation of the first electric waveform data and the second electric waveform data. For example, the power condition detector 218 can detect that a load variability corresponding to a difference between a maximum power and an average power over a time514913-1418-6318.1Docket No. 102555-0261window exceeds or equals a threshold. The power condition detector 218 can detect a fluctuation in power at or above a magnitude threshold or rate-of-change threshold. The power condition detector 218 can detect that an amount of power available for consumption may be less than a threshold based on a difference between a first amount of power provisioned for the rack cabinet and a second amount of power consumed for processing operations by the one or more processing units of the rack cabinet.

[0179] In some implementations, the power condition detector 218 can detect that an amount of power being consumed has increased over a threshold relative to an amount of power provisioned for at least one of the rack cabinets of the data center 201, a subset of the rack cabinets of the data center 201, the rack cabinet, at least one of a plurality of shelves, or one or more processing units of at least one of the plurality of shelves. The threshold relative to the amount of provisioned power can correspond to the amount of power provisioned minus a margin value, which can be predefined or set. The power condition detector 218 can provide an indication of the detected power condition to the action manager 214 to initiate one or more actions. For instance, the action manager 214 can select an action according to the type of power condition detected such that different power conditions are mapped to different actions associated with at least one of (i) the devices distributed across the power chain or (ii) one or more processing units of the rack cabinet. The action manager 214 can provide an indication of the selected action to the controller 220 to initiate the action.

[0180] The controller 220 can execute control actions to adjust one or more parameter such as power limits, power allocations, or other operational parameters of processing units or power distribution components within the data center 201. The controller 220 can execute control actions on other components local to or remote from the data center 201. The controller 220 can receive an action selected by the action manager 214 or a controller state selected by the controller state manager 216, for example. The controller 220 can generate an instruction to update at least a current power limit of one or more processing units based on the action or the controller state.

[0181] For example, the controller 220 can compute a second power limit to request for one or more processing units of a server based on a controller state, a total power, a power envelope value, or a current power limit of the one or more processing units. In some implementations, the controller 220 can determine and adjust a rate of increase from the current power limit to the second power limit when the controller state may be a ramp-up524913-1418-6318.1Docket No. 102555-0261state. In some implementations, additionally or alternatively to controlling DVFS or adjusting hardware power cap, the controller 220 can apply a workload-level actuator that throttles workload iteration timing (sometimes referred to as virtual frequency capping) to reduce power consumption responsiveness such as when hardware-based control latency is undesirable or insufficient. The control latency being insufficient can refer to the control latency being greater than or equal to a latency threshold.

[0182] The controller 220 can generate a signal comprising instructions to adjust one or more parameters associated with a corresponding component. For example, the one or more parameters can include at least one of a frequency of one or more processing units, a voltage of one or more processing units, a power limit of one or more processing units, a frequency of a memory in electrical communication with the one or more processing units, a capacity of the memory, or other parameters for DVFS. The controller 220 can select a parameter to adjust based on the controller state selected by the controller state manager 216, a response latency associated with each parameter, or a combination thereof.

[0183] The controller 220 can determine the second power limit for adjustment using a proportional derivative control based on at least one of a measured power, a margin, or the power envelope value such as depending on the controller state (e.g., the controller state may be a regulate state, a ramp-down state, or an excursion state). For example, the controller 220 can compute an error signal by subtracting the measured power and the margin from the power envelope value and apply proportional and derivative gains to the error signal to compute a change in power limit. The controller 220 can clip the second power limit to a range defined by a minimum power limit and a maximum power limit supported by a processing unit or specified by a user. In some implementations, the controller 220 can quantize the second power limit to discrete increments such as multiples of 5 watts to reduce a rate at which power limit changes are requested. In some configurations, the controller 220 can apply a cooldown period between instructions to update the power limit to prevent rapid oscillation of power limits. For example, the controller 220 can delay transmission of a new instruction until a predefined time interval has elapsed since transmission of a previous instruction. The controller 220 can generate an instruction during the cooldown period when the controller state may be an excursion state or a ramp-down state to respond rapidly to a detected or predicted power excursion.534913-1418-6318.1Docket No. 102555-0261

[0184] In some implementations, the controller 220 can transmit the instruction to one or more processing units via a control interface to adjust a power limit of a GPU, a CPU, or other types of processing units executing a processing operation. In some cases, the controller 220 can transmit the instruction to a PDU to adjust an output power limit for one or more output stages of the PDU or to adjust an allocation of available power from the PDU among a plurality of loads (e.g., among different rack cabinets) powered via the PDU. The controller 220 can initiate other types of control or adjustment for the data processing system 150, component(s) of the data center 201, component s) of the utility grid 100, etc. The features or operations of the data processing system 150 or components of the data processing system 150 can be described in conjunction with at least one of, but not limited to, FIGS. 22-26.

[0185] The data repository 222 can maintain information local to the data processing system 150. Information stored in the data repository 222 can be accessible by the components of the data processing system 150. The data repository 222 can be referred to as a data storage, database, memory device, etc. The data repository 222 can include data storage 224 to maintain electric waveform data, processed metrics, power consumption measurements, or other data collected from measurement circuits within the data center 201. For example, the data storage 224 can store electric waveform data measured at a plurality of devices distributed across a power chain of the data center 201 or electric waveform data measured for rack cabinets in the data center 201. The data storage 224 can store forecasting data including prediction values such as predicted power consumption data. The data storage 224 can store data received via the interface 202. The data storage 224 can store other types of data not limited to those discussed herein.

[0186] The data repository 222 can include operation characteristic storage 226 to maintain characteristics of operations executed on processing units. The characteristics of operations can include at least one of, but not limited to, workload types, service level agreements (SLAs), priority levels, or data flows received by processing units. The workload types can include, for instance, machine learning training operations, inference operations, batch processing operations, stream processing operations, or other processing tasks. The SLAs can refer to contractual or policy-defined objectives that specify performance targets such as maximum response time, minimum throughput, or availability percentage for processing operations executed by the one or more processing units. Different rack cabinets, servers, or processing units can be associated with (or assigned to) different SLAs. The544913-1418-6318.1Docket No. 102555-0261priority levels can refer to numerical or categorical rankings assigned to processing operations such as for determining scheduling order or resource allocation precedence among competing tasks. The data flows can include sequences of data packets, stream records, or structured data objects received by the one or more processing units for processing. The data flow can carry metadata indicating processing criteria or expected computational intensity, which can be used for forecasting workload, compute resource utilization, or power consumption, for example. In some implementations, the operation characteristic storage 226 can maintain information about compute resources allocated to processing operations, job scheduling parameters, or performance metrics associated with processing unit operations.

[0187] The data repository 222 can include model storage 228 to store or maintain one or more Al models. In some cases, the model storage 228 can store Al models to be trained by the model manager 210. The model storage 228 can store Al models trained by the model manager 210. The Al models can be accessed and used by the power predictor 212 for forecasting power consumption or by other components of the data processing system 150 to perform their respective operations. The data repository 222 can store parameters of Al models. For example, the model storage 228 can maintain weights of neural network layers, bias values, activation functions, or hyperparameters associated with one or more neural networks.

[0188] The data repository 222 can include parameter storage 230 to maintain configuration parameters for power management operations, such as power envelope values, threshold values for state transitions, cooldown periods, or quantization increments. The parameter storage 230 can store parameters for adjustment such as frequency settings, voltage levels, power limits, memory capacity configurations, or scheduling parameters for compute resources, to name a few. In some cases, the parameter storage 230 can maintain user-defined limits on power allocations, minimum and maximum power limits for processing units, or constraints on resource allocations. The parameter storage 230 can store other parameters associated with the component(s) of the data center 201 or parameters of the controller 220.

[0189] The data repository 222 can include controller state storage 232 to maintain controller states selected by the controller state manager 216. The controller state storage 232 can store state variables computed during control iterations. The controller state storage 232 can store historical records of state transitions. For example, the controller state storage 232 can maintain a current controller state for each rack cabinet or each processing unit, a554913-1418-6318.1Docket No. 102555-0261headroom value associated with each control iteration, or a time to excursion computed for each control iteration. The controller state storage 232 can store the types of controller states such as at least one of idle, ramp up, regulate, ramp down, or excursion states, along with associated threshold values and transition criteria for each state. The controller state storage 232 can include other states or criteria associated with the states.

[0190] The data repository 222 can include action storage 234. The action storage 234 can maintain actions selected by the action manager 214. The action storage 234 can store instructions generated by the controller 220. The action storage 234 can store records of adjustments made to power limits or operational parameters. In some implementations, the action storage 234 can maintain feedback signals received from processing units or devices after execution of control actions to facilitate evaluation of the effectiveness of the action. The action storage 234 can store other types of actions for operating the component(s) of the data center 201, not limited to those discussed herein. For instance, the action storage 234 can include one or more actions to be executed by the data processing system 150, the PDU, the PSU, one or more servers of a rack cabinet, the one or more processors of the rack cabinet, or a cooling system of the data center 201. The data repository 222 can store or maintain other non-limiting data. In some configurations, the data repository 222 may store data for external device(s). In some cases, the data processing system 150 can store at least a portion of the data discussed herein in a remote data repository.

[0191] FIG. 3 illustrates an example diagram 300 of changes in power utilization with increased optimization or functionalities of the systems and methods, in accordance with an implementation. The diagrams 300 include a plurality of subplots 302-308. Each bar in the diagram 300 can represent the total provisioned power and the allocation of the provisioned power, including the average utilization of the provisioned power, the maximum utilization of the provisioned power, load variability (or difference) between the maximum and the average load, and the reserved power. The load variability may be a driver of underutilized provisioned power. As shown, the subplot 302 can include a relatively large load variability. To optimize provisioned power, it may be desired to operate more consistently at the maximum load. The data center power strategies may compute the amount of reserve power on the expected load variability, hence, reducing load variability can have a compounding effect on overall power utilization. The provisioned power can be utilized for one or more racks in the data center 201, each of the racks including a plurality of processing units,564913-1418-6318.1Docket No. 102555-0261storage units, or devices, among others. The racks can include, correspond to, or be a part of at least one server.

[0192] A system roadmap can be described in conjunction with at least FIG. 3, including stages corresponding to respective subplots 304-308. The subplot 304 can include an example provisioned power with reduced variability per node. For example, the data processing system 150 can perform local node (rack-level) optimization of load variability using DVFS. In this example, the system objective can be distributed (across multiple racks), and each rack can optimize its performance independently. Optimization of load variability at the rack level may be achieved using power commands. The data processing system 150 can learn a response based on or according to compute-load characteristics to reduce load variability. Local node optimization (or rack-level optimization) can be performed at a per-node scale, accounting for local differences in computation, network, and power setups.

[0193] The data processing system 150 can learn a DVFS response based on computeload characteristics which reduces load variability. Learning the DVFS response based on the compute-load characteristics can refer to or involve iteratively training a model (e.g., AL based model or machine learning model) based on the DVFS response and the compute-load characteristics. The optimization can be performed at a per-node (e.g., per-rack) scale, which can account for local differences in at least one of but not limited to computation, network, or power setups.

[0194] The subplot 306 can include an example provisioned power with power sharing between nodes. The data processing system 150 can perform hierarchical optimization. As part of the hierarchical optimization, power can be shared or allocated between nodes. The data processing system 150 can control the power levels through DVFS at the node level and higher-level (e.g., hierarchical) control points that manage power sharing between nodes. For example, power sharing can be allowed between nodes. Power levels can be controlled at the node level and higher-level control points that manage power sharing. The data processing system 150 (e.g., higher-level system) can lower the power consumption of one node to increase another node, which then is taken into account at the edge nodes in their DVFS response. Changing the allocation of power between different nodes can lower or reduce the effective maximum utilization, thereby achieving lower load variability in the power system according to the load balancing between nodes. In this case, learning (e.g., training,574913-1418-6318.1Docket No. 102555-0261validating, or deploying a model) can be conducted at the edge node and higher in the hierarchy.

[0195] The subplot 308 can include an example provisioned power with utilization of energy storage. The data center 201 can include at least energy storage, cooling, or other power systems. The data processing system 150 can communicate with or receive information from the one or more power systems at the data center 201. The system structure and hierarchy can be similar to the hierarchical optimization technique. In this case, the data processing system 150 can utilize the energy storage as, at least in part, the mechanism for power sharing, for instance, instead of compute loads (directly). The utilization of energy storage system, e.g., by way of power sharing, can (further) reduce the effective maximum utilization and load variability. The data processing system 150 can utilize the cooling systems in the overall optimization and control objective. By including these factors and control points, the effective max utilization and load variability can be reduced. The overall reserved portion of the provisioned power can be reduced by implementing the various optimization techniques discussed herein.

[0196] The systems and methods can include a hierarchical power distribution system. The hierarchical structure of the data center 201 (or other types of facilities) may be similar to (or in some cases mirror) the hierarchical structure of the power distribution system.Examples of data center power distribution systems can be described herein in conjunction with at least one of but not limited to FIGS. 4-6. FIGS. 4-5 illustrate block diagrams 400, 500 of example data center power distribution systems, in accordance with an implementation. The block diagram 400 can represent an overview of a data center power distribution system shown in the block diagram 500. The data center power distribution system of FIGS. 4-5 can be a part of the data center 201.

[0197] As shown in FIG. 4, a utility power source 402 (e.g., one or more components of the utility grid 100) can distribute or supply power to one or more racks (e.g., rack cabinets), such as a traditional rack 404 or open compute project (OCP) rack 406, via one or more power distribution paths. A rack cabinet can include or correspond to a physical enclosure configured to house a plurality of shelves supporting one or more servers, each of the servers can be associated with one or more processing units configured to execute processing operation of or for the server. The rack cabinet can receive and distribute electrical power for executing processing operations within the data center 201. Power can be distributed to the584913-1418-6318.1Docket No. 102555-0261traditional rack 404 via a (first) path with at least one central uninterrupted power supply (UPS). The UPS system along the path can provide backup power to the traditional rack 404 in the data center 201. Power can be distributed to the OCP rack 406 via a (second) path with no UPS. For instance, the OCP rack 406 can receive power supplied directly from the utility power source 402, a generator, or one or more intermediate power distribution or management components, without the UPS as an intermediary, for example. In some arrangements, reference to the rack cabinet(s) can refer to one or more of the traditional rack(s) 404 or the OCP rack(s) 406.

[0198] In the hierarchical structure of the data center 201, as shown in at least one of FIG.5, power can flow from the utility feed (e.g., utility power source 402 or utility grid 100) to at least an automatic transfer switch (ATS) for power distribution throughout the data center 201. A generator (e.g., electric generator) can be integrated with the ATS. The ATS can pass the AC current through a UPS to distribute power to one or more racks, e.g., servers, storage network equipment, etc. The UPS can be electrically coupled to one or more UPS batteries to supply power in scenarios where power is not received from the utility power source 402 or the ATS, for example. In some cases, a PDU can provide power to one or more server level power supply units (PSUs) or a series of PSUs which feed a common rack bus. The ATS can distribute power to other devices, components, or equipment throughout the data center 201, including but not limited to at least one chiller device, at least one cooler device, or at least one climate control device, among others.

[0199] In various implementations, the data processing system 150 can be implemented or distributed throughout the data center 201, including in the power supply, on the rack, or within the PDU. The data processing system 150 can facilitate data exchange throughout the data center 201. With relatively high speed communication interfaces, such as Infiniband, ethernet, PCIe, or others, the platform of the data processing system 150 can provide visibility, insight, and control on a specific processing unit-level or on a broader site-wide scale. The data processing system 150 can include an orchestrator, which can be dynamically centralized or distributed, can coordinate various worker nodes which are performing the measurement, analysis, and action of the power management.

[0200] As part of the data center- wide orchestration, the data processing system 150 can learn or acquire information related to grid conditions external from the data center 201. For example, the data processing system 150 can obtain electrical distribution data at multiple594913-1418-6318.1Docket No. 102555-0261points upstream or downstream from a power transformer of the utility grid 100, such as described in conjunction with at least FIG. 6. In some cases, the data processing system 150 can obtain the data external from the data center 201 from other data processing systems on the utility grid 100. The data processing system 150 can integrate the grid condition information, e.g., value of electricity or power quality, to inform or manage the control signals to one or more components of the data center 201 (or in some cases the utility grid 100) to achieve a desired power consumption objective.

[0201] FIG. 6 illustrates a block diagram 600 of an example power distribution to the data center 201, in accordance with an implementation. The data center 201 can include one or more components or electrical connections similar, additional, or alternative to those described in conjunction with at least FIGS. 4-5. Power can be distributed to the data center 201 from the utility power source (e.g., 402) via a power transformer. The data center 201 can include various racks, rows of racks, pods, or equipment powered at least in part by electricity from the utility power source.

[0202] For power optimization techniques, the data processing system 150 can collect (measurement) data from a plurality of points or locations throughout the electrical distribution networks. For example, the data processing system 150 (e.g., one or more DPS modules) can be configured to obtain data from upstream of the power transformer, downstream of the power transformer (e.g., between the site of the data center 201 and the power transformer), the ATS, one or more outlets at the data center 201, the one or more racks (e.g., power input or output), the components within individual racks, etc. The data processing system 150 may be installed at the locations where the measurements are obtained. The data processing system 150 may receive data from a measurement device, a sensor, or a metering device installed at the measured location. The data processing system 150 can perform analysis and control one or more components of the data center 201 according to or based on the received measurement data.

[0203] FIG. 7 illustrates a bar graph 700 of an example change in data center microgrid efficiency, in accordance with an implementation. The bar graph 700 can include a first bar representing an example baseline power consumption and reserve power. The bar graph 700 can include a second bar representing an example power consumption and reserve power when implementing, deploying, or integrating the data processing system 150 (or a plurality of data processing systems) at different levels of the hierarchical system, including base604913-1418-6318.1Docket No. 102555-0261command DVFS. The second bar graph can reflect an example result from the deployment of the data processing system 150 for analyzing the power distribution and consumption at various points in the hierarchy to control one or more components within the data center 201 (or external components from the data center 201). As shown, with the data processing system 150 and the base command DVFS utilized for data analysis (of measured power) and providing controlled responses, the base power can be maximized or increased for additional compute load and reserve power can be reduced to avoid excess resource. For example, with the integration of the data processing system 150, a certain amount of gain can be obtained at each level of the hierarchy, e.g., a gain at the rack, a gain at the row, a gain at the section, a gain on the overall site, etc. Individual gains from the integration of the data processing system 150 can aggregated into additional compute capacity, thereby minimizing the reserved power (e.g., excess power). The reduction of the provisioned power and optimization of power utilization can be described in conjunction with at least FIG. 3.

[0204] The systems and methods can implement edge nodes for power optimization. The core of the edge nodes can be based on the platform of the data processing system 150 (e.g., system platform discussed herein), capable of at least measuring electrical signals, performing significant compute and optimization operations, and executing controls.Example implementations of the data processing system 150 can be provided herein to provide different types of functionalities utilized to implement or execute the overall optimization. The system platform can be a suitable compute or communication platform to serve as an intermediate node in a hierarchical system.

[0205] In certain systems, power management equipment can operate within the alternative current (AC) domain. The data processing system 150 can provide techniques for analyzing the AC waveform (e.g., real-time waveform analysis) which can be applied to loads within the data center 201 (or other facilities). For example, the data processing system 150 can leverage one or more Al models to analyze electric waveform data in real-time. Analyzing the electric waveform in real-time can refer to analyzing the electric waveform responsive to receiving measured or sensed electric waveform from the metering device 118 or the sensor capturing the electrical data. The data processing system 150 can receive insights into power consumption patterns according to the analysis and identify opportunities for optimization.

[0206] An example real-time mapping of the AC power consumption and total GPU614913-1418-6318.1Docket No. 102555-0261consumption during a series of LLM queries can be described in conjunction with at least FIG. 12. For example, FIG. 12 illustrates a graph 1200 of an example mapping of AC power consumption and total GPU consumption, in accordance with an implementation. The waveforms associated with AC power consumption and total GPU (power) consumption can be mapped on the graph 1200. For instance, the AC power consumption can be represented by waveform 1202 and the total GPU consumption can be represented by waveform 1204. The data processing system 150 can utilize the mapping of the AC power consumption and total GPU consumption for optimization of data center power utilization.

[0207] Additionally or alternatively, in certain deployments, such as power supplies, the data processing system 150 can obtain or measure the direct current (DC) power. The DC power may not exhibit the sinusoidal behavior of an AC waveform. Instead, the DC power can exhibit other characteristics which can be measured at a relatively high resolution (e.g., 1 kHz, 5 kHz, or 10 kHz). Bridging across these domains can allow for visibility into the (direct) usage of the workload (on the DC) and the relatively broader electrical flows throughout the data center 201 (on the AC). By connecting these domains, and integrating information related the workload (e.g., compute load), the data processing system 150 can provide enhance insight and analysis of the real-time activity of the operating environment associated with the data center 201. The visibility into the DC and AC power can be described in conjunction with at least FIG. 9. In some configurations, the data processing system 150 may execute one or more functions for processing the electrical power, including but not limited to fast Fourier transform (FFT), e.g., at least for the DC power data.

[0208] In various implementations, the data processing system 150 can utilize the electric waveform analyzed in real-time dynamic parameter adjustment. For example, subsequent to the analysis, the data processing system 150 can dynamically adjust the GPU operational parameters, such as but not limited to frequency and power levels according to the Al-driven analysis. The data processing system 150 can adjust the operational parameter to satisfy a power management criteria, including to ensure that power consumption aligns with computational demands. The data processing system 150 may adjust one or more operational parameters of other components, not limited to the GPU, for example.

[0209] Given the operating conditions of the data center 201, various components can be designed or configured for in-use replacement, e.g., components within the data center 210 can be swappable without affect the function of other in-use components. In such624913-1418-6318.1Docket No. 102555-0261architectures, components such as power supplies, battery back-ups and intelligence modules may be replaceable without interrupting the operation of the overall system. The data processing system 150 (or a processing unit of the data processing system 150) can be integrated within the replaceable or swappable devices such that a traditional power monitoring system can be configured with the capacity or capability to perform waveform analytics on the AC power and generate / produce fine-grained power telemetry of the DC loads, among other aspects.

[0210] By measuring and analyzing the electrical characteristics of various data center components, the data processing system 150 can disaggregate the underlying system components. For example, each device (e.g., CPU, graphics processing unit (GPU), fan, memory, or other devices) can exhibit different or distinctive electrical characteristics. On the AC and DC side of the power, the data processing system 150 can identify values of the power draw from the one or more components (e.g., how much power is being consumed). According to the power consumption, the data processing system 150 can parse the system performance at a fine-grained level.

[0211] In some configurations, the data processing system 150 may not perform the core workload of the compute within the data center 201. However, the data processing system 150 can have awareness or received information indicative of what the workload is, e.g., CPU workload, GPU workload, memory workload, etc. With workload information, the data processing system 150 can be configured to manage the power used by the compute. For example, with a trained forecasting model (e.g., trained or generated by the data processing system 150 or other systems) and an input signal of characteristics of the workload, the data processing system 150 can forecast the power consumption and execute one or more appropriate actions. The forecasting model can be referred to as Al model or predictive model. The actions may be predefined. The data processing system 150 may train and validate the forecasting model to achieve an accuracy of greater than a predefined threshold. The data processing system 150 can continuously or iteratively train the forecasting model to forecast power demands or needs and implement real-time control actions, e.g., adjusting operational parameter of one or more components (e.g., GPU(s)) or request additional power from the power source. The predictive capability can enhance power efficiency and resource allocation.

[0212] In some implementations, the data processing system 150 can include a distributed634913-1418-6318.1Docket No. 102555-0261control system for optimizing power usage at local (e.g., rack-level) and / or hierarchical levels. The data processing system 150 can leverage existing control affordances to perform the distributed control approach / technique, for example. In some configurations, the data processing system 150 can execute Kubernetes scheduling. For instance, the data processing system 150 can embed different Al workloads within Kubernetes nodes to improve the management and optimization of resource allocation and power consumption. With the dynamic parameter adjustment, predictive modeling, distributed control technique, and Kubernetes scheduling, the among others, the data processing system 150 can reduce load variability, allow for power balancing between loads, and / or optimize the total load with the potential inclusion of energy storage. The data processing system 150 can utilize, deploy, or employ machine learning techniques to adapt control responses to local conditions, optimize the system, and perform predictions for system optimization.

[0213] Operating a GPU-based data center can involve controlling the temperature of the chips to optimize performance and efficiency. The data processing system 150 can be leveraged for temperature management. The controls for temperature management can include on-board fans, room cooling (of various techniques), liquid cooling on the door / rack, at the chip-level, or full immersion. Additionally or alternatively to the GPU power consumption, excessive energy may be expended to maintain appropriate thermal conditions. By ingesting or obtaining the environmental variables (e.g., data, information, or parameters), the data processing system 150 can synchronize the environmental information with the realtime power consumption of the compute load to optimize overall system electricity consumption. Outputs of the data processing system 150 (e.g., system platform) can include, additionally or alternatively to sending commands to the compute (e.g., loads or components), sending commands to the cooling systems to reduce power consumption or increase cooling. The temperatures can swing over different time frames. The data processing system 150 can account for the temperature as part of an integration into the compute consumption to manage for overall performance of the power distribution and compute load.

[0214] The output controls from the data processing system 150 may be utilized as inputs for managing load within the compute system. In certain situations, the data processing system 150 may (actively) inject power from other available sources, such as UPS systems, battery back-ups, or standby generators, for instance, as part of managing the load.

[0215] Additionally or alternatively to the controls within the power system of the data644913-1418-6318.1Docket No. 102555-0261center 201, the systems and methods (e.g., data processing system 150) can interface with ongoing computational (e.g., compute) tasks. For example, The data processing system 150 can perform workload management and integrate Kubernetes scheduling. The workload management can involve coordinating Al workloads to avoid (or minimize) simultaneous peaks and adjusting workloads to reduce peaks. The data processing system 150 can improve management of the gap between reserved power and power actually used at different levels, e.g., GPU, server, rack, row, data center 201, etc. Kubernetes scheduling integration can involve embedding different Al workloads within Kubernetes nodes to manage and optimize resource allocation and power consumption more effectively. Integrating Kubernetes scheduling can reduce the spikier nature of Al workloads which lead to underused power and compute, for example.

[0216] FIG. 8 illustrates a block diagram 800 of an example power flow from the data center 201 for controlling one or more components, in accordance with an implementation. As shown, the power from the data center 201 can flow into at least the PDU, the PSU, and the compute load. The data processing system 150 (e.g., labeled as “DPS”) can read or measure the power across the PDU, PSU, and the compute load (e.g., GPU, CPU, memory, or other devices). The power measurement can include supply power and consumed power. With the visibility from power measurements, the data processing system 150 can analyze the conditions and behavior of the components, and transmit control signals as inputs to the components. Multiple data processing systems may be communicatively coupled with each other across different servers or racks in the data center 201. The data processing systems can share information with each other for power optimization. One or more of the data processing systems shown in FIG. 8 can include one or more components or functionalities of the data processing system 150 as described in conjunction with at least FIGS. 2A-B.

[0217] FIG. 9 illustrates a block diagram 900 of example components of a PSU controlled using measured electrical data, in accordance with an implementation. The block diagram 900 can include an example power flow across a PSU, such as across a transformer, a rectifier, a filter, a regulator, and a load (e.g., compute load). The PSU of FIG. 9 can be a part of or correspond to the PSU of FIG. 8, for example. The data processing system 150 can read or measure the power across the components of the power supply, such as power consumption of and power provision to the one or more components. The measured power can include AC and DC power. The data processing system 150 can provide visibility to the654913-1418-6318.1Docket No. 102555-0261measured AC and DC power. The AC and DC domains can be connected or bridged to allow visibility into the workload usage (e.g., on the DC) and the relatively broader electrical flows throughout the data center 201 (e.g., on the AC). The data processing system 150 can analyze the measured data and provide control signals according to the insight and analysis of the real-time activity of the operating environment associated with the data center 201.

[0218] FIG. 10 illustrates a block diagram 1000 of an example PDU, in accordance with an implementation. The PDU of FIG. 10 can be described in conjunction with the PDU of FIG. 8, for example. The block diagram 1000 can include an external PSU supplying power to one or more components of the PDU via a power supply jack input protection and a power rail. The power supply jack input protection can be utilized to protect the input of the PSU (e.g., the jack or connector where external power is supplied) from potential damage, e.g., from overcurrent or voltage spikes. The power rail can be configured to supply power to various components of the PDU, such as the control system power regulation, gigabit Ethernet physical layer (PHY), local area network (LAN), field-programmable gate array (FPGA), output stages, or loads (e.g., processing units). The control system power regulation can regulate or manage power level to the components of the PDU, e.g., adjusting voltage, current, or power flow based on feedback from sensors or control algorithms. The gigabit Ethernet PHY and LAN can enable communication between the PDU and other devices within, for instance, the network 140. The FPGA can be integrated into the PDU to handle various tasks, including but not limited to at least one of monitoring, controlling, or distributing power, for example.

[0219] The FPGA can receive power rail voltage information monitored by the control system power regulation. The FPGA can receive alerts or feedback from one or more output stages or devices, such as overcurrent alerts from a current meter. The output stages can be configured to output or send power to the loads. The output stages can include but is not limited to at least one of a power switch stage (e.g., power delivery control), voltage regulation stage (e.g., regulate voltage level), protection stage (e.g., prevent electrical faults), or monitoring and control stage (e.g., monitor or track power usage or perform network communication). The FPGA can provide output variables to the one or more output devices, including at least one of power levels, PSU health status, etc. The loads can receive electrical power from the PDU.

[0220] The data processing system 150 can receive data or measurements across different664913-1418-6318.1Docket No. 102555-0261components of the PDU. The measurements can be from one or more sensors or monitoring devices distributed within the PDU. For example, the data processing system 150 can receive power consumption data from at least one of the power rail, the control system power regulation, one or more output devices (e.g., measured power to the loads), or other components. With the measurements (e.g., received in real-time), the data processing system 150 can provide control signals to the FPGA for managing at least the power provision or distribution to the one or more compute loads. In another example, the data processing system 150 can measure power input to the PDU or the one or more components of the PDU. According to the measured power, the data processing system 150 can perform at least one action, e.g., control the power level allocated or sent to individual output stages.

[0221] FIG. 11 illustrates a block diagram 1100 of an example processing unit compute 1102, in accordance with an implementation. The block diagram 1100 can include the data processing system 150 and the processing unit compute 1102 (e.g., compute load). The processing unit compute 1102 can include or correspond to at least one of a GPU compute or CPU compute, among others. The processing unit compute 1102 of FIG. 11 may refer to the processing unit compute of FIG. 8, for example. The processing unit compute 1102 can receive data from at least the PSU. In some cases, the processing unit compute 1102 may receive data from batteries or an external PSU. In various cases, the processing unit compute 1102 can be referred to as a processing unit.

[0222] The processing unit compute 1102 can receive a flow of compute data from one or more client devices. The compute data can include data for processing by the processing unit, e.g., prompt from at least one client device, Al chat inference, token input length, priority level, video processing data, simulation data, or imaging data, to name a few. The compute data can be communicated to the processing unit from other devices within the network 140, internal to or external from the data center 201. The data processing system 150 can obtain at least a portion of the compute data flowing to the processing unit. The data processing system 150 can factor the compute data in with the models (e.g., model executed by or receiving resources from the processing unit), e.g., leveraging metadata to or from the processing unit, and control the models by sending the desired power level to the processing unit.

[0223] The data processing system 150 can obtain measured or sensed electrical data input to, consumed by, or output from the processing unit compute 1102. The data processing system 150 can obtain data from other points or locations across different components within674913-1418-6318.1Docket No. 102555-0261or outside of the data center 201. Based on the analysis of the measured information, the data processing system 150 can send control signals to the processing unit compute 1102 for power optimization. For instance, the data processing system 150 can increase or decrease the workload of the processing unit compute 1102 according to power distributions throughout the data center 201 or across different processing units, loads, racks, rows of racks, or pods. By increasing or decreasing the workload, the data processing system 150 can increase or decrease the power consumption by the respective processing unit. The data processing system 150 can adjust the increase or decrease in the workload to maximize power utilization and efficiency, and share the power between different processing units (or nodes) to reduce maximum utilization of power, thereby lowering the load variability. Lowering the load variability can reduce the reserve power stored, hence, reducing excess electrical resources.

[0224] Optimizing power utilization can be crucial for the sustainability and efficiency of data centers. The Al-driven approach which leverages the Karman platform can address the challenges posed by GPU-intensive workloads and the restrictions of certain power management strategies. By leveraging real-time waveform analysis, dynamic parameter adjustments, distributed control mechanisms, and Kubemetes scheduling, among others, power efficiency and resource allocation can be enhanced for data centers. It should be noted that the features or functionalities discussed herein are non-limiting examples and additional or alternative operations, components, or techniques may be utilized or implemented.

[0225] In various configurations, the data processing system 150 can be utilized for data center power management in various environments. The data processing system 150 can include or be implemented on a device or a (custom) module (e.g., system on module (SoM)) in a relatively compact size, such as around, larger than, or smaller than 58 mm by 43 mm. The SoM with open software can allow edge Al to be embedded in data center infrastructure. In some cases, the data processing system 150 can include or be a part of the metering device 118 (e.g., Al smart meter), allowing for edge Al solution. For example, the data processing system 150 can be embedded into an electric meter (e.g., metering device 118) located at the grid edge or a site to provide edge Al capabilities for utilities or grid-edge monitoring, additionally or alternatively to deployment in the data center 201.

[0226] FIG. 13 illustrates a graph 1300 of an example estimated data center capacity demand, in accordance with an implementation. As shown in graph 1300, an increase or growth in global data center power demand may be projected with increases in Al workload684913-1418-6318.1Docket No. 102555-0261and other workload. As an illustrative example, the Al workload may increase by around 39 % (e.g., compound annual growth rate (CAGR)) and other workload may increase by around 16 % over a number of years. There may be a disproportionate increase in Al workload compared to another workload. The increase in workload can translate to an increase in power demand from the data center 201. The systems and methods discussed herein can provide features or functionalities for power-aware scheduling, real-time control, or related improvements to allow for increased compute within existing infrastructure, thereby maximizing compute (or avoid underutilized compute) without extensive changes in infrastructure components, equipment, or devices, for example.

[0227] FIG. 14 illustrates a graph 1400 of an example compute between different power profiles, in accordance with an implementation. The graph 1400 can provide a comparison between an existing power utilization profile (e.g., non-DPS-enabled power profile or a current power profile without leveraging the data processing system 150) and a power-optimized profile enabled by the data processing system 150 (e.g., DPS-enabled power profile). The compute generated or utilized by the profiles can be plotted as shown in the graph 1400.

[0228] For example, plot 1402 can represent an example compute from utilizing the non-DPS-enabled power profile and plot 1404 can represent an example compute from utilizing the DPS-enabled power profile. As shown, the data processing system 150, utilizing the DPS-enabled power profile, can provide microsecond power visibility and predictive controls to allow for a relatively higher utilized compute with the same provisioned power. The microsecond power visibility and predictive controls can unlock various categories of underutilized power. Examples of the categories can include the use of power-aware scheduling to maximize compute within a power budget (e.g., allocated power) of an entity or facility, and safe operation at provisioned power, thereby eliminating existing design margin predefined or built-in to compensate for historically lack of visibility and controls. By utilizing the DPS-enabled power profile from the data processing system 150, the compute can be increased above the design maximum load without extending beyond the provisioned power.

[0229] FIG. 15 illustrates an example diagram 1500 of end-to-end processing resolution, in accordance with an implementation. The diagram 1500 can include a plurality of functional blocks (e.g., one or more components of the data processing system 150)694913-1418-6318.1Docket No. 102555-0261configured to operate at a sub-millisecond speed and high resolution. The functional blocks of the diagram 1500 can be individual components or parts of the data processing system 150. For example, the functional blocks can include an analog-to-digital converter (ADC), sample transmitter, data service component, Al manager, Ethernet communications interface, and a power-aware scheduling component. At least one of the data service component, Al manager, or the Ethernet communications interface can be a part of or in communication with the SoM of the data processing system 150. At least one of an Ethernet communications interface or the power-aware scheduling component can be a part of or in communication with a control server. As shown, operations from associated with the SoM, such as at least the ADC, sample transmitter, data service component, Al manager, and Ethernet communications interface can operate at a sub-millisecond speed, e.g., less than 1 millisecond. The components herein can be a part of an end-to-end processing of the data processing system 150, allowing for realtime control decisions. Operating at the sub-millisecond speed can allow for high resolution, low-latency visibility, reliable and actionable control, and safe fallback and guard bands.

[0230] FIG. 16 illustrates edge Al platform 1600, in accordance with an implementation. The platform 1600 can be integrated or implemented in the data processing system 150. In some cases, the platform 1600 can include one or more independent (or external) components in communication with the data processing system 150. The platform 1600 can include hardware, software, or a combination of hardware and software components. Although components of the platform 1600 can be shown in FIG. 16, it should be noted that the platform 1600 can include additional or fewer components, not limited to those presented herein.

[0231] The platform 1600 can include optimized low latency pipeline allowing support of at least 16 channels of at least 1 MHz sampled power data into the processor. In some cases, the pipeline can provide the sampled power data into a processor (e.g., processing unit pipeline such as a GPU-accelerated pipeline) for low-latency inference and signal processing. In some cases, the platform 1600 can execute a board support package (BSP) for an operating system, and provide processor-accelerated libraries accessible at the edge. The examples provided herein are non-limiting and can vary across deployments. The platform 1600 can allow for hardware integration for data center supply chain. The platform 1600 can allow for various libraries or information from entities to be accessible at the edge. The platform 1600 can provide open platform for third-party or client software. Low latency and high resolution704913-1418-6318.1Docket No. 102555-0261data acquisition can be achieved between hardware and software of the platform 1600.

[0232] FIG. 17 illustrates an example diagram 1700 of open source integration for power-aware features, in accordance with an implementation. The diagram 1700 can illustrate an open source integrations provided to allow power-aware features of the data processing system 150. As shown, the data processing system 150 can receive raw data of an early analysis of workload. The data processing system 150 can provide power optimization signal to a scheduler. The schedule can include a power aware module or component. The scheduler can provide workload and power commands for managing the workload output. The operations herein can repeat for power optimization.

[0233] In various implementations, the data processing system 150 can provide real-time optimization, including forecast with visibility and control, power capping, job scheduling and preemption, and load balancing across site, to name a few. The data processing system 150 can operate with sub-millisecond latency and resolution tracked by millijoule throughout the power infrastructure.

[0234] FIG. 18 illustrates an example architecture 1800 for power envelope control. The example architecture 1800 can include components capable of performing optimization of the power envelope, where optimization responsibilities or operations can be distributed between local components (e.g., server-level components) and a global component (e.g., an orchestrator or scheduler in a control plane running at a site level, cluster level, or multi-node level). The global component can sometimes be referred to as a global optimizer, for instance, capable of aggregating metrics from multiple servers or rack cabinets and allocating power envelopes to each node based on predicted workload demand or service level agreement (SLA), among other parameters. The local components and the global component can operate in conjunction with each other to balance power consumption across multiple servers while enforcing individual server power envelopes.

[0235] As illustrated in FIG. 18, for purposes of providing examples, the example architecture 1800 can include a global optimizer 1802, a monitor (component) 1804, two or more servers (e.g., server 1812A or server 1812B), and two or more data processing systems (e.g., data processing system 1801A or data processing system 1801B). Each of the servers 1812A-B can be referred to as server(s) 1812. Each of the data processing systems 1801A-B can be referred to as data processing system(s) 1801.714913-1418-6318.1Docket No. 102555-0261

[0236] Each server 1812 can be associated with a respective data processing system 1801. For instance, server 1812A can be associated with (or communicatively coupled to) data processing system 1801 A, and server 1812B can be associated with (or communicatively coupled to) data processing system 180 IB. Each of the data processing systems 1801 can include one or more components or functionalities of the data processing system 150, such as described in conjunction with at least FIGS. 2A-B or FIGS. 22-26. The one or more components or functionalities of each of the data processing systems 1801 can be parts of the data processing system 150. It should be noted that the data processing system 1801 can include additional or alternative components, not limited to those illustrated in FIG. 18.

[0237] The server 1812 can include one or more processing units to execute processing operation(s) for the server. The server 1812 can be a part of a shelf in a rack cabinet of a data center 201, where the data center 201 can include a plurality of rack cabinets. The multiple servers 1812 can be on respective shelves in the rack cabinet. The multiple servers 1812 may be on different shelves in separate rack cabinets. In some arrangements, a first group (or subset) of processing units in a rack cabinet can be assigned to perform processing operations for the server 1812A and a second group of processing units in the same rack cabinet can be assigned to perform processing operations for the server 1812B, such that the servers 1812A-B can be a part of the same shelf of the rack cabinet.

[0238] The data processing system 1801 A can include a local optimizer 1806 A, a local forecaster 1808 A, and an analytic component 1810A (e.g., sometimes referred to as a metrology and power analytics) configured to perform operations of the data processing system 1801A. The data processing system 1801B can include respective components, for instance, a local optimizer 1806B, a local forecaster 1808B, and an analytic component 1810B. The local optimizer 1806A, 1806B can be referred to as local optimizer(s) 1806. The local forecaster 1808A, 1808B can be referred to as local forecasted s) 1808. The analytic component 1810A, 1810B can be referred to as analytic component 1810.

[0239] In various configurations, the analytic component 1810 can collect or process realtime power metrics or operational data from the server 1812 to provide power information to the local forecaster 1808. The local forecaster 1808 can operate to predict an amount of power consumed by the one or more processing units of the corresponding server 1812 at a future time window, e.g., generating forecasts of power draw or consumption. The local forecaster 1808 can utilize at least one of metrics or power information from the analytic724913-1418-6318.1Docket No. 102555-0261component 1810 or the server 1812 to generate the forecast values. The local optimizer 1806 can operate to adjust power limits for individual processing units to maintain consumption within an allocated power envelope (for the node) while maximizing performance.

[0240] The global optimizer 1802 can execute in a control plane to aggregate SLA-related metrics and power consumption data from multiple nodes, such as rack cabinets, servers, or components, distributed across the data center 201. For instance, the global optimizer 1802 can receive periodic or aperiodic reports from the monitor component 1804 that collects metrics from one or more of the local optimizers 1806 or servers 1812. The global optimizer 1802 can determine a power (allocation) envelope for each node based on the aggregated SLA metrics and power consumption data. The SLA metrics can refer to performance targets that specify at least one of, but not limited to, maximum response time, minimum throughput, availability percentage, or processing completion rate for processing operations executed by the one or more processing units. The power envelope can specify a maximum amount of power that each node may consume during a control iteration or time window, for example. The global optimizer 1802 can adjust the power allocation envelope for a node according to (or in response to) changes in SLA achievement or changes in actual power consumption measured at the node. The SLA achievement can refer to a degree to which one or more processing units, servers 1812, or rack cabinets meet or exceed performance targets specified in an SLA, such that the SLA achievement can be quantified by measuring at least one of token throughput rates, inter-token latency values, processing completion rates, or availability percentages and comparing the measured values against corresponding target thresholds defined in the SLA, for example.

[0241] For example, the global optimizer 1802 can increase the power envelope for a first node (e.g., rack cabinet) when SLA metrics for the first node indicate that processing operations are approaching or exceeding a latency threshold or available power capacity allocated from the total provisioned power for the data center 201. The global optimizer 1802 can allocate additional power to the first node to improve SLA performance. In another example, the global optimizer 1802 can decrease the power allocation envelope for a second node when actual power consumption at the second node falls below a utilization threshold, thereby freeing power capacity for reallocation to other nodes that may benefit from additional power to satisfy SLA targets. The global optimizer 1802 can transmit the adjusted power allocation envelopes to the respective local optimizers 1806, via a communication734913-1418-6318.1Docket No. 102555-0261interface, such that each local optimizer 1806 can adjust power limits for processing units of one or more processing units of the corresponding server 1812 to maintain total power consumption by the server 1812 within the allocated envelope, increase computing power, or minimize an amount of power being provisioned to the respective servers 1812 or the data center 201 (e.g., reduce energy consumption while maintaining the total power consumption by the server 1812 within the allocated power envelope, without affecting the performance or processing operation of the server(s) 1812).

[0242] The local optimizer 1806 can be a component of and execute on a corresponding data processing system 1801. The local optimizer 1806 can modulate power caps (e.g., power limits) for the one or more processing units (e.g., GPU(s) or CPU(s)) in the domain of the local optimizer 1806 or the data processing system 1801 associated with the local optimizer 1806. The local optimizer 1806 can execute modulation operations to achieve a power cap with minimal restrictions that maintains the node within the allocated power envelope received or assigned from the global optimizer 1802. The local optimizer 1806 can operate as a feedback controller that adjusts power limits for one or more processing units based on, for instance, forecasted power consumption generated by the local forecaster 1808 to compensate for the latency between a power cap request and the processing unit(s) operating at (or reaching) the power cap.

[0243] The local optimizer 1806 can send power cap commands to the server 1812 to adjust the power cap for one or more processing units. The local optimizer 1806 can receive updated power allocation values from the global optimizer 1802 at a respective time window, such that the local optimizer 1806 can enforce the power envelope constraint while the global optimizer 1802 can coordinate power distribution across multiple nodes in the example architecture 1800.

[0244] In some configurations, the global optimizer 1802 can include one or more components or functionalities of the data processing system 150, or vice versa, as described in conjunction with at least FIGS. 2A-B. In some cases, the global optimizer 1802 can operate as an orchestrator or a centralized data processing system configured to provide instructions to local data processing systems (e.g., data processing system 1801). In such cases, the global optimizer 1802 may be one of a plurality of data processing systems (e.g., 150) operating in the data center 201. In certain arrangements, the global optimizer 1802 may744913-1418-6318.1Docket No. 102555-0261be part of a rack cabinet, remote from rack cabinets, a part of a component in the data center 201, or a device remote from the data center 201.

[0245] The components of the example architecture 1800 can execute features or operations to control power envelope across servers 1812 or rack cabinets. The features or operations of the example architecture 1800 can be to maximize performance of the data center 201 from an SLA perspective for a certain amount of power capacity or power provisioned to the data center 201. A non-limiting example of maximizing performance of the data center 201 can include maximizing token throughput (e.g., token per second) for an allocated amount of power capacity of the data center 201. Examples of adjustable mechanisms or control inputs for achieving this objective (e.g., maximizing token throughput) can include a least (i) matching jobs to compute nodes (e.g., scheduling) or (ii) tuning operating settings processing unit of the processing unit to reduce power consumption (e.g., GPU frequency tuning or GPU power capping). The example architecture 1800 can be configured to ensure that SLAs are satisfied while adjusting the control inputs. For purposes of providing examples, the example architecture 1800 can configure the frequency of the processing unit (e.g., implemented as power capping) as a mechanism for controlling the power consumption of the server(s) while maintaining SLA performance. The example architecture 1800 can configure other operating settings not limited to the frequency of the processing unit, e.g., other DVFS settings.

[0246] The data center 201 can include a collection of compute nodes indexed by i = 1,2, ... , n. Each node can be associated with a power consumption time series Pi(t) reflecting power draw from corresponding processing units or other components, including at least one of GPUs, CPUs, cooling systems, memory, or other components drawing power through circuitry being monitored (e.g., by a monitor component 1804). A node can represent or correspond to a unit of compute monitored by at least one data processing system 1801 (e.g., a server 1812, a rack cabinet, or multiple rack cabinets) in a local domain. The example architecture 1800 can issue a schedule of power cap commands (t) at each node over time. The schedule of power cap command can comprise a vector of power caps (e.g., per-GPU power caps). For instance, the local optimizer 1806 can issue or send the schedule of power cap command(s) to the associated server 1812.

[0247] The example architecture 1800 can track SLA metrics for each node. For instance, each node can have or be associated with a collection of SLA metrics indexed by j =754913-1418-6318.1Docket No. 102555-02611,2, ... , m, with a nodal value of each metric at a particular time being represented by s t). The SLA metrics can be stored in the data processing system 1801 associated with a corresponding node (e.g., server or rack cabinet), the global optimizer 1802, or other devices within the example architecture 1800. The SLA metrics can include at least one of, but not limited to, inter-token latency, time to first token, or other SLA-related measures.

[0248] Because it may be cumbersome to refer to all metrics individually at given node, a bolded notation Sj(t) can be utilized to represent a vector of a group (or all) of SLA metrics at node i. The SLA-related telemetry, including SLA metrics, can be a part of or represented by “vLLM Metrics” (e.g., virtual Large Language Model (vLLM) metrics) or other “Metrics” provided from server 1812 or from analytic component 1810 to the monitor component 1804, the local forecaster 1808, or local optimizer 1806, among other components, as shown in FIG. 18.

[0249] The data center 201 can have a provisioned power pmaxthat cannot be exceeded, where: 2?=i Pi(t)pmax, The SLA-related telemetry can allow for local and global decision-making. The monitor component 1804 (or another monitoring component or service) can collect nodal power usage and related measurements and provide such information upstream (e.g., to global optimizer 1802), allowing for global enforcement of the provisioned-power constraint across nodes while supporting per-node control decisions.

[0250] Focusing on setting power caps for a processing unit (e.g., GPU), for example, the example architecture 1800 can be configured to design a schedule of power caps (t) for each node over a time horizon such that (i) one or more SLA targets are met and (ii) the provisioned power of the data center 201 is not exceeded. To capture or satisfy the SLA(s) in an optimization objective (e.g., for the purpose of at least optimizing power allocation across nodes based on workload demand and service level performance), the example architecture 1800 can define a reward function r(Sj) that scores the SLA metrics at each node. For example, if the optimization focuses on inter-token latency (ITL) metric, the reward function can be a piecewise-linear reward function that penalizes ITL exceeding a threshold relatively more than rewarding ITL below the threshold. The linear reward function can be presented as:rflT11 = - V f10‘ ( ITL(t) - ITL„„),1TL > ITLmax1 JZ4 ITL(t) - ITLmaxITL < ITL max764913-1418-6318.1Docket No. 102555-0261

[0251] The relatively greater penalization of the ITL exceeding the threshold compared to rewarding ITL that is below the threshold can encourage extra performance when sufficient power is available (e.g., when there are a certain amount of available power from the provisioned power). In such cases, the optimization objective may reflect that nodal power draw and nodal SLA metrics depend on the applied schedule for power capping, which can exhibit complex and time-varying relationships, for example. The optimization objective can refer to at least one function or criterion within an optimization problem that quantifies the performance measure to be maximized or minimized, guiding the selection of optimal decision variables subject to given constraints. The optimization problem can refer to the complete a function that includes the optimization objective along with relevant decision variables or constraints, defining the feasible set of solutions and establishing the framework for identifying at least one solution according to the objective. In some implementations, the optimization objective can include a reward function that quantifies performance improvement based on SLA metrics, and the optimization problem can include decision variables representing power allocations for each node, subject to constraints on total provisioned power and per-node power limits. In this case, the optimization problem can be represented as:V illmaximize: > r(si)variables:v 1TI / Pi(0 — Pmax' ^f>i=l

[0252] It should be noted that the power draw Pi(t) (e.g., sometimes referred to as power consumption or an amount of power being consumed by one or more components) and SLA metrics Sj(t) at each node can depend on the power capping schedule (t).

[0253] To having the control plane performing forecasting of power consumption and imposing caps for the nodes, the example architecture 1800 can employ a (primal) decomposition approach or technique to separate the optimization problem into, for instance, a global problem (e.g., executed by global optimizer 1802 in a control plane) and a local problem for each node (e.g., executed by local optimizer 1806 on a local unit such as the data processing system 1801). The decomposition approach can allow the global problem to be executed in the control plane and the local problem to be executed on the data processing 774913-1418-6318.1Docket No. 102555-0261system 1801 (e.g., local to the node), thereby avoiding excessive communication between the devices to reduce latency, minimize communication resources, and provide parallel processing.

[0254] In this example approach, each node can be assigned a power allocation Uj(t), representing a maximum amount of power that the node is allowed to consume at a time. In FIG. 18, the allocations for the maximum amount of power can correspond to or be labeled as “Power Allocation” outputs produced by the global optimizer 1802 or the monitor component 1804, provided to one or more of the data processing systems 1801 (e.g., the local optimizer 1806 of the data processing system 1801). With the added allocations, the optimization problem can be represented as:maximize: > r(Sj)^—>1=1variables:Vt, iSubject to: Pi(t) < Uj(t), Vt, i / ttf(t) < Pmax' ^f

[0255] One or more sub-problems (e.g., local problem) of the optimization problem can be defined. A local problem can refer to a sub-optimization task focused on controlling and optimizing power usage at an individual computing node or unit, such as a rack cabinet or server, for instance, by adjusting parameters such as power caps to ensure the node operates within an allocated power envelope while maximizing performance in compliance with local constraints. For example, for each node i, £be the maximum of the following local problem:maximize: r(Sj)variables: q(t), Vtsubject to: Pi(t) < Uj(t), Vt

[0256] The global problem can be expressed in terms of the local problem solutions across nodes such that the control plane (e.g., global optimizer 1802) can be responsible for deciding how power is to be allocated to each node. For instance, the global problem can be represented as:784913-1418-6318.1Docket No. 102555-0261maximize: > ^i(ai)^—>1=1variables: at(t), Vt, isubject to: > ctj(t) < pmax, Vt^—>1=1

[0257] Further, the (local) data processing systems 1801 (or the local optimizer 1806) can be responsible for scheduling or issuing power caps for individual processing units of the corresponding server 1812 such that the actual (measured) power consumption at the node does not exceed the power cap of the processing unit. In some implementations, the local sub-problem for a node can be framed such that, given the assigned power allocation, the node (or the data processing system 1801 associated with the node) can select a schedule for power capping to maximize a corresponding local reward while ensuring the node does not exceed the power cap (of the processing unit). With the data processing systems 1801 managing power caps of respective processing units, the power consumption of the data center 201 can be maintained at or below the provisioned power or power envelope, for example.

[0258] In various configurations, to solve a local problem, it may be assumed that the reward function and SLA metrics are such that the local objective is non-decreasing in the power cap (t). That is, increasing a power cap can assist with achieving a relatively higher SLA reward. Under this assumption, the local optimizer 1806 may not require information regarding the detailed reward function (e.g., r(Sj)) for the purpose of choosing or selecting power caps (or optimizing the power caps). The (optimal) local power cap can be obtained by computing or solving the following:maximize: q(t)variables: Cj(t), Vtsubject to: Pi(t) < cij(t), Vt

[0259] In such cases, the local optimizer 1806 can select the highest (e.g., least restrictive) power cap(s) feasible while ensuring that the power consumption by the corresponding node does not violate the assigned power allocation (e.g., does not exceed the power envelope). In such cases, the power draw pj(t) may depend on q(t) (e.g., the power cap at node i and time t), where the relationship between these variables may be indirect or not explicitly defined. Such an arrangement can allow for the utilization of locally available data to generate forecasts of power consumption by one or more processors of the794913-1418-6318.1Docket No. 102555-0261corresponding server 1812 under a current configuration. For instance, the local forecaster 1808 can use at least one of metrics from the server 1812 and power analytics from the analytic component 1810 to forecast power consumption.

[0260] In some implementations, the metrics can include or be a part of telemetry or control information, for instance, from a system management API of the server 1812 configured for monitoring and managing various states of the processing units. The metrics can include at least one of current power cap settings applied to one or more processing units of the server 1812, utilization percentages representing a fraction of available compute capacity currently in use by each processing unit, temperature readings of the processing units, clock frequency values indicating an operating frequency of each processing unit, memory bandwidth measurements representing a rate of data transfer to or from memory devices in electrical communication with the processing units, workload type identifiers indicating a category of computational task being executed by the processing units, throughput metrics representing a quantity of operations completed per unit time, or SLA parameters specifying performance targets associated with the processing operations executed by the server 1812, among others.

[0261] Based on the forecast values, the local optimizer 1806 can select the largest power cap(s) such that the power draw p;(t) does not exceed an amount of power provisioned to the server 1812 (e.g., allocation amount ctj (t)) or at least with relatively high probability (e.g., 85%, 90%, 95%, or 99%) that the power draw Pi(t) does not exceed allocation amount cq(t). The local selection of power caps can be treated as a stochastic optimization problem, where the (local) data processing system 1801 targets obtaining (and act on) an upper prediction interval for future power draw, rather than relying solely on a point forecast.

[0262] The local optimizer 1806 can monitor the power draw Pi(t) at a predefined frequency or interval, and adjust the power cap (t) based on the power draw Pi(t), for instance, to maintain compliance with the allocation. The local optimizer 1806 can account for a delay between requesting a power cap and the processing unit(s) realizing the power cap, e.g., may involve around 1 second delay. In such cases, the local optimization problem can operate as a feedback control problem in which the local optimizer 1806 uses forecasted power (from local forecaster 1808) and observed metrics or power (from analytic component 1810 or server 1812) to compute updated power caps for corresponding one or more processing units over successive control iterations.804913-1418-6318.1Docket No. 102555-0261

[0263] In some configurations, for solving the global problem, the example architecture 1800 can replace the inequality i=iai(t) < Pmax withanequality. Given the assumption that the reward function and SLA metrics are such that r(Sj) is non-decreasing in (t), an optimum that allocates all of the provisioned power can be identified. For instance, if increasing caps improves reward, the global optimum can allocate all provisioned power rather than withholding capacity. The global optimizer 1802 can solve for power allocation schedules using a Lagrange multipliers technique (among other techniques or methods). The (optimal) allocations schedules Uj(t) can admit (or can be characterized by) a multiplier A(t) (e.g., a time series of multipliers) satisfying a system of nonlinear formula, such that:dii— — - = A(t),Vtdai(tP)

[0264] This, together with the 2”=i «i(t) < Pmax constraint, can represent a system of n + 1 nonlinear equations at each time t, with n + 1 unknowns, e.g., the n power allocations at the time / , and the value of the Lagrange multiplier. For purposes of analysis, the temporal dependence of the power allocations can be disregarded, treating the allocations as static or considering the dynamic allocation problem at a singular instantaneous point in time, with individual allocations aggregated into a single vector a, and a corresponding vector-valued function being defined as:^2(^2)—f(a, A) =-inai ~ Pmax-li=l

[0265] In various implementations, the root of the vector-valued function can represent or yield the solution. For instance, the global optimizer 1802 can find the root f(a, A) = 0. The global optimizer 1802 can utilize any suitable techniques for solving nonlinear systems of formulas, such as Newton’s method, which can involve constructing a sequence of iterative guesses for the solution (a®, A®), (a^\ A^1^), (a®, A®) ... that converges to the optimum.

[0266] In some aspects, computing derivatives of local rewards with respect to local allocations may be impractical because the relationship can be relatively complex, stochastic, and time-varying. In such cases, the global optimizer 1802 (executing in the control plane) can monitor how local rewards respond to changes in allocations and thereby creating a814913-1418-6318.1Docket No. 102555-0261model of the reward function. In some implementations, the modeling can be provided for reinforcement learning. It should be noted that the techniques, formulas, or types of data discussed hereinabove are provided as non-limiting examples, and that other techniques, formulas, or types of data may be utilized as part of operating at least the data processing system 1801 or the global optimizer 1802 to optimize power allocation, forecast power consumption, control power envelopes, or manage resource utilization across the data center environment.

[0267] In various configurations, the data processing system 150 (or the data processing system 1801) can control a (local) power envelope, e.g., for one or more processing units of a rack cabinet (or a server 1812). The features or functionalities of the data processing system 150 for enforcing the power envelope can be described in conjunction with at least one or more components or functionalities of the data processing system 1801 of FIG. 18. For instance, the power envelope enforcement can be part of the functionalities of the local optimizer 1806 of the data processing system 1801 operating for a corresponding server 1812. For instance, the data processing system 150 can include or correspond to a local controller for the server 1812. In this case, the local controller can be a finite state machine.

[0268] The data processing system 150 (e.g., controller) can maintain an overall state. For each control iteration, the data processing system 150 may transition into a new state based on sensor values such as electric waveform data, metrics (e.g., RMS values, power consumption values, or power envelope value) computed from the electric waveform data, etc. Each state can result in qualitatively different behavior or actions from the data processing system 150. In this case, the control objective of the data processing system 150 can be to maintain the power consumption of the server (e.g., server 1812 or other servers) at a level that is approximate to or approaching a power envelope value (e.g., varies over time) without exceeding the power envelope value.

[0269] In some configurations, five controller states can be configured, e.g., three states corresponding to “normal” operating conditions and two states corresponding to “emergency” operating conditions. A normal operating condition can refer to an operational state in which total power consumption (of the corresponding server or processing unit(s)) remains within an acceptable proximity or threshold to (without exceeding) the power envelope value and no immediate corrective action may be required to prevent power excursions, for example.824913-1418-6318.1Docket No. 102555-0261

[0270] An emergency operating condition can refer to an operational state in which the total power consumption (i) approaches the power envelope value beyond a predefined threshold or (ii) exceeds the power envelope value such that immediate corrective action (e.g., within a predefined time window) may be executed to reduce power consumption or maintain operation within the power envelope. It should be noted that additional or fewer number of controller states can be configured or defined, not limited to the example controller states discussed herein. In some cases, alternative controller states can be configured to replace at least one of the five controller states. The example five controller states can be provided in example Table 1.State DescriptionThe controller state may be designated as an idle state when the server is idling or when power data is missing or invalid. In such cases, the data processing system 150 can maintain the power limit of IDLE the processing unit(s) at or below a predefined threshold, or reduces the power limit as desired, thereby preserving a predefined headroom value to accommodate potential sudden increases in workload and associated power consumption.The controller state may be designated as a ramp up state when the server operates under a certain amount of computational load, while having a headroom value (e.g., margin between a total power RAMP UP consumption by the server and the power envelope value) greater than or equal to a predefined threshold. In such cases, the data processing system 150 can increase the power limit of the processing unit(s) incrementally to allow for additional computational throughput. The controller state may be designated as a regulate state when an amount of power consumed (e.g., total power consumption) is within predefined range or margin of the power envelope value, e.g., the REGULATE headroom value is below a predefined threshold. In such cases, the data processing system 150 can adjust the power limits of the processing unit(s), e.g., increasing or decreasing the power limit value, to maintain the power consumption value in proximity to the834913-1418-6318.1Docket No. 102555-0261power envelope value (minus a margin) without exceeding the power envelope value.The controller state may be designated as a ramp down state when one or more recent (historical) power samples indicate that an excursion may occur if at least one corrective action is not taken RAMP DOWN!during a (current) control iteration. In such cases, the data processing system 150 can preemptively reduce power limits now to prevent an excursion in a subsequent time window.The controller state may be designated as an excursion state during an occurrence of an excursion (e.g., the power consumption exceeds the power envelope value). In such cases, the data processing system 150 EXCURSION! can reduce power limits in response to transitioning to the excursion state, thereby reducing the power consumption below the power envelope value and restoring operation within the allocated power budget.Example Table 1: Controller States

[0271] Each control iteration of the data processing system 150 can proceed through several phases. The phases can include, for example, an aggregation phase, a state estimation phase, a state transition phase, a compute new power limit phase, a clipping and quantization phase, and a rate limiting phase. Each control iteration can include more or fewer number of phases, or alternative phases.

[0272] In an aggregation phase, the data processing system 150 (e.g., data collector 204) can aggregate power measurements or values from various sources relevant to the envelope being enforced. The data processing system 150 can cache the latest sample of power values from respective sources, e.g., store in the data repository 222 or other data storage. When all sources have been updated, the data processing system 150 (e.g., data processor 208) can estimate total power as a sum of the cached samples. The data processing system 150 can initiate a new control iteration with a respective new estimate of total power, for instance, once per AC cycle at intervals (e.g., around 16 to 17 milliseconds, in some cases).

[0273] In a state estimation phase, the data processing system 150 (e.g., data processor 208 or controller state manager 216) can estimate, based on the most recent sample of total844913-1418-6318.1Docket No. 102555-0261power consumption, one or more variables (e.g., sometimes referred to as state variables) used to determine state transitions and compute control actions.

[0274] In a state transition phase, the data processing system 150 (e.g., controller state manager 216) can determine the next controller state. In a compute new power limits phase, the data processing system 150 (e.g., data processor 208) can compute new power limits to request for one or more processing units (e.g., GPUs or CPUs) based on one or more of (i) a new controller state, (ii) estimates for each state variable, or (iii) current power limits of the one or more processing units, among others.

[0275] In a clipping and quantization phase, the data processing system 150 (e.g., data processor 208) can clip new limits to a certain range, such as a range that correspond to minimum and maximum power limits supported by control software for the one or more processing units. In some cases, the range can be set or predefined by a user. The data processing system 150 can quantize power levels to increments such as multiples of 5 watts to reduce the rate at which limit changes are requested.

[0276] In a rate limiting phase, the data processing system 150 (e.g., controller 220) can impose a cooldown period (e.g., 250 milliseconds) between successive commands to suppress irregular or uncontrolled variations in the behavior of the electric waveform data (e.g., AC power) associated with relatively high frequency of changes in the power limit. In such cases, the data processing system 150 can send new or updated limits after a predefined number of iterations such as every fifteenth iteration. The data processing system 150 can allow emergency commands (e.g., generated for at least one emergency operating condition) to interrupt an ongoing cooldown, e.g., when the controller state is a ramp-up state or an excursion state. In such cases, the data processing system 150 can bypass the cooldown and send at least one new limit to the processing unit(s) for adjustment.

[0277] At the start of each control iteration, the data processing system 150 (e.g., controller state manager 216 or data processor 208) can update estimates for a plurality of state variables based on the most recent sample of total power. Examples of state variables can be provided in example Table 2. It should be noted that additional or alternative state variables may be included. There may be more or fewer number of state variables, for instance, to control the transitioning of the controller state.854913-1418-6318.1Docket No. 102555-0261Variable Units DescriptionData integrity Boolean Indicates if new data is missing or invalid.flagHeadroom Watts Difference between the power envelope value and actual measured power (e.g., power consumption). A negative value for the headroom can indicate an excursion, e.g., the power envelope value is less than the amount of power being consumed.Power movingWatts Exponential moving average of total power, calculated average accounting for the time elapsed since the last power sample.Power rate-of- Watts per Instantaneous rate of change in total power, which can change millisecond be estimated from at least two most recent samples.An average of instantaneous rates of change can be taken when more than two most recent samples are used for the estimation, for example.Time to Millisecond An amount of time remaining before an occurrence of excursion an excursion under the current rate of change. The time to excursion can be calculated if the headroom value and the power rate-of-change are positive.Example Table 2: State Variables Updated Per Control Iteration

[0278] The data processing system 150 (e.g., controller state manager 216) can determine state transitions by evaluating a plurality of predefined conditions (e.g., transition conditions) in a prioritized order. The conditions can be evaluated sequentially such that conditions with relatively higher priority take precedence over conditions with relatively lower priority. Examples of the conditions for transitioning the controller state can be provided in example Table 3.If... ...then transition to:There is a data integrity flag (e.g., flag = 1) IDLEHeadroom value is below a (negative)EXCURSION!predefined threshold864913-1418-6318.1Docket No. 102555-0261Time to excursion is below a predefinedRAMP DOWN!thresholdPower moving-average is below aIDLEpredefined thresholdHeadroom value is below a (positive) user- REGULATEdefined thresholdAll other scenarios (e.g., none of theRAMP UPforegoing conditions are satisfied)Example Table 3: State Transitions

[0279] For purposes of providing examples herein, the example conditions in example Table 3 can be listed in a priority order. In this case, the example conditions can be listed in a descending priority order. For example, the condition for transitioning to the idle state can be assigned as the highest priority relative to other conditions and the condition for ramp up (e.g., all other scenarios such as when there is headroom to increase the power limit) can be assigned as the lowest priority relative to other conditions. The priorities can be adjusted or configured, not limited to the example order shown in example Table 3. In some cases, additional or alternative conditions can be configured.

[0280] The data processing system 150 (e.g., controller state manager 216 or controller 220) can initiate or execute one or more control actions based on the controller state. For example, when operating in the idle state, such as due to invalid data or because measured power has fallen below a predefined threshold, the data processing system 150 can apply a (conservative) power limit to the one or more processing units to preserve a predefined headroom value to accommodate potential sudden increases in workload and associated power consumption. For example, the data processing system 150 can set the power limit to a predefined value (or margin) that accounts for potential surges in power consumption that may not be reflected in current measurements. The predefined value can be based on a maximum power draw at a corresponding processing unit limit including the potential of boosting clock frequency or voltage of the processing unit, such that the predefined value represents an estimate with a predefined margin to ensure there is a certain amount of headroom for transient power spikes.

[0281] In a ramp up state, the data processing system 150 can increase the power limit of the one or more processing units at a predefined rate. The predefined rate can be predefined874913-1418-6318.1Docket No. 102555-0261or configured by a user. In some cases, the predefined rate can be determined based on system configuration parameters. For example, the data processing system 150 can increase the power limit at a rate of 100 watts per second, among other rates. The data processing system 150 can compute a step size for each control iteration based on a cooldown time period. For instance, when the predefined rate is 100 watts per second and the cooldown time is 250 milliseconds, the data processing system 150 can increase the power limit by 25 watts during each control iteration.

[0282] The regulate state can implement a proportional-derivative control technique or operation that operates on an error signal. The error signal can be represented as:e = P + Pmargin + Penv- where P can represent the measured power, Pmargincanrepresent a predefined (safety) margin, and Penvcan represent the power envelope value during the time period of control (e.g., Penvmay change over time). The requested change in the power limit of the processing unit can be computed as:Pitmit kp6 max{kdd, dnm

[0283] For the computation of the power limit for the processing unit, given proportional and derivative gains kp, kd, the power rate-of-change d, and the predefined limit dUm(e.g., set by the user) on how much the derivative term can increase the change in power limit.

[0284] To facilitate a relatively smooth transition from the ramp up state to the regulate state, the data processing system 150 can select the proportional gain such that both states result in the same change in power limit at a headroom value that separates the two states. A smooth transition can refer to, for example, a continuous or consistent adjustment in power limit without abrupt jumps or discontinuities, thereby preventing oscillations or instability in power control as the processing unit(s) transitions between the ramp up and regulate states. For example, when the ramp up state increases the power limit by 25 watts per step (or control iteration), a headroom value (e.g., boundary) separating the ramp up state from the regulate state is 4000 watts, a safety margin is 2000 watts, and the derivative term is negligible, the data processing system 150 can set the proportional gain to 0.0125. It should be noted that the values are provided as non-limiting examples for illustrative purposes.

[0285] The two emergency states (e.g., ramp down state and the excursion state) can have similar or identical control behavior. These emergency states may be separated for observability and debugging purposes, e.g., to distinguish between the conditions or states,884913-1418-6318.1Docket No. 102555-0261when sending reports, current state information, or displaying the controller state to the user via a graphical user interface device. The emergency states can implement proportional-derivative control with control parameters that are relatively more aggressive than those used in the REGULATE state. Having a relatively more aggressive control parameters can refer to applying relatively higher proportional and derivative gain coefficients to expedite corrective actions, thereby allowing for a relatively faster reduction of power limits (e.g., compared to adjustments in other controller states) to responsively mitigate power excursions or restore operation of the processing unit within the allocated power envelope.

[0286] In the emergency states, the data processing system 150 can set the derivative limit as dUm= 0 such that the derivative term can serve as a brake by counteracting rising power. For example, the data processing system 150 can configure the emergency states such that measured power is expected to return to a target value of (Penv— Pmargin) within a predefined time window, such as 60 milliseconds. The predefined time window for the measured power to return to the target value can be based on at least one of system responsiveness criteria or operational stability criteria to ensure timely mitigation of power excursions without causing instability or oscillations in power control, for example.

[0287] FIGS. 19-21 illustrate graphs 1900, 2000, and 2100 related to the operations for forecasting power consumption over subsequent time window(s). In various configurations, the data processing system 150 (or one or more components of the data processing system 150) can perform the forecasting of power consumption. The one or more components or functionalities of the data processing system 150 can be described in conjunction with at least the data processing system 150 (e.g., local forecaster 1808) to predict the power consumption of one or more processing units of the server 1812 or the rack cabinet.

[0288] The data processing system 150 (e.g., model manager 210, power predictor 212, or local forecaster 1808) can execute a forecasting technique to predict power consumption of the server (e.g., server 1812) over a varying time horizon, including short-term horizon and long-term horizon. For example, short-term horizon can correspond to a time window spanning approximately 10 milliseconds to 100 milliseconds. It should be noted that the values are provided herein as examples, and may be adjusted or configured to other nonlimiting values.894913-1418-6318.1Docket No. 102555-0261

[0289] As part of the forecasting technique, the data processing system 150 can receive one or more metrics (e.g., sometimes referred to as summary metrics) as input for one or more neural networks. The metrics can be computed from AC waveform power data. The metrics can include at least RMS values derived from the AC waveform power data over one or more time windows. The metrics can include other types of data, not limited to the RMS values. A window length for computing the metrics can correspond to one AC cycle. In some cases, the window length for the computing the metrics can corresponding to more than one AC cycle. In some other cases, the window length can be configured to be smaller than one AC cycle, for instance, to improve time resolution of the metrics. The data processing system 150 can generate, via executing the forecasting technique, predictions on a per-phase basis or on a three-phase sum basis.

[0290] The forecasting technique can be configured to avoid power excursions above a predefined power limit. Power excursions exceeding the predefined power limit can trigger activation of protection equipment, such that maintaining predicted power consumption below the predefined power limit can reduce a frequency of protection equipment activation events. To achieve this objective, the data processing system 150 (e.g., model manager 210) can train at least one forecasting model (or neural network or artificial intelligence model) using a pinball loss function with a predefined quantile parameter q, e.g., q set to 0.95.

[0291] FIG. 19 illustrates a graph 1900 of an example pinball loss. Setting the quantile parameter q to 0.95 can allow the forecasting model to predict a 95th percentile of power consumption, instead of a mean or median value. The choice of pinball loss with q = 0.95 can (asymmetrically) penalize prediction errors such that under-predictions (e.g., forecasted power values lower than actual power consumption) receive a relatively higher penalty weight compared to over-predictions (e.g., forecasted power values higher than actual power consumption). In the plot, true values (e.g., actual power consumption measurement) can be provided as line 1904. Prediction residuals can be shown as vertical lines, indicating the over for over-prediction (e.g., positive residual) and under-prediction (e.g., negative residual). Examples of negative residuals can be provided in region 1902, with weight penalty of 0.95. The positive residuals can have a weight penalty of 0.05, for example. As such, the forecasting model can be biased toward power consumption estimates that are relatively higher than expected actual power usage, providing a safety margin that reduce a likelihood of power excursions.904913-1418-6318.1Docket No. 102555-0261

[0292] The forecasting model can comprise a fusion architecture combining multiple neural network components. The data processing system 150 (e.g., model manager 210 or power predictor 212) can deploy the neural network components concurrently or sequentially. For example, the forecasting model can include a first neural network component. The first neural network component can process longer-term power samples (e.g., long-term horizon) spanning a first time window. As an example, the first time window can include a duration of approximately one second. The longer-term power samples can be provided as input to a onedimensional convolutional neural network (CNN).

[0293] The forecasting model can include a second neural network component. The second neural network component can process shorter-term power samples (e.g., short-term horizon) spanning a second time window. As an example, the second time window can include a duration of approximately 100 milliseconds. The shorter-term power samples can be provided as input to a first linear neural network. In some implementations, a plurality of second time windows can correspond to the first time window such as 10 second time windows can correspond to the duration of one first time window. In some implementations, the second time window can represent a fraction or a portion of the first time window, e.g., the second time window can be a tenth of the first time window. The first time window and the second time window can be predefined, configured, or adjusted.

[0294] The forecasting model can include a third neural network component. The third neural network component can receive as input a first output from the one-dimensional CNN and a second output from the first linear neural network. The third neural network component can comprise a second linear neural network to join or combine the first output and the second output. The forecasting model can include a fourth neural network component. The fourth neural network component can receive, as input, a third output from the second linear neural network. The fourth neural network component can comprise a decoder neural network to generate forecast values representing predicted power consumption over a subsequent time window. In some configurations, the forecasting model can predict multiple future time steps in parallel.

[0295] In some implementations, the data processing system 150 (e.g., power predictor 212) can receive the output from the third neural network for processing, e.g., with or without the fourth neural network, to generate the forecast values for the power consumption of the server. Other types of neural network models can be implemented or utilized, not limited to914913-1418-6318.1Docket No. 102555-0261the example neural networks herein. More or fewer number of neural networks can be utilized.

[0296] FIG. 20 illustrates a graph 2000 of an example input and output data for neural networks. Recent data (e.g., input 1 of around 100 milliseconds) can be provided to the first neural network to generate a first output. The long-term historical data (e.g., input 2) can be partitioned into fixed-size windows. As such, input 2 can include a plurality of time windows including the time window of input 1. The long-term historical data can be provided to the second neural network to generate a second output. The metrics can be computed for each window, and the aggregated features can be passed into a two-layer neural network. The data processing system 150 (e.g., data processor 208, model manager 210, or power predictor 212) can process the recent data to obtain metrics such as but not limited to RMS power values, rate of change, or short-term temporal features (thereby providing a relatively granular information compared to the long-term historical data). The data processing system 150 can process the long-term historical data to obtain metrics such as at least minimum value, maximum value, or mean value for individual windows.

[0297] The outputs from the first neural network component and the second neural network component can be fused to generate predictions at multiple horizons simultaneously, for instance, via at least one of the third neural network component or the fourth neural network component. An example of a vector comprising the forecast values can be shown in the “Output” of FIG. 20. The forecast window can include a predefined window duration. For instance, the forecasting window duration can be similar to or different from the first window duration.

[0298] For purposes of providing examples, the subsequent (future) time window for the forecast values can be referred to as a third time window. After performing the prediction, the data processing system 150 can perform measurements at the third time window to generate actual power consumption data, which can be incorporated into the historical dataset by updating the plurality of time windows to include this latest measurement. The oldest window in the (long-term) historical data can be discarded to maintain a fixed-length dataset, ensuring that the forecasting model continuously trains and predicts based on the most recent power consumption patterns. For subsequent forecasting, the metrics associated with the third time window can be used as the short-term data for generating forecast values for a fourth time window subsequent to the third time window, for example. The data processing system 150924913-1418-6318.1Docket No. 102555-0261(e.g., model manager 210) can iteratively update or train the forecasting model to refine the accuracy of the model over time by comparing predicted values against real measurements and adjusting model parameters accordingly.

[0299] The data processing system 150 (e.g., model manager 210) can train the forecasting model by minimizing an error between a forecast region (e.g., 1902) of historical time-series data and output values (e.g., “Output” region of FIG. 20) generated by the neural network (e.g., the third neural network or the fourth (decoder) neural network). The forecast region can include a portion of the historical time-series data representing future power consumption relative to a reference time point in the historical data.

[0300] FIG. 21 illustrates a graph 2100 of an example forecasting results. As shown, graph 2100 can include an example time series with the prediction residuals plotted on top of the power data. Each power datapoint can be the RMS of the three-phase server power at a 1-cycle time resolution. The graph 2100 can include inference using a horizon of around 83 milliseconds, for example. True data (e.g., actual measured data) may be shown as line 2102, and residuals can be shown as vertical lines 2104 and 2106. The pinball loss can penalize under-prediction residuals (e.g., vertical lines 2106) greater than (e.g., 19 times more than) over-prediction residuals (e.g., vertical lines 2104). The overestimation shown by vertical lines 2104 before the spikes can indicate that the forecasting model can anticipate or account for potential spikes in power consumption before the occurrences of the spikes.

[0301] The forecasting model can generate predictions for multiple future time steps in parallel, for instance, instead of independently predicting each future time step in sequence. Parallel prediction of multiple future time steps can improve computational efficiency and generate predictions that exhibit relatively greater temporal consistency compared to independent sequential predictions, for example. The forecasting model can be configured to incorporate various neural network architectures, input feature sets, or training methodologies to accommodate diverse operational scenarios and performance requirements.

[0302] In certain implementations, a performance (e.g., effectiveness and quality) of implementing a control strategy executed by the data processing system 150 can be evaluated using one or more metrics. The one or more metrics can include but not limited to RMS excess power, maximum excess power, or unused capacity. Such metrics can be employed to quantify at least one of: (i) compliance with thermal or steady-state operational regimes, (ii)934913-1418-6318.1Docket No. 102555-0261severity of power excursions relative to protection limits, or (iii) efficiency of power utilization with respect to provisioned capacity, to name a few. The one or more metrics can be provided as non-limiting examples and additional or alternative metrics may be used to assess control performance of the data processing system 150, among other devices for optimizing the power envelope and the compute performance of the data center 201.

[0303] FIG. 22 is an example flow diagram of a method 2200 for provisioning power used by rack cabinets in a data center, in accordance with an implementation. The example method 2200 can be executed, performed, or otherwise carried out by the data processing system 150, one or more components of the utility grid 100 (e.g., computing device, metering devices 118, data processing system 150B, etc.), one or more components of the data center 201 (e.g., data processing system 150A, server 236, etc.), other components of the system 200, or computing device 2700, etc. Although the data processing system 150 is described herein to execute or perform the method 2200, other devices can be configured to perform similar features or functionalities as the data processing system to perform the method 2200.

[0304] The data processing system 150 can perform the method 2200 to optimize power utilization by adjusting power allocations for processing operations by processing units based on predicted power consumption across rack cabinets in the data center 201. The method 2200 can include obtaining electric waveform data, at ACT 2202. At ACT 2204, the method 2200 can include identifying an amount of power provisioned. At ACT 2206, the method 2200 can include predicting a first amount of power and a second amount of power. At ACT 2208, the method 2200 can include determining a third amount of power for provisioning a processing operation. At ACT 2210, the method 2200 can include determining whether the third amount of power is different from a current amount of power. At ACT 2212, the method 2200 can include adjusting one or more parameters.

[0305] At ACT 2202, the data processing system 150 (e.g., data collector 204) can obtain electric waveform data measured for a rack cabinet in the data center 201 at a first time window. The first time window can refer to a current time window when the electric waveform data is obtained, measured, or otherwise received by the data processing system 150 or one or more circuits monitoring or measuring the electric waveform. The rack cabinet can include a plurality of shelves such as physical shelves distributed across the rack cabinet. Each of the plurality of shelves can include a respective one or more processing units. In some cases, each of the shelves can refer to a row of one or more processing units. The one or944913-1418-6318.1Docket No. 102555-0261more processing units on each of the shelves can operate as or perform operations for a respective server, where each server can execute a distinct set of processing operations associated with one or more workloads allocated to the server. For example, a first shelf in the rack cabinet may house processing unit(s) operating as a first server executing machine learning training tasks, and a second shelf in the rack cabinet may house processing units operating as a second server executing inference operations on trained models. In some cases, multiple shelves of processing units can operate as a server. In some other cases, a subset of a shelf of the rack cabinet can operate as a server.

[0306] The electric waveform data can include, but are not limited to, at least one of voltage waveform data or current waveform data. To obtain the electric waveform data, the data processing system 150 (e.g., data collector 204) can monitor, via one or more measurement circuits disposed at a power distribution unit electrically coupled to the rack cabinet, voltage waveform data or current waveform data associated with electric power delivered to the plurality of shelves of the rack cabinet. In some cases, the data processing system 150 can monitor, via one or more measurement circuits disposed at a power supply unit electrically coupled to the rack cabinet, voltage waveform data or current waveform data associated with electric power delivered to the plurality of shelves of the rack cabinet. In some arrangements, the data processing system 150 can receive electric waveform data from one or more devices distributed across a power chain such as at least one device coupled to the input or output of the rack cabinet to measure electrical data to or from components of the rack cabinet and send the measurements to the data processing system 150 (e.g., received by the interface 202). The data processing system 150 can be local to the rack cabinet to directly receive the electric waveform data or remote from the rack cabinet to receive the electric waveform data via an intermediary device or from one or more devices of the rack cabinet, for example.

[0307] The one or more processing units can include GPUs, CPUs, or other processing units for processing operations. The data processing system 150 (e.g., data processor 208) can convert the electric waveform data to at least one of a quantitative value representing an amount of power consumed at the rack cabinet, a load variability, an output from a fast Fourier transform (FFT), root mean square (RMS) values, or other values at the first time window. For example, the data processing system 150 can apply an RMS calculation to the electric waveform data (e.g., AC waveform samples) to determine an RMS power value for a954913-1418-6318.1Docket No. 102555-0261time window corresponding to one or more cycles of the waveform. The data processing system 150 can compute an amount of power consumed by the one or more processing units (or by each of the shelves, the rack cabinet, or multiple rack cabinets) using any suitable computational technique(s) or function(s). In some configurations, the data processing system 150 can receive processed data from other devices such as directly receiving or obtaining the amount of power consumed by the one or more processing units from at least one device that processed the electric waveform data.

[0308] It should be noted that an amount of power consumed by a component can refer to a power consumption amount, an amount of consumed power, used power, or a power consumption level associated with the component. Provisioned power can refer to an upper bound of allocated power or a power envelope, where used power herein can refer to measured consumption, and available power can refer to the difference between the provisioned power and the used power, for example.

[0309] At ACT 2204, the data processing system 150 (e.g., power envelope identifier 206) can identify, for the first time window, an amount of power provisioned to the data center 201 and used by the rack cabinets in the data center 201. The amount of power provisioned to a component can sometimes be referred to as an amount of provisioned power or a power envelope of the component. The power envelope can refer to an amount of electrical power allocated or made available to a data center, rack cabinet, or processing units for use during operation (e.g., for processing operation(s)). The power envelope can be independent of an actual power consumed by components of the data center 201. For instance, a portion of the power envelope may be reserved such as for contingencies, including transient load surges, redundancy requirements, and / or equipment fault conditions, as described in conjunction with at least FIG. 3. However, it may be challenging for data centers to maximize or otherwise optimize the use of the power that has been provisioned to the data center in an efficient, effective, and reliable manner, thereby resulting in reduced computing capacity for the data center. Hence, the data processing system 150 can manage and optimize power utilization across a facility such as a data center to facilitate the increased, optimal, or full utilization of provisioned power for compute operations, without exceeding the provisioned power, e.g., by minimizing load variability, increasing maximum utilization of provisioned power, etc.964913-1418-6318.1Docket No. 102555-0261

[0310] The data processing system 150 (e.g., data collector 204) can monitor the amount of power at a metering device 118 electrically coupled with the rack cabinets used in the data center 201, where the amount of power can represent a power envelope or a total power available for use by at least the rack cabinets for the processing operations. In some cases, the data processing system 150 (e.g., power envelope identifier 206) can receive information from the utility grid 100 indicating a total amount of power allocated for use by the data center 201 during a time window such as the first time window. In some implementations, the data processing system 150 can access a power allocation record stored in the data repository 222 that specifies a maximum power capacity provisioned to the data center 201 by an external power source. For example, the power allocation record can specify at least one of a peak power limit negotiated with a utility provider or a predefined maximum power corresponding to at least a combined capacity of all electrical circuits feeding the data center 201.

[0311] The amount of power provisioned to the data center 201 and used by the rack cabinets may be affected by one or more conditions and / or variables such as at least one of time of day, weather condition, operating conditions at an electricity distribution grid (e.g., the utility grid 100), or power allocation limits supported by the data center 201. For example, there may be a relatively higher amount of electricity available for provisioning the data center 201 at nighttime compared to daytime, e.g., because aggregate electrical consumption across all loads served by the utility grid 100 may be lower at night than during peak daytime hours, thereby freeing additional capacity that the utility grid 100 can allocate to the data center 201. In further examples, the weather condition can affect electrical usage across loads served by the utility grid 100.

[0312] The operating conditions at the electricity distribution grid can include at least one of grid voltage stability metrics, frequency regulation parameters, load balancing states across transmission lines, reactive power compensation levels, transformer loading percentages, or fault detection flags. For example, the operating conditions at the electricity distribution grid can include a grid voltage stability metric representing a deviation of measured voltage from a nominal voltage setpoint across one or more distribution feeders, wherein the deviation can be expressed as a percentage or an absolute voltage difference. In some cases, the operating conditions at the electricity distribution grid can include a frequency regulation parameter representing a rate of change of grid frequency over a predefined time window. The974913-1418-6318.1Docket No. 102555-0261frequency regulation parameter can indicate whether the grid frequency may be increasing, decreasing, or remaining substantially constant relative to a target frequency value.

[0313] The power allocation limits supported by the data center 201 can include maximum power thresholds indicated by at least one of electrical service agreements with a utility provider, circuit breaker ratings of one or more distribution equipment, transformer capacity ratings, or thermal limits of power distribution conductors within the data center 201, among others. For example, the power allocation limits supported by the data center 201 can include a maximum power threshold specified in an electrical service agreement that restricts total power draw to a contractual limit expressed in megawatts or a circuit breaker rating that defines a maximum permissible current flow through a primary distribution panel before triggering a power disconnection for protection of the electrical equipment. In some cases, the power allocation limits supported by the data center 201 can be predefined values stored in the data repository 222 or can be adjusted dynamically by the data processing system 150 based on at least one of changes in operating conditions at the electricity distribution grid, updates to electrical service agreements, or detected changes in equipment ratings, for example.

[0314] In some scenarios, the data processing system 150 can identify a first subset of the amount of power provisioned to the rack cabinet and a second subset of the amount of power provisioned to one or more additional rack cabinets. Each subset of the power provisioned to the respective rack cabinet(s) can correspond to at least a portion of the total amount of power provisioned to the data center 201 and used by the rack cabinets. In some cases, a portion of the total amount of power provisioned to the data center 201 can be allocated for other components, not limited to the rack cabinets, such as ventilation or cooling systems of the data center 201, lighting systems, or electrical devices connected to outlets of the data center 201. In some configurations, the total amount of power provisioned to the data center 201 can refer to the power envelope configured for processing operations of the rack cabinets.

[0315] At ACT 2206, the data processing system 150 (e.g., power predictor 212) can predict, for a second time window subsequent to the first time window, a first amount of power for a processing operation by the one or more processing units of the rack cabinet and a second amount of power for processing operations by one or more additional racks of the data center 201. The amount of power for processing operation(s) can refer to the amount of power to be consumed (e.g., power consumption amount) by component(s) such as the one or984913-1418-6318.1Docket No. 102555-0261more processing units to perform the processing operation(s). The processing operation can refer to a computational task executed by the one or more processing units to generate an output. The computational task can be defined by at least one of a workload type, a priority level, a service level agreement (SLA), or a data flow received by the one or more processing units. In this case, the second time window can refer to a future time window that has not already occurred.

[0316] The data processing system 150 can perform the prediction via or by executing at least one artificial intelligence model using one or more inputs. For example, the input can include (i) at least one characteristic of operation executed on the one or more processing units, (ii) the electric waveform data, and (iii) the amount of power provisioned to the data center 201 and used by the rack cabinets. In some implementations, a characteristic of operation can refer to metadata describing how processing unit(s) are expected to operate, including workload type, SLA, priority, or incoming data flow, among others, which can influence predicted power consumption, and it should be noted that other types of characteristics of operations can be included, not limited to those discussed herein. Other input information can be provided to the artificial intelligence model to predict the amount of power for processing operations associated with the rack cabinet and one or more other rack cabinets. The amount of power provisioned to the data center 201 can represent an upper bound on total power available for distribution across all rack cabinets, such that the artificial intelligence model can use this input (e.g., constraint) to identify a maximum amount of power to be used by the rack cabinets for predicting the first amount of power and the second amount of power. The sum of at least the first amount of power and the second amount of power may be below the maximum amount of power available to the rack cabinets.

[0317] The at least one characteristic of operation can include priorities associated with the respective rack cabinets, such that the data processing system 150 can predict, via executing the artificial intelligence model using (i) the priorities associated with the respective rack cabinets, (ii) the electric waveform data, and (iii) the amount of power, for the second time window, the first amount of power for the processing operation by the one or more processing units of the rack cabinet and the second amount of power for processing operations by the one or more additional racks of the data center 201. For example, the priorities can refer to at least one of numerical or categorical rankings assigned to respective rack cabinets, servers, processing units, or processing operations to indicate at least one of994913-1418-6318.1Docket No. 102555-0261scheduling precedence, resource allocation precedence, or relative urgency for execution. Relatively higher-priority rack cabinets may receive preferential access to available power capacity or compute resources when multiple rack cabinets compete for the same provisioned power subset. Relatively lower-priority rack cabinets may receive reduced power allocations during periods of constrained provisioned capacity. In some instances, the data processing system 150 may reallocate power from lower-priority rack cabinets to higher-priority rack cabinets to satisfy performance targets associated with relatively critical processing operations. These expected allocation of compute resources depending on the priorities of the rack cabinets or the types of servers can be accounted for by the artificial intelligence model to perform the prediction.

[0318] In some configurations, the at least one characteristic of operation can include a Service Level Agreement (SLA) associated with at least the one or more processing units. The SLA can represent a respective allocation of compute resources supported by the respective one or more processing units. The data processing system 150 can predict, via executing the artificial intelligence model using (i) the SLA associated with at least the one or more processing units, (ii) the electric waveform data, and (iii) the amount of power, for the second time window, the first amount of power for the processing operation by the one or more processing units of the rack cabinet and the second amount of power for processing operations by the one or more additional racks of the data center 201. The SLA may indicate a type of servers assigned to the one or more shelves of the rack cabinets or associated with individual rack cabinets. For instance, the SLA can indicate that a first server may be allocated a predefined first amount of compute resources under a free-tier agreement and a second server may be allocated a predefined second amount of compute resources under a paid-tier agreement, with the second amount of compute resources being relatively higher than the first amount of compute resources. Given the type of server providing different level of services or compute resources, the SLA can be an indicator of an amount of compute resources expected to be consumed by a respective rack cabinet. The artificial intelligence model can consider the SLAs of the processing units, servers, or rack cabinets to predict the power consumption.

[0319] In some implementations, the at least one characteristic of operation can include a data flow that is received by the one or more processing units. The data flow can include, for instance, metadata specifying expected computational intensity of a workload to be1004913-1418-6318.1Docket No. 102555-0261processed, computation resources scheduled for a time window, or the amount of data to be processed. In some cases, the data flow can be indicative of computation resources to be used by the one or more processing units for the processing operation in the second time window. The computation resources can be used to predict the first amount of power consumed by the one or more processors for the processing operation in the rack cabinet. Based on the data flow indicating the expected computational intensity, among others, the artificial intelligence model can utilize the data flow to perform the prediction of power consumption for one or more rack cabinets or one or more processing units of the rack cabinets.

[0320] The artificial intelligence model can be trained by the data processing system 150 (e.g., model manager 210) using historical electric waveform data, power consumption metrics, or operation characteristics collected from prior time windows, to provide a few examples of the training data. The model manager 210 can execute a supervised learning procedure in which input feature vectors comprising the electric waveform data and the operation characteristics are associated with target power consumption values measured during corresponding time windows. The model manager 210 can execute other training techniques to train the artificial intelligence model. In some cases, the model manager 210 can receive or retrieve a pre-trained model from another device.

[0321] Once deployed, the artificial intelligence model can receive input data such as from the data collector 204, the power envelope identifier 206, the data processor 208, or other components. The artificial intelligence model can generate forecast values representing predicted power consumption for a subsequent time window. The artificial intelligence model can use the at least one characteristic of operation to perform the prediction by incorporating metadata describing workload types, priority levels, SLAs, or data flows into input feature vectors that are processed by one or more neural network layers to generate an output vector representing predicted power consumption. The workload type can be encoded as a categorical feature affecting the predicted power consumption. The priority level can be encoded as a numerical weight that affects the predicted power consumption. For example, when the at least one characteristic of operation includes a service level agreement (SLA) indicating a high-priority workload having a minimum throughput, the artificial intelligence model can increase a predicted power consumption value for the processing operation relative to a baseline prediction to account for elevated computational intensity associated with satisfying the SLA. In another example, for relatively higher priority rack cabinet or servers,1014913-1418-6318.1Docket No. 102555-0261the artificial intelligence model can increase the predicted power consumption value for the processing operation relative to the baseline prediction. In yet another example, for relatively higher compute operations, as indicated by the data flow, the artificial intelligence model can increase the predicted power consumption value for the processing operation relative to the baseline prediction. Other types of characteristic of operation can be considered by the artificial intelligence model to perform the prediction.

[0322] At ACT 2208, the data processing system 150 (e.g., data processor 208) can determine, based on the first amount of power and the second amount of power from the artificial intelligence model, a third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet at the second time window. In this case, the third amount of power can represent an amount of power to provision the one or more processing units of the rack cabinet. The data processing system 150 (e.g., action manager 214 or controller 220) can determine and provide commands to adjust one or more parameters associated with the rack cabinet provision the third amount of power for the one or more processing units of the rack cabinet. Adjustment to the one or more parameters can be an increase or a decrease in at least one parameter value. In some cases, the data processing system 150 can adjust one or more parameters of other component(s) of the data center 201 additionally or alternatively to the parameters associated with the rack cabinet.

[0323] For example, the data processing system 150 (e.g., data processor 208) can detect that the first amount of power exceeds a subset of the amount of power provisioned to rack cabinet of the data center 201, for instance, by comparing the first amount of power with the subset of the amount of power provisioned to the data center 201. The first amount exceeding the subset of provisioned power allocated for the rack cabinet can indicate a potential power excursion or an increase in power consumption greater than the allowable amount at the rack cabinet at the second time window. The data processing system 150 can determine, responsive to detection of the first amount of power exceeds the subset of the amount of power, via executing the artificial intelligence model using the first amount of power and the second amount of power, the third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet such that the first amount of power predicted for the processing operation by the one or more processing units of the rack cabinet is reduced below the subset of the amount of power at the second time window. In this case, the data processing system 150 (e.g., controller 220) can utilize the third amount of power to1024913-1418-6318.1Docket No. 102555-0261adjust one or more parameters associated with the rack cabinet to reduce the power consumption (e.g., the first amount of power) at the second time window is below the power envelope (e.g., the subset of the amount of power) for the rack cabinet. In some other cases, the data processing system 150 (e.g., controller 220) may increase the power provisioned for the rack cabinet to account for the first amount of power at the second time window. The adjusted parameter(s) can be temporary or permanent based on system configuration or specification of the component of the data center 201.

[0324] In some implementations, the data processing system 150 can detect that the second amount of power exceeds a subset of the amount of power provisioned to the one or more additional racks of the data center 201. The data processing system 150 can determine, responsive to detection of the second amount of power exceeds the subset of the amount of power, via executing the artificial intelligence model using the first amount of power and the second amount of power, the third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet such that the second amount of power predicted for the processing operations by the one or more additional racks of the data center 201 is reduced below the subset of the amount of power at the second time window. In this case, the data processing system 150 (e.g., action manager 214 or controller 220) can determine to adjust one or more parameters of the rack cabinet, using the third amount of power, by increasing the power envelope of the rack cabinet such that the one or more processing units of the rack cabinet can receive (e.g., offload) computational resources from the one or more additional rack cabinets to reduce power consumption at the one or more additional racks at the second time window. In some cases, the data processing system 150 can use the third amount of power to adjust the parameter by decreasing the power envelope at the rack cabinet, for instance, for increasing the power envelope of the one or more additional rack cabinets, thereby supporting relatively higher computation resources at the second time window, maintaining the second amount of power below the subset of the amount of power provisioned to the one or more additional racks.

[0325] At ACT 2210, the data processing system 150 (e.g., data processor 208) can evaluate whether the third amount of power is different from a current amount of power provisioned to the rack cabinet for the processing operation. The data processing system 150 can compare the third amount of power with the current amount of power to determine a difference. If the data processing system 150 determines that the third amount of power is not1034913-1418-6318.1Docket No. 102555-0261different from the current amount of power (or that the difference is less than a threshold), the method 2200 can return to ACT 2206 to continue predicting power consumption for subsequent time windows. If the data processing system 150 determines that the third amount of power is different from the current amount of power (or the difference is greater than or equal to the threshold), the method 2200 can proceed to ACT 2212. In some configurations, ACT 2210 may be skipped such that the data processing system 150 can proceed directly to ACT 2212 from ACT 2208 to update the one or more parameters. In some other cases, the third amount of power being the same as (or within a threshold range of) the current amount of power provisioned to the rack cabinet can be a trigger for the data processing system 150 to re-execute the artificial intelligence model or in some cases retrain the artificial intelligence model. In some implementations, the third amount of power being substantially the same as the current amount of power can indicate that the one or more parameters are to be maintained.

[0326] At ACT 2212, the data processing system 150 (e.g., controller 220) can adjust, based on the third amount of power, one or more parameters associated with the rack cabinet at the second time window. The one or more parameters can include at least one of a frequency of a processing unit, a voltage of a processing unit, a power limit of a processing unit, a frequency of a memory in electrical communication with a processing unit, a capacity of the memory, or a scheduling of compute resources for the processing operations of the one or more processing units, the frequency or voltage of the processing unit can include or be a part of a DVFS of the processing unit. Other parameters can be adjusted not limited to those discussed herein. For example, the data processing system 150 can adjust the one or more parameters by increasing or decreasing at least one of the frequency of the processing unit, the voltage of the processing unit, the power limit of the processing unit, the frequency of the memory, or the capacity of the memory based on the third amount of power provisioned for the processing operation. In some configurations, the data processing system 150 (e.g., action manager 214) can determine which of the one or more parameters to adjust based on a response latency associated with each parameter, a predicted magnitude of power consumption change resulting from adjustment of each parameter, or a current operational state of the one or more processing units, for example. The response latency can represent a time interval between transmission of a control signal to adjust a parameter and detection of a corresponding change in measured power consumption, such that parameters having1044913-1418-6318.1Docket No. 102555-0261relatively shorter response latencies can be selected when a rapid reduction in power consumption may be desired.

[0327] In further examples, increasing a power limit of the processing unit can include adjusting from 300 watts to 350 watts based on the third amount of power for provisioning the processing operation supporting the relatively higher power consumption. Reducing the power limit of the processing unit can include adjusting from 350 watts to 275 watts based on the reduced third amount of power provisioning the processing operation. Adjusting the frequency of the processing unit can include, e.g., increasing the frequency from 1.5 gigahertz (GHz) to 1.8 GHz or decreasing the frequency from 1.8 GHz to 1.5 GHz. Adjusting the voltage decrease a voltage of the processing unit can include, e.g., increasing from 1.0 volts to 1.2 volts or decreasing from 1.2 volts to 1.0 volts. It should be noted that the values presented hereinabove are used as non-limiting quantitative examples, and that the data processing system 150 (e.g., controller 220) can adjust the parameter(s) to other values depending on at least one of, but not limited to, the power envelope or provisioned power, the capability of the processing unit, or the type of parameter being adjusted.

[0328] The data processing system 150 (e.g., controller 220) can adjust the one or more parameters such that the power envelope of the rack cabinet approaches or reaches the third amount of power. For example, when the third amount of power is greater than a current power envelope value for the rack cabinet, the data processing system 150 can increase at least one parameter value associated with the one or more processing units to allow for relatively higher power consumption up to the third amount of power, thereby utilizing additional available power capacity within the provisioned power for the data center 201. In another example, when the third amount of power is less than the current power envelope value for the rack cabinet, the data processing system 150 can decrease at least one parameter value associated with the one or more processing units to reduce power consumption below the third amount of power, thereby preventing power excursions and maintaining operation within the updated power envelope. The data processing system 150 can adjust the power envelope by modifying at least one of a power limit of one or more processing units, a frequency of the one or more processing units, or a voltage of the one or more processing units to bring the actual power consumption into alignment with the third amount of power, for example. Other parameters can be adjusted to increase or decrease the power provisioned to the one or more processing units.1054913-1418-6318.1Docket No. 102555-0261

[0329] In some configurations, adjusting the power envelope can include adjusting the total amount of power provisioned to the component. In some configurations, adjusting the power envelope can include adjusting a reserved portion of the total amount of provisioned power, e.g., reducing the reserved portion increases the amount of power allowed for consumption and increasing the reserved portion reduces the amount of power allowed for consumption. In some configurations, adjusting the power envelope can be associated with, linked to, or mapped to the adjustment of computational power of the processing unit.

[0330] In some implementations, when the forecast values indicate that additional power headroom may be available within the power envelope value or that increasing compute resource allocation would not exceed the power envelope value at the second time window, the data processing system 150 (e.g., controller 220) can increase at least one value associated with the one or more parameters. In another example, the data processing system 150 can generate a signal comprising an instruction to adjust the power limit of the processing unit based on the forecast values. The power limit can be increased based on an indication that total power consumption remains below the power envelope value minus a predefined safety margin during the second time window, for example. The data processing system 150 can transmit the signal to the processing unit via the interface 202 to adjust the power limit (or other parameters) prior to the start of the second time window. In some cases, the data processing system 150 can transmit the signal to the processing unit to adjust the power limit (or other parameters) at or within the second time window. Within the second time window can refer to during the duration of the second time window while the power consumption is increasing or decreasing towards the predicted power consumption value, for example.

[0331] In some implementations, the data processing system 150 (e.g., controller 220) can adjust the one or more parameters by decreasing at least one of the frequency of the processing unit, the voltage of the processing unit, the power limit of the processing unit, the frequency of the memory, or the capacity of the memory when the forecast values indicate that power consumption may exceed the power envelope value during the second time window. For example, the data processing system 150 can generate a signal comprising an instruction to reduce the power limit of the processing based on the forecast values indicating that total power consumption may exceed the power envelope value minus the safety margin during the second time window. The data processing system 150 can transmit the signal to1064913-1418-6318.1Docket No. 102555-0261the processing unit to maintain the total power consumption at or below the adjusted power limit (within the power envelope) during the second time window. In some implementations, the data processing system 150 can adjust the one or more parameters by reallocating compute resources from one processing unit to another processing unit when the forecast values indicate that a first processing unit may consume power below a target utilization level and a second processing unit may require additional compute resources. For example, the data processing system 150 can transfer at least one processing task from the second processing unit to the first processing unit such that the first processing unit can execute the processing task using excess available power capacity while the second processing unit reduces power consumption to remain within an allocated power limit.

[0332] The data processing system 150 (e.g., controller 220) can increase a current amount of power provisioned to the one or more processing units of the rack cabinet for the second time window, increase an allocation of compute resources for the processing operation by the one or more processing units for the second time window, or increase at least one of a frequency of the one or more processing units, a voltage of the one or more processing units, a power limit of the one or more processing units, a frequency of a memory in electrical communication with the one or more processing units, or a capacity of the memory. In some implementations, the data processing system 150 can decrease an allocation of compute resources for the processing operations by the one or more additional racks of the data center 201 for the second time window. The data processing system 150 can adjust, based on the third amount of power, scheduling of workload across the rack cabinets for the one or more processing units of the rack cabinet to operate at the third amount of power, for example, by reallocating processing tasks from one rack cabinet to another rack cabinet to distribute power consumption such that individual rack cabinets remain within respective power envelopes.

[0333] In some implementations, the data processing system 150 can determine, based on the first amount of power and the second amount of power from the artificial intelligence model, and the at least one characteristic of operation including an SLA associated with at least the one or more processing units, the third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet at the second time window. In such cases, the data processing system 150 can adjust, at the second time window, the one or more parameters to increase a current amount of power for the rack1074913-1418-6318.1Docket No. 102555-0261cabinet to the third amount of power for supporting a higher allocation of compute resources relative to the one or more additional racks of the data center 201 based on the SLA. For instance, the artificial intelligence model can utilize or account for the SLA (or other characteristic of operation) to determine the third amount of power to provision for the rack cabinet. The SLA can indicate whether the rack cabinet can be assigned a relatively higher allocation of provisioned power (e.g., based on performance designated for the rack cabinet according to the SLA). For instance, the rack cabinets associated with paid-tier SLAs may receive priority access to available power capacity over rack cabinets associated with free-tier SLAs when the data processing system 150 determines the amount of power to provision across the rack cabinets in the data center 201.

[0334] In some aspects, the data processing system 150 can obtain, at the second time window, second electric waveform data measured for the rack cabinet. The second electric waveform data can be the actual measured electric waveform data at the second time window after adjusting the one or more parameters. The data processing system 150 (e.g., model manager 210) can update the artificial intelligence model using at least the first amount of power and the second amount of power predicted for the second time window, the second electric waveform data obtained at the second time window, and the one or more parameters associated with the rack cabinet adjusted at the second time window. For example, the artificial intelligence model can be updated by computing residuals between the forecasted power consumption values generated by the model for the second time window and the actual power measurements obtained from the electric waveform data captured during the second time window, and adjusting model parameters to reduce the magnitude of the residuals in subsequent predictions. The data processing system 150 can utilize one or more machine learning techniques to train the model. After training or retraining the model, the data processing system 150 can deploy the artificial intelligence model for subsequent time windows (e.g., at least a third time window subsequent to the second time window).

[0335] FIG. 23 is an example flow diagram of a method 2300 for managing controller state, in accordance with an implementation. The example method 2300 can be executed, performed, or otherwise carried out by the data processing system 150, one or more components of the utility grid 100 (e.g., computing device, metering devices 118, data processing system 150B, etc.), one or more components of the data center 201 (e.g., data processing system 150A, server, etc.), other components of the system 200, or computing1084913-1418-6318.1Docket No. 102555-0261device 2700, among others. Although the data processing system 150 is described herein to execute and / or perform the method 2300, other devices can be configured to perform similar features and / or functionalities as the data processing system 150 to perform the method 2300.

[0336] The data processing system 150 can perform the method 2300 to enforce power envelope constraints by selecting a controller state based on monitored electric waveform data and one or more state variables, computing a power limit for one or more processing units according to the selected controller state, and generating an instruction to update the current power limit. The method 2300 can include monitoring electric waveform data, at ACT 2302. At ACT 2304, the method 2300 can include aggregating the electric waveform data. At ACT 2306, the method 2300 can include identifying a power envelope value. At ACT 2308, the method 2300 can include determining one or more state variables. At ACT 2310, the method 2300 can include determining whether to transition. At ACT 2312, the method 2300 can include selecting a controller state. At ACT 2314, the method 2300 can include computing a second power limit. At ACT 2316, the method 2300 can include generating an instruction.

[0337] At ACT 2302, the data processing system 150 (e.g., data collector 204) can monitor, via one or more measurement circuits, electric waveform data for electric power delivered to a server on a shelf in a rack cabinet. The one or more measurement circuits can be disposed at a power distribution unit electrically coupled to the rack cabinet or at a power supply unit electrically coupled to the rack cabinet, one or more shelves of the rack cabinet, or individual processing units such as to capture voltage waveform data or current waveform data. The server can be one of a plurality of servers distributed across the shelves in the rack cabinet, where each shelf can include one or more processing units such as GPUs or CPUs. In some cases, the data processing system 150 can receive the electric waveform data at a sampling rate corresponding to at least one cycle of AC waveform data. In some other cases, the data processing system 150 can receive the electric waveform data at a sampling rate corresponding to a portion of a cycle of the AC waveform data. The electric waveform data can include time-series samples of voltage or current measured at one or more power input ports associated with the one or more processing units. The data processing system 150 can monitor the electric waveform data at other locations or receive the electric waveform data from other devices.1094913-1418-6318.1Docket No. 102555-0261

[0338] At ACT 2304, the data processing system 150 (e.g., data processor 208) can aggregate the electric waveform data to estimate a total power for a control iteration. The data processing system 150 can aggregate the electric waveform data by combining power measurements from a plurality of sources associated with the server or one or more processing units relevant to or associated with a power envelope managed by the data processing system 150. A power source can be relevant to the power envelope when electrical power drawn from that power source contributes to the total power consumption subject to the power envelope constraint, such that the data processing system 150 can aggregate measurements from each relevant power source to determine the total power consumed during the control iteration.

[0339] For example, the data processing system 150 can sum a first power value corresponding to a first processing unit and a second power value corresponding to a second processing unit of a shelf or a rack cabinet to generate the total power estimate for the control iteration. The data processing system 150 can sum power values corresponding to other processing units to generate the total power estimate for the control iteration. In some implementations, the data processing system 150 can compute root mean square (RMS) values from the electric waveform data for individual power sources over a predefined time window prior to summation, such that the total power estimate represents an RMS power consumption across all power sources during the predefined time window. The data processing system 150 can initiate the control iteration upon completing the aggregation operation, such that the control iteration can proceed using the total power estimate as an input to subsequent state variable determination and controller state selection operations.

[0340] The data processing system 150 can cache a latest power sample from each of a plurality of power sources associated with the server, for example, by maintaining a buffer of most recent power measurements from each measurement circuit coupled to the server. The data processing system 150 can replace the latest power sample with another latest power sample in subsequent iteration or cycle, e.g., cached data can be removed or replaced with new data. Responsive to receipt of updated samples from the plurality of power sources, the data processing system 150 can estimate the total power as a sum of latest power samples of the plurality of power sources and initiate the control iteration. The control iteration can be initiated at a frequency corresponding to the update rate of the electric waveform data1104913-1418-6318.1Docket No. 102555-0261samples, for example, once per AC cycle or at a higher frequency when sub-cycle measurements are available.

[0341] At ACT 2306, the data processing system 150 (e.g., power envelope identifier 206) can identify a power envelope value for the control iteration. The power envelope can represent a maximum amount of power provisioned for use by the server (e.g., one or more processing units executing processing operations for the server) during the control iteration and may be specified in a power allocation record stored in a data repository 222 or received from a higher-level power orchestration controller. In some implementations, the power envelope value can vary over time according to changes to power allocation policy that adjusts provisioned power based on workload characteristics, service level agreements (SLAs), or grid conditions, among others. In some cases, the power envelope value can be configured or adjusted by the data processing system 150 (e.g., controller 220) or other devices. The data processing system 150 can retrieve the power envelope value from a configuration parameter storage or receive the power envelope value via a communication interface from an external orchestration system. The power envelope value can be defined as a total power limit (e.g., including any reserved amount of power) applicable to the processing unit(s) of the server or as a set of per-component power limits that sum to the total power limit.

[0342] At ACT 2308, the data processing system 150 (e.g., data processor 208) can determine, based on or using the total power and the power envelope value, one or more state variables comprising a headroom value. The headroom value can include or refer to an amount of power that is available for consumption or that has not been consumed by the processing unit(s). For instance, the headroom value can be determined based on a difference between the power envelope value and the total power. The headroom value can indicate whether the total power satisfies the power envelope value or exceeds the power envelope value. For example, a positive headroom value (e.g., power envelop value minus the total power) can indicate that the total power may be less than the power envelope value by the magnitude (e.g., difference value) of the headroom value. In another example, a negative headroom value can indicate that an excursion has occurred in which the total power exceeds the power envelope value.

[0343] The one or more state variables can include at least one of a data integrity flag, a (power) moving average of the total power, a rate of change of the total power, or a time to1114913-1418-6318.1Docket No. 102555-0261excursion. The data integrity flag can indicate whether new electric waveform data may be missing or invalid based on timestamp discontinuities or out-of-range measurements. The moving average of the total power can be computed as an exponential moving average over a plurality of control iterations. The moving average can refer to a weighted average of the total power computed over multiple control iterations. In some implementations, the moving average can be computed as an exponential moving average in which more recent power samples receive greater weight than older power samples, and the decay factor applied to older samples can be adjusted based on the time elapsed since the last power sample such that the moving average reflects the recency and temporal spacing of the measurements.

[0344] The rate of change of the total power can be determined by computing a difference between consecutive total power samples and dividing by a time interval between the samples, for example. The time to excursion can refer to an amount of time remaining before an excursion will occur under the current rate of change. The time to excursion can be computed by dividing the headroom value by the rate of change of the total power. The time to excursion may be computed when both the headroom value and the rate of change of the total power are positive. The time to excursion may not be computed when at least one of the headroom value or the rate of change of the total power is a negative value. The data processing system 150 can use other values or information to determine the one or more state variables. There can be other state variables not limited to those discussed herein.

[0345] At ACT 2310, the data processing system 150 (e.g., controller state manager 216) can evaluate whether a state transition should occur based on at least the one or more state variables and predefined transition criteria. The transition criteria can specify conditions under which the controller state is to be changed from a current state to a different state, for example, whether a data integrity flag is present, one or more predefined thresholds for the headroom value, a predefined threshold for the time to excursion, a predefined threshold for the moving average of the total power, or combinations thereof. In some implementations, the transition criteria can be ordered by priority such that conditions higher in the priority order take precedence over conditions lower in the priority order. For example...

Claims

Docket No. 102555-0261What is Claimed is:

1. A system to provision power used by rack cabinets in a data center, comprising: one or more processors coupled with memory, to:obtain, at a first time window, electric waveform data measured for a rack cabinet in the data center, the rack cabinet comprising a plurality of shelves, each of the plurality of shelves comprising a respective one or more processing units;identify, for the first time window, an amount of power provisioned to the data center and used by the rack cabinets;predict, via executing an artificial intelligence model using (i) at least one characteristic of operation executed on the one or more processing units, (ii) the electric waveform data, and (iii) the amount of power, for a second time window subsequent to the first time window, a first amount of power for a processing operation by the one or more processing units of the rack cabinet and a second amount of power for processing operations by one or more additional racks of the data center;determine, based on the first amount of power and the second amount of power from the artificial intelligence model, a third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet at the second time window; andadjust, based on the third amount of power, one or more parameters associated with the rack cabinet at the second time window.

2. The system of claim 1, wherein the one or more processors further:monitor, via one or more measurement circuits disposed at a power distribution unit electrically coupled to the rack cabinet, voltage waveform data or current waveform data associated with electric power delivered to the plurality of shelves of the rack cabinet.

3. The system of claim 1, wherein the one or more processors further:monitor, via one or more measurement circuits disposed at a power supply unit electrically coupled to the rack cabinet, voltage waveform data or current waveform data associated with electric power delivered to the plurality of shelves of the rack cabinet.

4. The system of claim 1, wherein the one or more processors further:1794913-1418-6318.1Docket No. 102555-0261convert the electric waveform data to at least one of a quantitative value representing an amount of power consumed at the rack cabinet, a load variability, or an output from a fast Fourier transform (FFT) at the first time window.

5. The system of claim 1, wherein the one or more processors further:monitor the amount of power at a metering device electrically coupled with the rack cabinets used in the data center, wherein the amount of power represents a total power available for use by processing units of the rack cabinets for the processing operations, and wherein the amount of power is affected by at least one of time of day, weather condition, operating conditions at an electricity distribution grid, or power allocation limits supported by the data center.

6. The system of claim 1, wherein the at least one characteristic of operation comprises priorities associated with the respective rack cabinets, and wherein the one or more processors further:predict, via executing the artificial intelligence model using (i) the priorities associated with the respective rack cabinets, (ii) the electric waveform data, and (iii) the amount of power, for the second time window, the first amount of power for the processing operation by the one or more processing units of the rack cabinet and the second amount of power for processing operations by the one or more additional racks of the data center.

7. The system of claim 1, wherein the at least one characteristic of operation comprises a Service Level Agreement (SLA) associated with at least the one or more processing units, the SLA representing a respective allocation of compute resources supported by the respective one or more processing units, and wherein the one or more processors further:predict, via executing the artificial intelligence model using (i) the SLA associated with at least the one or more processing units, (ii) the electric waveform data, and (iii) the amount of power, for the second time window, the first amount of power for the processing operation by the one or more processing units of the rack cabinet and the second amount of power for processing operations by the one or more additional racks of the data center.

8. The system of claim 1, wherein the at least one characteristic of operation comprises a data flow directly received by the one or more processing units, the data flow indicative of1804913-1418-6318.1Docket No. 102555-0261computation resources to be used by the one or more processing units for the processing operation in the second time window, and wherein the one or more processors further:detect that the first amount of power exceeds a subset of the amount of power provisioned to the rack cabinet of the data center; anddetermine, responsive to detection of the first amount of power exceeds the subset of the amount of power, via executing the artificial intelligence model using the first amount of power and the second amount of power, the third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet such that the first amount of power predicted for the processing operation by the one or more processing units of the rack cabinet is reduced below the subset of the amount of power at the second time window.

9. The system of claim 1, wherein the one or more processors further:detect that the second amount of power exceeds a subset of the amount of power provisioned to the one or more additional racks of the data center; anddetermine, responsive to detection of the second amount of power exceeds the subset of the amount of power, via executing the artificial intelligence model using the first amount of power and the second amount of power, the third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet such that the second amount of power predicted for the processing operations by the one or more additional racks of the data center is reduced below the subset of the amount of power at the second time window.

10. The system of claim 1, wherein the one or more processors further:increase a current amount of power provisioned to the one or more processing units of the rack cabinet for the second time window;increase an allocation of compute resources for the processing operation by the one or more processing units for the second time window; orincrease at least one of a frequency of the one or more processing units, a voltage of the one or more processing units, a power limit of the one or more processing units, a frequency of a memory in electrical communication with the one or more processing units, or a capacity of the memory.

11. The system of claim 1, wherein the one or more processors further:1814913-1418-6318.1Docket No. 102555-0261decrease an allocation of compute resources for the processing operations by the one or more additional racks of the data center for the second time window.

12. The system of claim 1, wherein the one or more processors further:adjust, based on the third amount of power, scheduling of workload across the rack cabinets for the one or more processing units of the rack cabinet to operate at the third amount of power.

13. The system of claim 1, wherein the one or more processors further:determine, based on the first amount of power and the second amount of power from the artificial intelligence model, and the at least one characteristic of operation comprising a Service Level Agreement (SLA) associated with at least the one or more processing units, the third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet at the second time window; andadjust, at the second time window, the one or more parameters to increase a current amount of power for the rack cabinet to the third amount of power for supporting a higher allocation of compute resources relative to the one or more additional racks of the data center based on the SLA.

14. The system of claim 1, wherein the one or more processing units comprise at least one of graphics processing units (GPUs) or central processing units (CPUs) for the processing operation.

15. The system of claim 1, wherein the one or more processors further:obtain, at the second time window, second electric waveform data measured for the rack cabinet;update the artificial intelligence model using at least the first amount of power and the second amount of power predicted for the second time window, the second electric waveform data obtained at the second time window, and the one or more parameters associated with the rack cabinet adjusted at the second time window; anddeploy the artificial intelligence model subsequent to updating for executing a prediction for a third time window subsequent to the second time window.

16. A method for provisioning power used by rack cabinets in a data center, comprising:1824913-1418-6318.1Docket No. 102555-0261obtaining, by one or more processors coupled with memory, at a first time window, electric waveform data measured for a rack cabinet in the data center, the rack cabinet comprising a plurality of shelves, each of the plurality of shelves comprising a respective one or more processing units;identifying, by the one or more processors, for the first time window, an amount of power provisioned to the data center and used by the rack cabinets;predicting, by the one or more processors, via executing an artificial intelligence model using (i) at least one characteristic of operation executed on the one or more processing units, (ii) the electric waveform data, and (iii) the amount of power, for a second time window subsequent to the first time window, a first amount of power for a processing operation by the one or more processing units of the rack cabinet and a second amount of power for processing operations by one or more additional racks of the data center;determining, by the one or more processors, based on the first amount of power and the second amount of power from the artificial intelligence model, a third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet at the second time window; andadjusting, by the one or more processors, based on the third amount of power, one or more parameters associated with the rack cabinet at the second time window.

17. The method of claim 16, further comprising:monitoring, by the one or more processors, via one or more measurement circuits disposed at a power distribution unit electrically coupled to the rack cabinet, voltage waveform data or current waveform data associated with electric power delivered to the plurality of shelves of the rack cabinet.

18. The method of claim 16, further comprising:monitoring, by the one or more processors, via one or more measurement circuits disposed at a power supply unit electrically coupled to the rack cabinet, voltage waveform data or current waveform data associated with electric power delivered to the plurality of shelves of the rack cabinet.

19. A non-transitory computer-readable medium storing processor executable instructions that, when executed by one or more processors, cause the one or more processors to:1834913-1418-6318.1Docket No. 102555-0261obtain, at a first time window, electric waveform data measured for a rack cabinet of rack cabinets in a data center, the rack cabinet comprising a plurality of shelves, each of the plurality of shelves comprising a respective one or more processing units;identify, for the first time window, an amount of power provisioned to the data center and used by the rack cabinets;predict, via executing an artificial intelligence model using (i) at least one characteristic of operation executed on the one or more processing units, (ii) the electric waveform data, and (iii) the amount of power, for a second time window subsequent to the first time window, a first amount of power for a processing operation by the one or more processing units of the rack cabinet and a second amount of power for processing operations by one or more additional racks of the data center;determine, based on the first amount of power and the second amount of power from the artificial intelligence model, a third amount of power for provisioning the processing operation by the one or more processing units of the rack cabinet at the second time window; andadjust, based on the third amount of power, one or more parameters associated with the rack cabinet at the second time window.

20. The non-transitory computer-readable medium of claim 19, wherein the at least one characteristic of operation comprises priorities associated with the respective rack cabinets, and wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:predict, via executing the artificial intelligence model using (i) the priorities associated with the respective rack cabinets, (ii) the electric waveform data, and (iii) the amount of power, for the second time window, the first amount of power for the processing operation by the one or more processing units of the rack cabinet and the second amount of power for processing operations by the one or more additional racks of the data center.

21. A system, comprising:one or more processors, coupled with memory, to:monitor, via one or more measurement circuits, electric waveform data for electric power delivered to a server on a shelf in a rack cabinet;aggregate the electric waveform data to estimate a total power for a control iteration;1844913-1418-6318.1Docket No. 102555-0261identify a power envelope value for the control iteration;determine, based on the total power and the power envelope value, one or more state variables comprising a headroom value;select a controller state based on the one or more state variables and the headroom value;compute a second power limit to request for one or more processing units of the server based on the controller state, the total power, the power envelope value, and a current power limit of the one or more processing units; andgenerate an instruction to update the current power limit of the one or more processing units based on the second power limit.

22. The system of claim 21, wherein the one or more processors further:determine the headroom value based on a difference between the power envelope value and the total power, the headroom value indicating whether the total power satisfies the power envelope value or is predicted to exceed the power envelope value.

23. The system of claim 21, wherein the one or more processors further:determine the one or more state variables comprising at least one of a data integrity flag, a moving average of the total power, a rate of change of the total power, or a time to excursion.

24. The system of claim 21, wherein the one or more processors further:select the controller state from at least one of an idle state, a ramp up state, a regulate state, a ramp down state, or an excursion state.

25. The system of claim 21, wherein the controller state is a ramp up state, and the one or more processors further:determine, based on the ramp up state and the current power limit of the one or more processing units of the server, a rate of increase from the current power limit to the second power limit to request for the one or more processing units.

26. The system of claim 21, wherein the controller state is one of a regulate state, a ramp down state, or an excursion state, and the one or more processors further:1854913-1418-6318.1Docket No. 102555-0261determine, based on the controller state and the current power limit of the one or more processing units of the server, the second power limit using a proportional derivative control based on a measured power, a margin, and the power envelope value.

27. The system of claim 21, wherein the one or more processors further:clip the second power limit to a range; andapply a cooldown between commands to adjust power limits, wherein the cooldown is between transmission of the instruction and at least one subsequent instruction.

28. The system of claim 21, wherein the one or more processors further:cache a latest power sample from each of a plurality of power sources associated with the server; andresponsive to receipt of updated samples from the plurality of power sources, (i) estimate the total power as a sum of latest power samples of the plurality of power sources, and (ii) initiate the control iteration.

29. The system of claim 21, wherein the one or more processors further:quantize the second power limit to discrete increments prior to generation of the instruction to update the current power limit.

30. The system of claim 21, wherein the one or more processors further:apply a cooldown period between instructions to update the current power limit; and responsive to the controller state being one of an excursion state or a ramp down state, generate the instruction during the cooldown period.

31. The system of claim 21, wherein the one or more processors further:apply a cooldown between commands to adjust power limits;select a second controller state in a second control iteration based on the one or more state variables and the headroom value, the second controller state is one of a ramp down state or an excursion state; andgenerate, during the cooldown, a second instruction to update the second power limit of the one or more processing units based on a third power limit compute for the second control iteration.1864913-1418-6318.1Docket No. 102555-026132. The system of claim 21, wherein the controller state is one of a ramp down state or an excursion state, and the one or more processors further:generate the instruction to reduce the current power limit of the one or more processing units to the second power limit such that a measured power returns to the power envelope value minus a margin within a predefined time window.

33. A method, comprising:monitoring, by one or more processors coupled with memory, via one or more measurement circuits, electric waveform data for electric power delivered to a server on a shelf in a rack cabinet;aggregating, by the one or more processors, the electric waveform data to estimate a total power for a control iteration;identifying, by the one or more processors, a power envelope value for the control iteration;determining, by the one or more processors, based on the total power and the power envelope value, one or more state variables comprising a headroom value;selecting, by the one or more processors, a controller state based on the one or more state variables and the headroom value;computing, by the one or more processors, a second power limit to request for one or more processing units of the server based on the controller state, the total power, the power envelope value, and a current power limit of the one or more processing units; and generating, by the one or more processors, an instruction to update the current power limit of the one or more processing units based on the second power limit.

34. The method of claim 33, further comprising:determining, by the one or more processors, the headroom value based on a difference between the power envelope value and the total power, the headroom value indicating whether the total power satisfies the power envelope value or is predicted to exceed the power envelope value.

35. The method of claim 33, further comprising:determining, by the one or more processors, the one or more state variables comprising at least one of a data integrity flag, a moving average of the total power, a rate of change of the total power, or a time to excursion.1874913-1418-6318.1Docket No. 102555-026136. The method of claim 33, further comprising:selecting, by the one or more processors, the controller state from at least one of an idle state, a ramp up state, a regulate state, a ramp down state, or an excursion state.

37. The method of claim 33, wherein the controller state is a ramp up state, and the method further comprises:determining, by the one or more processors, based on the ramp up state and the current power limit of the one or more processing units of the server, a rate of increase from the current power limit to the second power limit to request for the one or more processing units.

38. A non-transitory computer-readable medium storing processor executable instructions that, when executed by one or more processors, cause the one or more processors to:monitor, via one or more measurement circuits, electric waveform data for electric power delivered to a server on a shelf in a rack cabinet;aggregate the electric waveform data to estimate a total power for a control iteration; identify a power envelope value for the control iteration;determine, based on the total power and the power envelope value, one or more state variables comprising a headroom value;select a controller state based on the one or more state variables and the headroom value;compute a second power limit to request for one or more processing units of the server based on the controller state, the total power, the power envelope value, and a current power limit of the one or more processing units; andgenerate an instruction to update the current power limit of the one or more processing units based on the second power limit.

39. The non-transitory computer-readable medium of claim 38, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:determine, based on the controller state and the current power limit of the one or more processing units of the server, the second power limit using a proportional derivative control based on a measured power, a margin, and the power envelope value.1884913-1418-6318.1Docket No. 102555-026140. The non-transitory computer-readable medium of claim 38, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:clip the second power limit to a range; andapply a cooldown between commands to adjust power limits, wherein the cooldown is between transmission of the instruction and at least one subsequent instruction.

41. A system, comprising:one or more processors coupled with memory, to:receive first electric waveform data measured at a plurality of devices distributed across a power chain of a data center;receive second electric waveform data measured for a rack cabinet of rack cabinets in the data center, the rack cabinet comprising a plurality of shelves, each of the plurality of shelves comprising a respective one or more processing units, wherein the second electric waveform data is indicative of power consumed for processing operations by the respective one or more processing units across the plurality of shelves of the rack cabinet;correlate the first electric waveform data measured at the plurality of devices and the second electric waveform data measured for the rack cabinet;detect at least one power condition based on correlation of the first electric waveform data and the second electric waveform data; andinitiate one or more actions for at least one of the plurality of devices or at least one of the one or more processing units according to the at least one power condition.

42. The system of claim 41, wherein the first electric waveform data comprise alternating current (AC) data and the second electric waveform data comprise direct current (DC) data.

43. The system of claim 41, wherein the plurality of devices distributed across the power chain comprise at least one of a utility feed, an automatic transfer switch (ATS), an uninterruptible power supply (UPS), a generator, a circuit breaker, a power distribution unit (PDU), a power supply unit (PSU), or one or more of the rack cabinets.

44. The system of claim 41, wherein the one or more processors further:1894913-1418-6318.1Docket No. 102555-0261compute at least one root mean square (RMS) value using one or more functions on the second electric waveform data measured for the rack cabinet; anddetermine, based on the at least one RMS value, the power consumed for the processing operations by the respective one or more processing units.

45. The system of claim 41, wherein the second electric waveform data is measured at one or more of (i) one or more power conversion stages of a power supply unit (PSU) supplying power to the rack cabinet, or (ii) at least one power input port associated with the one or more processing units.

46. The system of claim 41, wherein the one or more processors further:align timestamps between the first electric waveform data and the second electric waveform data; anddetermine a relationship between changes in the first electric waveform data and changes in the second electric waveform data in a same time window to correlate the first electric waveform data and the second electric waveform data.

47. The system of claim 41, wherein one or more processors further:detect that changes in the first electric waveform data are caused by changes in the second electric waveform data; andadjust, responsive to the detection, one or more parameters associated with at least one of the one or more processing units, the one or more parameters comprising at least one of a frequency of the one or more processing units, a voltage of the one or more processing units, a power limit of the one or more processing units, or a scheduling of compute resources for the processing operations of the one or more processing units.

48. The system of claim 41, wherein the one or more processors further:detect that changes in the first electric waveform data are independent from changes in the second electric waveform data; andresponsive to the detection, one of:adjust first one or more parameters associated with at least one of the plurality of devices based on the at least one power condition detected from the first electric waveform data; or1904913-1418-6318.1Docket No. 102555-0261adjust second one or more parameters associated with at least one of the one or more processing units based on the at least one power condition detected from the second electric waveform data.

49. The system of claim 41, wherein the at least one power condition comprises at least one of:(i) a load variability being greater than or equal to a threshold, the load variability corresponding to a difference between a maximum power and an average power over a time window,(ii) a fluctuation in power at or above a magnitude threshold or rate-of-change threshold,(iii) an amount of power available for consumption is less than a threshold based on a difference between a first amount of power provisioned for the rack cabinet and a second amount of power consumed for the processing operations by the one or more processing units of the rack cabinet, or(iv) an indication that an amount of power being consumed has increased over a threshold relative to an amount of power provisioned for at least one of the rack cabinets of the data center, a subset of the rack cabinets of the data center, the rack cabinet, at least one of the plurality of shelves, or the one or more processing units of at least one of the plurality of shelves.

50. The system of claim 41, wherein the one or more processors further:select the at least one of the plurality of devices;generate a signal comprising one or more parameters associated with the at least one of the plurality of devices according to the at least one power condition; andtransmit the signal to the at least one of the plurality of devices to adjust the one or more parameters, the one or more parameters comprising:(i) a target power level for power delivery by a power distribution unit (PDU) to one or more loads,(ii) an output power limit for one or more output stages of the PDU,(iii) an enable state, a disable state, or a switching state for at least one output stage of the PDU,(iv) a configuration to control an allocation of available power from the PDU among a plurality of loads powered via the PDU, or1914913-1418-6318.1Docket No. 102555-0261(v) a setpoint of power level for one or more cooling systems of the data center.

51. The system of claim 41, wherein the one or more processors further:generate a signal comprising one or more parameters associated with the one or more processing units according to the at least one power condition; andtransmit the signal to the one or more processing units to adjust the one or more parameters, the one or more parameters comprising at least one of a frequency of a processing unit, a frequency of a memory in electrical communication with the processing unit, a voltage of the processing unit, a power limit of the processing unit, a capacity of the memory, or a scheduling of compute resources for the processing operations of the one or more processing units.

52. The system of claim 41, wherein the one or more processors further:initiate the one or more actions responsive to the at least one power condition satisfying at least one trigger criterion to trigger initiation of the one or more actions, the at least one trigger criterion comprising at least one of:(i) a first amount of power consumed across the power chain of the data center by the rack cabinets being greater than or equal to a first threshold, the first threshold based on an amount of power provisioned to the data center, (ii) a second amount of power consumed for the processing operations by the respective one or more processing units of the rack cabinet being greater than or equal to a second threshold, the second amount of power computed from the second electric waveform data, and the second threshold based on an amount of power provisioned to the rack cabinet,(iii) a first amount of provisioned power available for consumption being less than or equal to a third threshold, or(iv) a second amount of provisioned power available for consumption being greater than or equal to a fourth threshold.

53. The system of claim 41, wherein the one or more processors further:receive feedback signals from at least one of the plurality of devices or at least one of the one or more processing units subsequent to initiating the one or more actions; and dynamically adjust the one or more actions according to the feedback signals.1924913-1418-6318.1Docket No. 102555-026154. A method, comprising:receiving, by one or more processors coupled with memory, first electric waveform data measured at a plurality of devices distributed across a power chain of a data center; receiving, the one or more processors, second electric waveform data measured for a rack cabinet of rack cabinets in the data center, the rack cabinet comprising a plurality of shelves, each of the plurality of shelves comprising a respective one or more processing units, wherein the second electric waveform data is indicative of power consumed for processing operations by the respective one or more processing units across the plurality of shelves of the rack cabinet;correlating, the one or more processors, the first electric waveform data measured at the plurality of devices and the second electric waveform data measured for the rack cabinet;detecting, the one or more processors, at least one power condition based on correlation of the first electric waveform data and the second electric waveform data; and initiating, the one or more processors, one or more actions for at least one of the plurality of devices or at least one of the one or more processing units according to the at least one power condition.

55. The method of claim 54, wherein the first electric waveform data comprise alternating current (AC) data and the second electric waveform data comprise direct current (DC) data.

56. The method of claim 54, wherein the plurality of devices distributed across the power chain comprise at least one of a utility feed, an automatic transfer switch (ATS), an uninterruptible power supply (UPS), a generator, a circuit breaker, a power distribution unit (PDU), a power supply unit (PSU), or one or more of the rack cabinets.

57. The method of claim 54, further comprising:aligning, by the one or more processors, timestamps between the first electric waveform data and the second electric waveform data; anddetermining, by the one or more processors, a relationship between changes in the first electric waveform data and changes in the second electric waveform data in a same time window to correlate the first electric waveform data and the second electric waveform data.

58. A non-transitory computer-readable medium storing processor executable instructions that, when executed by one or more processors, cause the one or more processors to:1934913-1418-6318.1Docket No. 102555-0261receive first electric waveform data measured at a plurality of devices distributed across a power chain of a data center;receive second electric waveform data measured for a rack cabinet of rack cabinets in the data center, the rack cabinet comprising a plurality of shelves, each of the plurality of shelves comprising a respective one or more processing units, wherein the second electric waveform data is indicative of power consumed for processing operations by the respective one or more processing units across the plurality of shelves of the rack cabinet;correlate the first electric waveform data measured at the plurality of devices and the second electric waveform data measured for the rack cabinet;detect at least one power condition based on correlation of the first electric waveform data and the second electric waveform data; andinitiate one or more actions for at least one of the plurality of devices or at least one of the one or more processing units according to the at least one power condition.

59. The non-transitory computer-readable medium of claim 58, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:detect that changes in the first electric waveform data are caused by changes in the second electric waveform data; andadjust, responsive to the detection, one or more parameters associated with at least one of the one or more processing units, the one or more parameters comprising at least one of a frequency of the one or more processing units, a voltage of the one or more processing units, a power limit of the one or more processing units, or a scheduling of compute resources for the processing operations of the one or more processing units.

60. The non-transitory computer-readable medium of claim 58, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:detect that changes in the first electric waveform data are independent from changes in the second electric waveform data; andresponsive to the detection, one of:adjust first one or more parameters associated with at least one of the plurality of devices based on the at least one power condition detected from the first electric waveform data; or1944913-1418-6318.1Docket No. 102555-0261adjust second one or more parameters associated with at least one of the one or more processing units based on the at least one power condition detected from the second electric waveform data.

61. A system, comprising:one or more processors coupled with memory, to:obtain electric waveform data for electric power delivered to a server on a shelf in a rack cabinet;compute, from the electric waveform data, metrics over a plurality of time windows;execute a first neural network using the metrics over the plurality of time windows as input;select a first subset of the metrics corresponding to a first time window of the plurality of time windows;execute a second neural network using the selected first subset of the metrics over the first time window of the plurality of time windows as input;execute a third neural network using a first output from the first neural network and a second output from the second neural network as input;generate, based on a third output from the third neural network, forecast values representing power consumption by one or more processing units of the server over a second time window subsequent to the plurality of time windows; andadjust, for the second time window, one or more parameters of the one or more processing units according to the forecast values.

62. The system of claim 61, wherein the electric waveform data comprise alternating current waveform data.

63. The system of claim 61, wherein the rack cabinet comprises a plurality of shelves including the shelf, each of the plurality of shelves associated with a respective server.

64. The system of claim 61, wherein each of the plurality of time windows comprises a window length corresponding to at least one cycle of the electric waveform data or a portion of a cycle of the electric waveform data.1954913-1418-6318.1Docket No. 102555-026165. The system of claim 61, wherein the plurality of time windows have a first predefined length and the first time window have a second predefined length, the first predefined length corresponds to multiples of the second predefined length, and wherein the second time window have the second predefined length.

66. The system of claim 61, wherein the metrics comprise at least one of root mean square (RMS) values or power values computed from the electric waveform data.

67. The system of claim 61, wherein the first time window corresponds to a latest one of the plurality of time windows.

68. The system of claim 61, wherein the first neural network comprises a convolutional neural network (CNN), the second neural network comprises a first linear neural network, and the third neural network comprises a second linear neural network.

69. The system of claim 61, wherein the first neural network processes the metrics over the plurality of time windows using a one-dimensional convolution technique to generate the first output comprising a first vector representing first one or more characteristics of power consumption over the first time window.

70. The system of claim 61, wherein the second neural network processes the first subset of the metrics using linear transformations to generate the second output comprising a second vector representing second one or more characteristics of power consumption over the plurality of time windows.

71. The system of claim 61, wherein the third neural network processes the first output and the second output using linear transformations to generate the third output comprising a vector representing a combination of first one or more characteristics of power consumption over the first time window and second one or more characteristics of power consumption over the plurality of time windows.

72. The system of claim 61, wherein the one or more processors further:execute a fourth neural network using a third output from the third neural network as input, the fourth neural network comprising a decoder neural network; and1964913-1418-6318.1Docket No. 102555-0261generate, responsive to execution of the fourth neural network, the forecast values of power consumption as a fourth output from the fourth neural network.

73. The system of claim 61, wherein the one or more processors further:compute, from the electric waveform data, second metrics over the second time window; andtrain at least one of the first neural network, the second neural network, or the third neural network using the second metrics and the forecast values from the second time window, wherein one or more weights are applied to differences between the second metrics and the forecast values based on (i) respective magnitude of the differences and (ii) respective forecast values being greater than or less than the second metrics.

74. The system of claim 61, wherein the metrics over the plurality of time windows are stored in the memory, and the one or more processors further:compute, from the electrical waveform data, second metrics over the second time window;store the second metrics of the second time window in the memory; anddiscard a portion of the metrics associated with an earliest one of the plurality of time windows such that the second time window is a latest one of the plurality of time windows.

75. A method, comprising:obtaining, by one or more processors coupled with memory, electric waveform data for electric power delivered to a server on a shelf in a rack cabinet;computing, by the one or more processors, from the electric waveform data, metrics over a plurality of time windows;executing, by the one or more processors, a first neural network using the metrics over the plurality of time windows as input;selecting, by the one or more processors, a first subset of the metrics corresponding to a first time window of the plurality of time windows;executing, by the one or more processors, a second neural network using the selected first subset of the metrics over the first time window of the plurality of time windows as input;executing, by the one or more processors, a third neural network using a first output from the first neural network and a second output from the second neural network as input;1974913-1418-6318.1Docket No. 102555-0261generating, by the one or more processors, based on a third output from the third neural network, forecast values representing power consumption by one or more processing units of the server over a second time window subsequent to the plurality of time windows; andadjusting, by the one or more processors, for the second time window, one or more parameters of the one or more processing units according to the forecast values.

76. The method of claim 75, wherein each of the plurality of time windows comprises a window length corresponding to at least one cycle of the electric waveform data or a portion of a cycle of the electric waveform data.

77. The method of claim 75, wherein the plurality of time windows have a first predefined length and the first time window have a second predefined length, the first predefined length corresponds to multiples of the second predefined length, and wherein the second time window have the second predefined length.

78. A non-transitory computer-readable medium storing processor executable instructions that, when executed by one or more processors, cause the one or more processors to:obtain electric waveform data for electric power delivered to a server on a shelf in a rack cabinet;compute, from the electric waveform data, metrics over a plurality of time windows; execute a first neural network using the metrics over the plurality of time windows as input;select a first subset of the metrics corresponding to a first time window of the plurality of time windows;execute a second neural network using the selected first subset of the metrics over the first time window of the plurality of time windows as input;execute a third neural network using a first output from the first neural network and a second output from the second neural network as input;generate, based on a third output from the third neural network, forecast values representing power consumption by one or more processing units of the server over a second time window subsequent to the plurality of time windows; andadjust, for the second time window, one or more parameters of the one or more processing units according to the forecast values.1984913-1418-6318.1Docket No. 102555-026179. The non-transitory computer-readable medium of claim 78, wherein the first neural network comprises a convolutional neural network (CNN), the second neural network comprises a first linear neural network, and the third neural network comprises a second linear neural network.

80. The non-transitory computer-readable medium of claim 78, wherein the metrics over the plurality of time windows are stored in a memory, and the instructions, when executed by the one or more processors, further cause the one or more processors to:compute, from the electrical waveform data, second metrics over the second time window;store the second metrics of the second time window in the memory; anddiscard a portion of the metrics associated with an earliest one of the plurality of time windows such that the second time window is a latest one of the plurality of time windows.

81. A system to share power across rack cabinets in a data center, comprising:one or more processors coupled with memory, to:monitor, via one or more measurement circuits, a plurality of electric waveform data measured for the rack cabinets in the data center;identify a first subset of power provisioned to a first rack cabinet and a second subset of power provisioned to a second rack cabinet, wherein the first subset of power and the second subset of power correspond to at least a portion of an amount of power provisioned to the data center and used by the rack cabinets;determine, based on first electric waveform data of the plurality of electric waveform data measured for the first rack cabinet and the first subset of power, a first amount of power available for consumption at the first rack cabinet;determine, based on second electric waveform data of the plurality of electric waveform data measured for the second rack cabinet and the second subset of power, a second amount of power available for consumption at the second rack cabinet; and adjust one or more parameters associated with the first rack cabinet or the second rack cabinet based on a difference between the first amount of power that is available and the second amount of power that is available being greater than or equal to a threshold.1994913-1418-6318.1Docket No. 102555-026182. The system of claim 81, wherein the first rack cabinet comprises a first plurality of shelves, each of the first plurality of shelves comprising a respective first one or more processing units, and wherein the second rack cabinet comprises a second plurality of shelves, each of the second plurality of shelves comprising a respective second one or more processing units.

83. The system of claim 81, the one or more processors further:obtain, via the one or more measurement circuits disposed at a power distribution unit electrically coupled to the rack cabinets, voltage waveform data or current waveform data associated with electric power delivered to a plurality of shelves of the rack cabinets.

84. The system of claim 81, the one or more processors further:obtain, via the one or more measurement circuits disposed at power supply units electrically coupled to the rack cabinets, voltage waveform data or current waveform data associated with electric power delivered to a plurality of shelves of the rack cabinets.

85. The system of claim 81, wherein the one or more processors further:determine, based on the first electric waveform data, a third amount of power consumed for a first processing operation of the first one or more processing units;determine, based on the second electric waveform data, a fourth amount of power consumed for a second processing operation of the second one or more processing units; determine the first amount of power available for consumption at the first rack cabinet based on a difference between the first subset of power and the third amount of power; and determine the second amount of power available for consumption at the second rack cabinet based on a difference between the second subset of power and the fourth amount of power.

86. The system of claim 81, wherein the one or more processors further:generate a signal comprising an indication to update a power limit of one or more processing units of the first rack or the second rack based on the difference between the first amount of power that is available and the second amount of power that is available being greater than or equal to the threshold.

87. The system of claim 81, wherein the one or more processors further:2004913-1418-6318.1Docket No. 102555-0261based on (i) the difference being greater than or equal to the threshold, and (ii) the first amount of power available for consumption being greater than the second amount of power available for consumption, increase at least one of a frequency of one or more processing units of the first rack cabinet, a voltage of the one or more processing units, a power limit of the one or more processing units, a frequency of a memory in electrical communication with the one or more processing units, or a capacity of the memory.

88. The system of claim 81, wherein the one or more processors further:based on (i) the difference being greater than or equal to the threshold, and (ii) the first amount of power available for consumption being greater than the second amount of power available for consumption, decrease at least one of a frequency of one or more processing units of the second rack cabinet, a voltage of the one or more processing units, a power limit of the one or more processing units, a frequency of a memory in electrical communication with the one or more processing units, or a capacity of the memory.

89. The system of claim 81, wherein the one or more processors further:based on (i) the difference being greater than or equal to the threshold, and (ii) the first amount of power available for consumption being less than the second amount of power available for consumption, decrease at least one of a frequency of one or more processing units of the first rack cabinet, a voltage of the one or more processing units, a power limit of the one or more processing units, a frequency of a memory in electrical communication with the one or more processing units, or a capacity of the memory.

90. The system of claim 81, wherein the one or more processors further:based on (i) the difference being greater than or equal to the threshold, and (ii) the first amount of power available for consumption being less than the second amount of power available for consumption, increase at least one of a frequency of one or more processing units of the second rack cabinet, a voltage of the one or more processing units, a power limit of the one or more processing units, a frequency of a memory in electrical communication with the one or more processing units, or a capacity of the memory.

91. The system of claim 81, wherein the one or more processors further:reallocate, based on (i) the difference being greater than or equal to the threshold, and (ii) the first amount of power available for consumption being greater than the second amount 2014913-1418-6318.1Docket No. 102555-0261of power available for consumption, compute resources from the second rack cabinet to the first rack cabinet for a processing operation by one or more processing units of the first rack cabinet.

92. The system of claim 81, wherein the one or more processors further:reallocate, based on (i) the difference being greater than or equal to the threshold, and (ii) the second amount of power available for consumption being greater than the first amount of power available for consumption, compute resources from the first rack cabinet to the second rack cabinet for a processing operation by one or more processing units of the second rack cabinet.

93. The system of claim 81, wherein the one or more processors further:adjust the one or more parameters associated with the first rack cabinet or the second rack cabinet based on (i) the difference between the first amount of power that is available and the second amount of power that is available being greater than or equal to the threshold, and (ii) a third amount of power consumed for a first processing operation of the first rack cabinet and a fourth amount of power consumed for a second processing operation of the second rack cabinet being greater than or equal to a second threshold.

94. The system of claim 81, wherein the one or more parameters are adjusted at a first time window, and wherein the one or more processors further:detect, at a second time window after adjustment of the one or more parameters in the first time window, that the difference between the first amount of power that is available and the second amount of power that is available is below the threshold;maintain the one or more parameters during the second time window;detect, at a third time window subsequent to the second time window, that the difference between the first amount of power that is available and the second amount of power that is available is greater than or equal to the threshold; andadjust, at the third time window, the one or more parameters associated with the first rack cabinet or the second rack cabinet based on the difference being greater than or equal to the threshold.2024913-1418-6318.1Docket No. 102555-026195. The system of claim 81, wherein the first electric waveform data and the second electric waveform data are obtained at a first time window, and wherein the one or more processors further:predict, for a second time window based on at least the first electric waveform data and the second electric waveform data, a third amount of power consumed for a first processing operation by first one or more processing units of the first rack cabinet and a fourth amount of power consumed for a second processing operation by second one or more processing units of the second rack cabinet;determine, based on the third amount of power and the first subset of power, the first amount of power available for consumption at the first rack cabinet for the second time window;determine, based on the fourth amount of power and the second subset of power, the second amount of power available for consumption at the second rack cabinet for the second time window; andadjust, for the second time window, the one or more parameters associated with the first rack cabinet or the second rack cabinet based on the difference between the first amount of power that is available and the second amount of power that is available being greater than or equal to the threshold.

96. The system of claim 81, wherein the one or more processors further:identify a third subset of power provisioned to a third rack cabinet of the amount of power, the third subset of power corresponding to another portion of the amount of power provisioned to the data center and used by the rack cabinets;determine, based on third electric waveform data of the plurality of electric waveform data measured for the third rack cabinet and the third subset of power, a third amount of power available for consumption at the third rack cabinet;determine a second difference between the third amount of power that is available and each of the first amount of power that is available and the second amount of power that is available; andadjust the one or more parameters associated with the third cabinet and at least one the first rack cabinet or the second rack cabinet based on the second difference being greater than or equal to the threshold.

97. The system of claim 81, wherein the one or more processors further:2034913-1418-6318.1Docket No. 102555-0261maintain the one or more parameters after adjustment for a predefined time window or until satisfying a criterion; andrevert the one or more parameters to one or more values before adjustment subsequent to the predefined time window or satisfying the criterion.

98. The system of claim 81, wherein the one or more processors further:adjust a second one or more parameters associated with an energy storage device based on the first amount of power that is available and the second amount of power that is available being greater than or equal to a second threshold or less than or equal to a third threshold, wherein the energy storage device is to:(i) supply a third amount of power for consumption at the first rack cabinet or the second rack cabinet based on the first amount of power that is available and the second amount of power that is available being less than or equal to the second threshold, or(ii) receive a fourth amount of power, that is unused at the first rack cabinet and the second rack cabinet, for storage during a time window when the first amount of power that is available and the second amount of power that is available being greater than or equal to the third threshold.

99. A method for sharing power across rack cabinets in a data center, comprising:monitoring, by one or more processors coupled with memory, via one or more measurement circuits, a plurality of electric waveform data measured for the rack cabinets in the data center;identifying, by the one or more processors, a first subset of power provisioned to a first rack cabinet and a second subset of power provisioned to a second rack cabinet, wherein the first subset of power and the second subset of power correspond to at least a portion of an amount of power provisioned to the data center and used by the rack cabinets;determining, by the one or more processors, based on first electric waveform data of the plurality of electric waveform data measured for the first rack cabinet and the first subset of power, a first amount of power available for consumption at the first rack cabinet;determining, by the one or more processors, based on second electric waveform data of the plurality of electric waveform data measured for the second rack cabinet and the second subset of power, a second amount of power available for consumption at the second rack cabinet; and2044913-1418-6318.1Docket No. 102555-0261adjusting, by the one or more processors, one or more parameters associated with the first rack cabinet or the second rack cabinet based on a difference between the first amount of power that is available and the second amount of power that is available being greater than or equal to a threshold.

100. A non-transitory computer-readable medium storing processor executable instructions that, when executed by one or more processors, cause the one or more processors to:monitor, via one or more measurement circuits, a plurality of electric waveform data measured for rack cabinets in a data center;identify a first subset of power provisioned to a first rack cabinet and a second subset of power provisioned to a second rack cabinet, wherein the first subset of power and the second subset of power correspond to at least a portion of an amount of power provisioned to the data center and used by the rack cabinets;determine, based on first electric waveform data of the plurality of electric waveform data measured for the first rack cabinet and the first subset of power, a first amount of power available for consumption at the first rack cabinet;determine, based on second electric waveform data of the plurality of electric waveform data measured for the second rack cabinet and the second subset of power, a second amount of power available for consumption at the second rack cabinet; and adjust one or more parameters associated with the first rack cabinet or the second rack cabinet based on a difference between the first amount of power that is available and the second amount of power that is available being greater than or equal to a threshold.2054913-1418-6318.1

Images