AI Job Batching Across Multiple Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI job batching solutions are limited to processing AI jobs from a single client using a single physical resource, which reduces system efficiency when multiple clients process the same AI model independently across different accelerators.
Innovation Solution
A system and method for dynamically batching AI jobs from multiple clients across multiple accelerators, where AI jobs are selected based on batching parameters and prohibited jobs, aggregated into batches, and processed by a compute engine dedicated to executing the batch.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If AI jobs from multiple clients are processed independently by different accelerators, then each client can be served, but system efficiency is reduced due to inability to batch across clients
Solution Approach 1:
A batching processor is introduced as an intermediary component between multiple clients and multiple accelerators. This mediator aggregates AI jobs from multiple clients and distributes them to accelerators in batches, enabling cross-client batching without requiring changes to the accelerators themselves. The batching processor manages the complexity of coordinating multiple clients and accelerators, while presenting a simple interface to both sides.
Solution Approach 2:
The batching processor is designed to handle multiple types of AI jobs from multiple different clients simultaneously, making it a universal component that can serve diverse workloads. It can aggregate jobs with different characteristics (batch sizes, priorities, models) and route them appropriately, providing multi-functional capability that improves overall system productivity without requiring separate processing paths for each client type.
2Productivity
If batching is performed to improve efficiency and reduce energy consumption, then processing throughput increases, but latency may increase due to aggregation time
Solution Approach 1:
The batching system dynamically adjusts batch formation parameters based on current workload conditions, job priorities, and accelerator availability. Instead of using fixed batch sizes or aggregation times, the system can adaptively determine optimal batch parameters in real-time, balancing the trade-off between throughput improvement and latency introduction. High-priority jobs can be processed with smaller batches or lower aggregation thresholds to minimize their latency.
Solution Approach 2:
The system performs preliminary actions by pre-aggregating jobs into batches before they are processed by accelerators. The batching processor collects and prepares batches in advance, organizing multiple client jobs into optimized groupings that maximize accelerator utilization. This preliminary aggregation allows the actual processing phase to proceed more efficiently, improving overall throughput while managing latency through controlled aggregation windows.
Data Source
AI summary
A sequencer and method for batching execution of artificial intelligence (AI) jobs comprising receiving, by an AI server, a plurality of AI jobs from a plurality of clients connected to an AI appliance over a network; dynamically selecting from the plurality of AI jobs a set of AI jobs to be batched, wherein the selection is based on at least one batching parameter and a list of AI jobs prohibited from batching; aggregating each of the selected AI jobs into a created batch; continuing aggregating newly received AI jobs related to the selected AI jobs in the created batch until at least one service parameter is met; and sending the batch of AI jobs to a compute engine dedicated to executing the batch.