AI Inference Load Balancing Using Server Capability Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional load balancing methods become complex and difficult in data centers with servers supporting various AI models and large-scale hardware, leading to inefficient task distribution and increased management complexity.
Innovation Solution
A load balancing method and system that utilizes a load balancing table to consider AI model and hardware information of each server, generating and updating load balancing information to efficiently distribute inference tasks based on a preset algorithm.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional load balancing methods are used in data centers with various AI models and hardware, then basic task distribution can be achieved, but the load balancing task becomes complex and difficult to manage
Solution Approach 1:
The patent segments the load balancing information into distinct components: AI model information, hardware information, connection information, and current load status. Each server publishes its capabilities as separate attribute sets, and the load balancing device queries and matches these segmented attributes against incoming inference requests. This segmentation transforms the complex holistic matching problem into manageable attribute-by-attribute comparison tasks.
Solution Approach 2:
The patent introduces an intermediary load balancing device that acts as a mediator between inference requests and servers. This intermediary maintains a registry of server capabilities, receives inference requests with required AI model and hardware specifications, and performs the complex matching logic centrally. The intermediary abstracts the complexity from individual servers and provides a unified interface for request routing.
2Productivity
If load balancing considers AI model and hardware information of each server, then task distribution efficiency is improved, but information processing and management complexity increases
Solution Approach 1:
The patent implements preliminary action by having each server publish its AI model support list, hardware configuration, and connection information to the load balancing device before receiving inference requests. The load balancing device stores this capability information in advance and uses it to pre-filter suitable servers for incoming requests. This preliminary registration and capability advertisement eliminates the need for complex real-time information exchange during task distribution.
Solution Approach 2:
The patent changes the information representation parameters by using structured data formats with standardized fields for AI model versions, hardware types, and connection parameters. The load balancing device compares incoming request parameters against server capability parameters using attribute matching. This parameterized approach transforms unstructured information management into systematic parameter comparison, reducing processing overhead.
3Measurement precision
If a load balancing table is generated and maintained with detailed server information, then accurate task routing is achieved, but the system requires continuous updates and monitoring increasing operational complexity
Solution Approach 1:
The patent implements feedback mechanisms where servers continuously publish updates to their AI model support lists, hardware status, and connection information to the load balancing device. The load balancing device maintains this information in a dynamically updated table and uses it for real-time routing decisions. When servers add or remove AI models or change hardware status, they automatically notify the load balancing device, which updates its internal table accordingly.
Solution Approach 2:
The patent enables self-service by having servers automatically publish their own capability information and status updates to the load balancing device without manual intervention. Each server autonomously monitors its own AI model availability, hardware status, and connection state, then proactively reports changes to the central load balancing device. This self-service approach eliminates the need for manual load balancing table maintenance while ensuring accurate, real-time information.
Data Source
AI summary
A load balancing method in an Artificial Intelligence (AI) service providing system, comprising: obtaining load balancing information of a plurality of servers, generating a load balancing table based on the load balancing information of the plurality of servers, obtaining an inference task request message for an AI service from a user device, deriving at least one target server among the plurality of servers based on the inference task message for the AI service and the load balancing table, and performing load balancing for an inference task of the AI service on the derived target server based on a preset load balancing algorithm, wherein the load balancing information includes connection information, AI model information, and supported hardware information of each server.