Routing Manager for ML Model Sharding and Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning serving infrastructures face challenges in efficiently managing the distribution of machine learning models across containers, particularly in handling varying resource capacities, model demands, and failures, while supporting multiple models and frameworks in a multi-tenant system.
Innovation Solution
A routing manager is introduced to manage the distribution of machine learning models by tracking container resources and model demands, using load balancing and rebalancing algorithms to ensure optimal resource utilization, handle failures, and support different model types and versions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If machine learning models are distributed across multiple containers in a multi-tenant system, then the system can support diverse model types and frameworks, but the complexity of managing resource allocation and load distribution increases
Solution Approach 1:
The patent introduces a routing manager as an intermediary component that sits between the client requests and the containerized machine learning models. This routing manager maintains configuration data mapping model identifiers to container locations and handles the complexity of resource allocation, load balancing, and failure management. By centralizing these management functions in the routing manager, the system achieves high adaptability for supporting diverse model types while keeping the complexity isolated in the routing layer rather than distributed across all system components.
2Productivity
If load balancing algorithms are used to distribute requests across containers, then resource utilization is optimized, but the system requires sophisticated tracking and management of container resources and model demands
Solution Approach 1:
The patent implements preliminary action by pre-configuring and maintaining a routing table that maps model identifiers to their deployed container locations before requests arrive. The routing manager proactively tracks container resource capacities and model demands, updating the configuration data in advance. This allows the system to perform simple lookups in the routing table during request handling rather than performing complex real-time analysis, thereby achieving efficient resource utilization through pre-computed routing decisions while minimizing runtime management overhead.
3Reliability
If the system handles failures and rebalances models across containers, then reliability is improved, but the complexity of managing model distribution and container states increases
Solution Approach 1:
The patent implements feedback mechanisms where the routing manager continuously monitors container health status, resource capacity, and model deployment states. When failures are detected or containers are added/removed, the routing manager receives feedback about the changed state and automatically updates the routing configuration accordingly. This feedback-driven approach enables the system to maintain high reliability through automatic failure handling and rebalancing while keeping the complexity managed through event-driven updates rather than continuous complex calculations.
4Adaptability or versatility
If multiple machine learning models are deployed in separate containers, then model isolation and independence are achieved, but the overhead of managing multiple container instances increases
Solution Approach 1:
The patent applies universality by designing the routing manager to handle multiple functions: it routes requests to appropriate containers, tracks resource capacity across all containers, manages model deployment states, handles failures, and performs load balancing. By creating a single universal routing manager that performs all these functions rather than having separate management components for each function, the system achieves model isolation through containerization while minimizing the overall management overhead through a consolidated, multi-functional control point.
Data Source
AI summary
A machine learning serving infrastructure implementing a method of receiving or detecting an update of container metrics including resource usage and serviced requests per model or per container, processing the container metrics per model or per container to determine recent resource usage and serviced requests per model or per container, and rebalancing distribution of models to a plurality of containers to decrease a detected load imbalance between containers or a stressed container in the plurality of containers.