Service Discovery via Metadata Collection for ML Inference Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing service discovery mechanisms in machine learning infrastructure require registration processes for nodes and clusters, which can lead to increased load times and complexity, especially in multi-tenant systems with varying resource usage and model requirements.
Innovation Solution
Implementing a node-based or cluster-based service discovery process that does not require registration, where nodes and clusters contain metadata describing their services, and a service discovery and configuration system collects and monitors this metadata to enable efficient routing of requests without explicit registration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a registration process is implemented for service discovery, then service routing can be established, but load times increase and system complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-generating service endpoint information from service metadata before runtime registration is needed. The service registry pre-processes and stores endpoint information in advance, so when a service instance comes online, the routing information is already available, eliminating the registration wait time and reducing load times while maintaining reliable service routing.
Solution Approach 2:
The patent applies beforehand cushioning by pre-computing and caching service endpoint resolutions. The system anticipates future service discovery needs by pre-processing metadata and storing resolved endpoint information, cushioning against the potential delay of real-time registration and lookup operations, thus reducing load times without compromising routing reliability.
2Reliability
If a registration process is implemented for service discovery, then service routing can be established, but system complexity increases
Solution Approach 1:
The patent applies taking out by extracting the service endpoint resolution process from the runtime registration flow. The system separates metadata processing and endpoint generation into a pre-computation phase that occurs independently of service instance registration, removing the complexity of real-time resolution from the critical service discovery path while maintaining reliable routing.
Solution Approach 2:
The patent applies preliminary action by pre-computing service endpoint information from metadata before it is needed at runtime. This shifts the computational complexity to a preliminary setup phase, simplifying the operational service discovery process while ensuring reliable service routing through pre-resolved endpoints.
3Loss of information
If nodes and clusters require explicit registration, then service information can be recorded, but integration of ML models with different requirements becomes more difficult
Solution Approach 1:
The patent applies universality by creating a unified service metadata model that can represent diverse ML model services with different requirements through a common interface. The metadata schema is designed to be universal, accommodating various ML model types, frameworks, and configurations while maintaining a consistent service discovery and routing mechanism, thus enabling easy integration of heterogeneous ML models without explicit registration variations.
Solution Approach 2:
The patent applies self-service by enabling services to automatically publish their metadata and endpoint information without manual registration. ML models with different requirements self-describe themselves through standardized metadata fields, and the system automatically processes this information for service discovery and routing, eliminating the need for complex registration processes while preserving complete service information.
Data Source
AI summary
A method performs service discovery in a machine learning service. The method includes detecting initialization of at least one service container, identifying label information in the at least one service container, collecting the label information for the initializing at least one service container, and storing the label information in a routing information storage to enable routing of requests to the at least one service container.