Inferencing Endpoint Discovery in Distributed Computing Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computing systems, discovering and accessing suitable inferencing endpoints for predictions is difficult due to the large number of endpoints deployed, leading to wasteful resource usage and inefficient resource consumption.
Innovation Solution
A model directory service that monitors and tracks deployed inferencing endpoints, allowing users to search based on criteria such as execution latency, prediction accuracy, and location, and instantiates new endpoints when necessary to meet user demands.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a new inferencing endpoint is instantiated upon receiving each request, then prediction availability is ensured, but computing resource waste increases
Solution Approach 1:
The system performs preliminary actions by pre-deploying multiple inferencing endpoints with different characteristics (accuracy, latency, cost) before requests arrive. The model directory service maintains a registry of these pre-configured endpoints, allowing the system to respond to requests by selecting from existing endpoints rather than instantiating new ones, thus ensuring prediction availability while avoiding resource waste.
Solution Approach 2:
The patent applies parameter changes by allowing users to specify different selection criteria (parameters) such as prediction accuracy, execution latency, and cost when querying the model directory service. The system then selects inferencing endpoints based on these variable parameters, enabling flexible resource allocation that balances prediction availability with computing resource efficiency based on actual request requirements.
2Adaptability or versatility
If thousands or millions of inferencing endpoints are deployed, then prediction coverage is improved, but endpoint discovery difficulty increases
Solution Approach 1:
The model directory service acts as an intermediary between users and the large number of deployed inferencing endpoints. It maintains a centralized registry that catalogs endpoint characteristics (accuracy, latency, location, cost) and provides search functionality, transforming the difficult task of manually discovering endpoints into a simple query process, thus enabling prediction coverage across millions of endpoints while reducing discovery difficulty.
Solution Approach 2:
The system segments the large set of inferencing endpoints into organized categories within the model directory service registry. Endpoints are structured with standardized metadata fields (accuracy, latency, location, cost) that allow users to search and filter through them systematically. This segmentation transforms the overwhelming task of discovering individual endpoints into an organized search process, improving prediction coverage while making endpoint discovery manageable.
3Loss of energy
If existing inferencing endpoints are reused, then resource consumption is reduced, but endpoint selection complexity increases
Solution Approach 1:
The model directory service provides self-service functionality by automatically maintaining the registry of inferencing endpoints and their characteristics. Users simply submit search queries with desired criteria (accuracy, latency, cost), and the system automatically filters and returns appropriate endpoints. This eliminates the need for users to manually evaluate or select from thousands of endpoints, reducing resource consumption through reuse while simplifying the selection process through automated directory services.
Data Source
AI summary
Techniques for machine learning inferencing endpoint discovery in a distributed computing system are discloses herein. In one example, a method includes searching a database containing machine learning endpoint records having data representing values of execution latency or prediction accuracy corresponding inferencing endpoints deployed in the distributed computing system. The method also includes generating a list of inferencing endpoints matching the individual target values and determining whether a count of the inferencing endpoints in the generated list exceeds a preset threshold. In response to determining that the identified count does not exceed the preset threshold, the method includes instantiating one or more additional inferencing endpoints in the distributed computing system based on the individual target values in the received query.