Router Instance Configuration Objects for ML Onboarding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Onboarding new machine learning applications in multi-tenant on-demand model serving infrastructures requires manual code changes and significant manual efforts to configure the router component, which does not scale well.
Innovation Solution
The use of configuration objects to instantiate router instances for machine learning applications, eliminating the need for manual code changes by providing relevant information for configuring the serving infrastructure, such as the name of the application and the cluster of scoring containers, allowing the router component to route incoming scoring requests appropriately.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If manual code changes are used to configure the router component for new machine learning applications, then the router can be configured to route scoring requests, but the onboarding process requires significant manual effort and does not scale well
Solution Approach 1:
The patent uses configuration objects as templates that define router instance settings. Instead of manually coding each router configuration, the system copies and instantiates router instances from these configuration objects, enabling automated onboarding of new machine learning applications without manual code changes.
Solution Approach 2:
The system enables self-service onboarding by allowing new machine learning applications to be automatically configured through configuration objects. The router component can automatically instantiate router instances based on configuration objects without requiring manual intervention, making the system self-configuring and scalable.
2Adaptability or versatility
If manual code changes are required to add new machine learning applications, then the router can be customized for each application, but the process becomes complex and time-consuming
Solution Approach 1:
The patent implements preliminary action by pre-defining router configuration templates in configuration objects before new machine learning applications are deployed. These configuration objects contain all necessary routing parameters and settings, allowing the router to be quickly instantiated and configured without time-consuming manual code changes when new applications are added.
Solution Approach 2:
The system uses configuration objects that contain parameters defining router instance behavior. By changing parameters in configuration objects rather than modifying code, the router can be adapted to different machine learning applications. This parameter-based configuration enables flexibility while reducing the time required to onboard new applications.
Data Source
AI summary
A method by a router component in a multi-tenant on-demand serving infrastructure to route scoring requests to scoring containers. The method includes receiving a scoring request, determining a machine learning application associated with the scoring request, determining whether a router instance for the machine learning application exists, and responsive to a determination that a router instance for the machine learning application does not exist, obtaining a configuration object for the machine learning application and instantiating the router instance for the machine learning application based on the configuration object for the machine learning application. The method further includes invoking the router instance for the machine learning application to route the scoring request associated with the machine learning application to a scoring container that provides scoring functionality for the machine learning application.