Multi-tenant ML Serving Infrastructure with Dynamic DAG Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning infrastructures lack support for multi-tenancy and dynamic operations, requiring static pipeline definitions and containerization of each operation, limiting their scalability and flexibility in handling multiple tenants and machine learning models.
Innovation Solution
A multi-tenant machine learning serving infrastructure that supports dynamic, scalable operations using a directed acyclic graph (DAG) structure, enabling multi-model scoring and allowing developers to define machine learning applications through a domain-specific language (DSL) or human-readable data serialization languages like YAML or JSON, with automated container management and orchestration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing machine learning infrastructures use static pipeline definitions and containerization of each operation, then system stability and isolation are improved, but scalability and flexibility deteriorate when handling multiple tenants and machine learning models
Solution Approach 1:
The patent segments the machine learning infrastructure into isolated tenant environments where each tenant's models and data are separated. The system divides the processing pipeline into discrete operations that can be independently managed and executed, allowing multiple tenants to coexist without interference while maintaining system stability.
Solution Approach 2:
The patent creates a universal multi-tenant machine learning infrastructure that can handle multiple types of machine learning models and operations through a single system. The framework provides multi-functional capabilities to support different tenants, models, and operations without requiring separate infrastructure for each, thereby improving flexibility while maintaining reliability through standardized interfaces.
2Reliability
If existing machine learning infrastructures require containerization of each operation, then operation isolation and security are improved, but device complexity and deployment difficulty increase
Solution Approach 1:
The patent merges multiple containerized operations into a unified execution framework that manages isolation and security at the framework level rather than requiring separate containers for each operation. This combining approach reduces deployment complexity while maintaining operation isolation through centralized security policies and managed execution environments.
Solution Approach 2:
The patent introduces an intermediary execution framework that sits between the machine learning operations and the underlying container infrastructure. This mediator layer provides standardized interfaces and automated management, reducing the complexity of direct container orchestration while maintaining the isolation benefits through enforced security boundaries.
3Ease of manufacture
If existing machine learning infrastructures use static pipeline definitions, then system simplicity and ease of implementation are improved, but adaptability and dynamic operations capability deteriorate
Solution Approach 1:
The patent transforms the static pipeline definition into a dynamic execution model where the framework can adaptively route operations, load different models, and adjust processing flows based on tenant requirements and operational context. The system maintains implementation simplicity through standardized entry points while enabling dynamic behavior through the flexible execution engine.
Solution Approach 2:
The patent enables dynamic operations by allowing configuration parameters to be changed at runtime without redefining the entire pipeline. The framework accepts parameterized operations that can be adjusted dynamically, allowing the same implementation structure to support varying operational requirements through parameter modification rather than structural changes.
4Device complexity
If existing machine learning infrastructures lack multi-tenancy support, then system simplicity is improved, but scalability and ability to handle multiple tenants deteriorate
Solution Approach 1:
The patent segments the system into isolated tenant contexts where each tenant's data, models, and operations are separated into distinct execution environments. This segmentation enables scalability to multiple tenants while maintaining relative simplicity through standardized isolation mechanisms and centralized management of tenant resources.
Solution Approach 2:
The patent creates a universal multi-tenant framework that provides scalability to handle multiple tenants through a single unified system. The infrastructure delivers multi-functional capabilities to support different tenants with varying requirements while maintaining simplicity through standardized interfaces and centralized resource management.
Data Source
AI summary
Methods and systems for multi-model scoring in a multi-tenant system are presented. A request for a machine learning application is received from a tenant application. A tenant identifier that identifies one of the multiple tenants is determined. Based on the tenant identifier and a type of the machine learning application, a first and a second machine learning models are determined. The first machine learning model was generated based on a first training data set associated with the tenant identifier. The second machine learning model that was generated based on a second training data set associated with the tenant identifier. A flow of operations that includes running the first and second machine learning models with data related to the request is executed to obtain a scoring result. The scoring result is returned to the tenant application in response to the request.