ML Versioning Service for Multi-Tenant Model Routing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current machine learning serving infrastructures face challenges in managing multiple versions of ML frameworks and models across different tenants, requiring efficient versioning and routing mechanisms to support real-time prediction services and multi-tenancy, while ensuring resource optimization and load balancing.

Innovation Solution

The implementation of a machine learning versioning service (MVS) within the ML serving infrastructure, which enables the deployment and management of multiple ML framework and model versions, utilizing a routing service for load balancing and resource management, and a service discovery system for tracking and routing requests to appropriate containers, supporting multi-tenancy and real-time prediction services.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If multiple ML framework and model versions are deployed across different tenants, then model versioning capability and multi-tenancy support are improved, but system complexity and resource management difficulty increase

Engineering Contradiction:

Improvemodel versioning capabilityVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent segments the ML serving system into isolated containers, each hosting specific ML framework and model versions. This segmentation allows multiple versions to coexist without interference, resolving the contradiction by organizing complexity into manageable, isolated units that can be independently managed and scaled.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a service discovery system as an intermediary layer between clients and ML model containers. This mediator manages version routing, load balancing, and container discovery, thereby handling the system complexity centrally while maintaining simplicity for individual tenants and enabling robust multi-version support.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If version routing and load balancing mechanisms are implemented, then request routing accuracy and resource utilization are improved, but system complexity and operational overhead increase

Engineering Contradiction:

Improveresource utilizationVSAvoidoperational overhead

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent implements self-service mechanisms where the service discovery system autonomously performs health checks, capacity assessments, and routing decisions without manual intervention. This automation improves resource utilization through dynamic load balancing while reducing operational overhead by eliminating the need for manual system management.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system incorporates feedback loops where the service discovery system continuously monitors container health, performance metrics, and resource availability. This feedback enables dynamic routing decisions that optimize resource utilization while automatically adapting to changing system conditions, thereby improving productivity without proportionally increasing operational overhead.

Inventive Principle:

Principle #23Feedback

3Reliability

If real-time prediction services are supported with multiple model versions, then service availability and model selection flexibility are improved, but system complexity and deployment difficulty increase

Engineering Contradiction:

Improveservice availabilityVSAvoiddeployment difficulty

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent employs preliminary action by pre-configuring multiple ML model containers with different versions and frameworks before deployment. The service discovery system pre-establishes routing rules and health check mechanisms, enabling real-time prediction services to be activated with minimal deployment effort while maintaining high service availability through pre-prepared redundancy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The service discovery system is designed as a universal platform that can discover, manage, and route to multiple types of ML containers (different frameworks, versions, and architectures) through a unified interface. This multi-functionality improves service availability by supporting diverse model versions while reducing deployment difficulty through a single, standardized management approach.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11614932B2Method and system for machine learning framework and model versioning in a machine learning serving infrastructure

Publication Date: 2023.03.28 SALESFORCE INC

AI summary

Machine learning version management method for a prediction service includes receiving a prediction request, determining application metadata for the request that defines routing logic and a machine learning framework version, determining model metadata for the request that defines at least one model and at least one model version, forwarding the prediction request to the at least one model with the at least one model version, and returning a prediction from the at least one model to a requestor.