Tiered Storage for ML Models Using ISTIO

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current machine learning model serving systems face challenges in efficiently storing and accessing models due to the use of a single type of memory, which fails to prioritize models based on their service level agreements (SLAs) and is cost-prohibitive, especially when scaling, as all models are treated equally regardless of their importance or usage frequency.

Innovation Solution

Implementing tiered storage systems that utilize different types of storage devices such as Hard Disk Drives, Solid State Drives, and Persistent Memory, where models are stored based on their SLAs, allowing for dynamic reassignment across tiers based on network conditions and SLA requirements, and leveraging ISTIO for network monitoring and management.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a single type of memory is used to store all models, then all models can be accessed uniformly, but the system becomes cost-prohibitive and cannot prioritize models based on service level agreements

Engineering Contradiction:

ImproveSLA complianceVSAvoidstorage system complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The storage system is segmented into multiple tiers with different performance characteristics (high-performance storage, standard storage, and archival storage). Models are divided and placed in appropriate tiers based on their SLA requirements, allowing differentiated service levels without requiring a single complex high-performance storage system for all models.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different storage tiers provide different quality levels of access performance. High-performance storage provides fast access for critical models with strict SLAs, while standard and archival storages provide adequate access for less critical models. Each model receives the appropriate quality of service localized to its specific requirements.

Inventive Principle:

Principle #3Local quality

2Speed

If all models are stored in high-performance memory to meet SLA requirements, then access speed is improved, but the cost becomes prohibitive

Engineering Contradiction:

Improvemodel access speedVSAvoidstorage cost

Core Design Contradiction:

SpeedVSQuantity of substance

Solution Approach 1:

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system changes the performance parameter of storage dynamically by moving models between tiers based on their accessed patterns and SLA requirements. Frequently accessed models with strict SLAs are placed in high-performance storage, while less frequently accessed models are moved to lower-cost storage tiers, optimizing the balance between access speed and cost.

Inventive Principle:

Principle #35Parameter changes

3Adaptability or versatility

If models are statically assigned to storage tiers, then system simplicity is maintained, but the system cannot adapt to changing network conditions and SLA requirements

Engineering Contradiction:

Improvestorage allocation flexibilityVSAvoidstorage management complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The storage allocation system is made dynamic through automated model migration capabilities. Models can be moved between storage tiers based on changing conditions such as updated SLA requirements, changes in access patterns, or network conditions. This dynamic adaptation allows the system to respond to changing requirements without manual intervention.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system implements feedback mechanisms that monitor model access patterns, network conditions, and SLA compliance. Based on this feedback, the system automatically adjusts model placement in storage tiers, migrating models as needed to maintain optimal performance and compliance while adapting to changing conditions.

Inventive Principle:

Principle #23Feedback

4Productivity

If models are frequently migrated between storage tiers to optimize performance, then SLA compliance is improved, but the complexity of storage management increases

Engineering Contradiction:

Improvemodel serving efficiencyVSAvoidstorage management complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The storage management system operates autonomously using automated policies and algorithms that evaluate SLA requirements and access patterns to determine optimal model placement. The system self-manages the migration of models between tiers without requiring complex manual management, reducing the operational complexity while maintaining high productivity.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS10972364B2Using tiered storage and ISTIO to satisfy SLA in model serving and updates

Publication Date: 2021.04.06 CISCO TECHNOLOGY INC

AI summary

Systems, methods, and computer-readable storage media are provided for storing machine learned models in a tiered storage. The model serving network evaluates where the models should be stored based on the model corresponding service level agreement. The model is generally stored at the lowest tiered storage device that is still capable of satisfying the model's service level agreement. In this way, the model serving network aims to store data that achieves the cheapest cost.