Model Manager Architecture for Shared-Memory ML Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deploying large machine learning models across different environments presents challenges related to memory allocation and model performance, particularly on devices with limited computational resources.
Innovation Solution
A model manager system that concurrently manages multiple machine learning models with shared memory allocations, schedules runtimes across devices, and employs a flexible plug-in architecture to accommodate diverse models, including a model catalog for resource management and dynamic loading of adapters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If large machine learning models are deployed across different environments, then model performance and versatility are improved, but memory allocation and computational resource requirements worsen
Solution Approach 1:
The patent segments machine learning models into base models and separate adapter modules. Each adapter contains task-specific parameters that can be independently loaded and unloaded. This segmentation allows the system to deploy versatile models by combining a shared base model with different adapters for different tasks, reducing overall memory requirements compared to deploying complete separate models for each task.
Solution Approach 2:
The patent implements a universal base model that can serve multiple functions by loading different adapters. The base model architecture remains the same across different deployment scenarios, while adapters provide task-specific functionality. This multi-functionality approach allows a single model deployment to handle diverse tasks, improving versatility without proportionally increasing memory allocation.
2Productivity
If multiple machine learning models are executed concurrently on devices with limited computational resources, then task diversity and productivity are improved, but resource consumption and device complexity worsen
Solution Approach 1:
The patent merges multiple model execution contexts by sharing a common base model instance across different tasks. Instead of loading complete separate models for each task, the system combines a single base model with multiple lightweight adapters, allowing concurrent execution of diverse tasks while reducing overall resource consumption and simplifying device requirements.
Solution Approach 2:
The patent implements dynamic loading and unloading of adapters based on task requirements. The system can dynamically allocate memory and computational resources by loading only the necessary adapters for current tasks and unloading them when no longer needed. This dynamic resource management enables concurrent model execution while adapting resource allocation to actual workload demands, reducing average device complexity.
3Quantity of substance
If adapters are dynamically loaded for different tasks, then memory efficiency and resource utilization are improved, but system complexity and loading time worsen
Solution Approach 1:
The patent implements preliminary loading of adapters into a cached state when they are anticipated to be needed soon. The system can pre-load adapters based on task queues or usage patterns, so that when a task requires a specific adapter, it is already available in memory or quickly accessible from cache. This preliminary action reduces the actual loading time during task execution while maintaining memory efficiency through selective caching.
Data Source
AI summary
The subject technology provides for a model management and deployment system for machine learning models. A system includes a model manager configured to schedule execution of one or more machine learning models on one or more electronic devices or servers. The system also includes a model catalog configured to store information associated with the one or more machine learning models. The model manager may access the model catalog to determine scheduling priorities based on the stored information.