API Resource Prediction with Shared Model Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for scaling APIs with AI and ML are inefficient and inaccurate in managing RAM usage, leading to inefficiencies in handling multiple user requests and model updates.
Innovation Solution
A system and method that utilize a versioning logic mechanism and a least recently used (LRU) caching technique to optimize RAM usage by loading the most recently used models and preventing unnecessary reloading, while using a common base manager to load large files across multiple worker processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If each worker subprocess loads large model files independently, then model availability for inference is improved, but RAM usage increases and scaling efficiency deteriorates
Solution Approach 1:
The patent implements a shared model loading mechanism where a single instance of a model manager class loads and manages model files that are shared across all worker subprocesses. This eliminates redundant model loading and reduces RAM consumption while maintaining model availability through centralized management and reference counting.
2Measurement precision
If models are continuously reloaded to ensure updated versions are used, then prediction accuracy is improved, but loading time increases and system latency worsens
Solution Approach 1:
The system pre-loads model files into memory when they become available and maintains them in a shared cache. Model versions are managed in advance with versioning logic, allowing the system to serve predictions using pre-loaded models rather than loading them on-demand during request processing, thus reducing latency.
Solution Approach 2:
The model manager implements a feedback mechanism that monitors model usage patterns, tracks which models are most recently used, and uses this information to determine when and how to load new model versions. This intelligent caching strategy prevents unnecessary reloads while ensuring updated models are available when needed.
3Device complexity
If all model files are loaded into memory simultaneously, then model version management is simplified, but RAM consumption increases and resource efficiency deteriorates
Solution Approach 1:
The system dynamically manages model loading based on actual usage patterns rather than statically loading all models. The model manager uses reference counting and usage tracking to determine which models to keep in memory and which can be unloaded, allowing RAM consumption to scale with actual demand while maintaining simple version management through centralized control.
Data Source
AI summary
The present disclosure provides a system and method for utilizing an application programming interface (API) for handling multiple requests/models from various users. The API uses a machine learning (ML) model for processing the various requests and producing a plurality of trained models. The system is resilient to a race condition and incorporates an optimized random access memory (RAM) usage. Further, the system efficiently manages the limited available resources by loading and unloading the machine learning models based on their usage, thus maximizing the throughput of the API.