Inference Model Manager Dynamic Deployment Scaling

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Static deployments of inference models in distributed environments fail to adapt to changing inference frequency requirements of downstream consumers, leading to inefficient use of computing resources and potential inability to meet demand.

Innovation Solution

An inference model manager dynamically modifies the deployment of inference models by obtaining an execution plan that adjusts the number of instances and complexity of models based on predicted demand, using historical and operational data to optimize resource allocation across data processing systems.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If static deployment of inference models is used, then device complexity is reduced and ease of operation is improved, but adaptability to changing inference frequency requirements deteriorates

Engineering Contradiction:

Improveadaptability to changing inference frequency requirementsVSAvoiddeployment management complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements dynamic deployment by continuously monitoring inference frequency requirements and automatically adjusting the number of model instances deployed. The system transitions from static configuration to dynamic adaptation where deployment parameters change based on real-time demand signals from downstream consumers, resolving the contradiction between adaptability and complexity through automated control

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system establishes a feedback loop where inference frequency requirements from downstream consumers are continuously monitored and fed back to the deployment management system. This feedback mechanism enables automatic adjustment of model instance counts and complexity levels, allowing the system to adapt to changing requirements without manual intervention while maintaining manageable complexity through rule-based or ML-driven decision logic

Inventive Principle:

Principle #23Feedback

2Productivity

If more instances of inference models are deployed to meet higher frequency requirements, then productivity is improved, but use of computing resources increases

Engineering Contradiction:

Improveinference generation frequencyVSAvoidcomputing resource usage

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The system dynamically adjusts the number of model instances deployed based on actual inference frequency requirements. When demand increases, additional instances are provisioned to maintain productivity; when demand decreases, instances are deprovisioned to reduce resource consumption. This dynamic scaling resolves the contradiction by aligning resource usage with actual productivity needs

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system changes deployment parameters (number of instances, model complexity level) in response to varying inference frequency requirements. By adjusting these parameters dynamically rather than maintaining fixed high-capacity deployment, the system achieves high productivity only when necessary, thereby optimizing the balance between inference generation frequency and computing resource usage

Inventive Principle:

Principle #35Parameter changes

3Use of energy by moving object

If simpler inference models are used, then computing resource usage is reduced, but the ability to meet complex inference frequency requirements deteriorates

Engineering Contradiction:

Improvecomputing resource usageVSAvoidability to meet inference frequency requirements

Core Design Contradiction:

Use of energy by moving objectVSReliability

Solution Approach 1:

The system segments the inference model deployment into multiple instances of simpler models rather than relying on a single complex model. By parallelizing multiple simpler instances, the system achieves both reduced computing resource usage per instance and the ability to meet high frequency requirements through aggregate capacity, resolving the contradiction between simplicity and reliability

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system merges multiple simpler inference model instances to collectively meet complex inference frequency requirements. By combining the output capacity of multiple lightweight models, the system achieves reliability and high-frequency performance without the resource overhead of deploying single complex models, thus resolving the contradiction between computing resource usage and ability to meet requirements

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS20240177024A1System and method for managing inference models based on inference generation frequencies

Publication Date: 2024.05.30 DELL PROD LP

AI summary

Methods and systems for managing execution of an inference model hosted by data processing systems are disclosed. To manage execution of the inference model, a system may include an inference model manager and any number of data processing systems. The inference model manager may identify an inference frequency capability of the inference model hosted by the data processing systems and may determine whether the inference frequency capability of the inference model meets an inference frequency requirement of a downstream consumer during a future period of time. If the inference frequency capability does not meet the inference frequency requirement of the downstream consumer, the inference model manager may modify a deployment of the first inference model to meet the inference frequency requirement of the downstream consumer.