Adaptive Model Engine for Edge Inference Latency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Machine learning models deployed on edge devices face resource constraints, making it impractical to run models that require significant computational power, leading to suboptimal performance in terms of throughput, latency, and accuracy due to limited hardware capabilities compared to web services providers.

Innovation Solution

An adaptive model engine and service that dynamically adjust ML model execution by analyzing edge device and web services provider characteristics, such as FLOPS, RAM, and power usage, to generate optimized model variants or profiles that can be executed on the edge device or web services provider, allowing for real-time modifications like quantization, layer removal, or operator changes to meet performance objectives.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If ML models are deployed on edge devices, then local inference capability is improved, but performance in terms of throughput, latency, and accuracy deteriorates due to limited hardware capabilities

Engineering Contradiction:

Improvelocal inference capabilityVSAvoidthroughput

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The system dynamically adapts model execution by analyzing edge device characteristics (FLOPS, RAM, power usage) and web service provider capabilities to generate optimized model variants or profiles. This allows the system to transition between edge device and cloud execution based on current resource availability, optimizing throughput while maintaining local inference capability when conditions permit.

Inventive Principle:

Principle #15Dynamics

2Adaptability or versatility

If ML models are deployed on edge devices, then local inference capability is improved, but performance in terms of throughput, latency, and accuracy deteriorates due to limited hardware capabilities

Engineering Contradiction:

Improvelocal inference capabilityVSAvoidlatency

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

Inventive Principle:

Principle #15Dynamics

3Adaptability or versatility

If ML models are deployed on edge devices, then local inference capability is improved, but performance in terms of throughput, latency, and accuracy deteriorates due to limited hardware capabilities

Engineering Contradiction:

Improvelocal inference capabilityVSAvoidaccuracy

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

Inventive Principle:

Principle #15Dynamics

4Adaptability or versatility

If complex ML models are executed on edge devices, then model functionality is improved, but resource consumption exceeds available hardware capabilities

Engineering Contradiction:

Improvemodel functionalityVSAvoidpower usage

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The system analyzes edge device characteristics including power usage constraints and dynamically generates optimized model variants by modifying model parameters such as quantization levels, layer configurations, and operator selections. This allows complex model functionality to be maintained while adapting resource consumption to match available hardware capabilities.

Inventive Principle:

Principle #35Parameter changes

5Adaptability or versatility

If complex ML models are executed on edge devices, then model functionality is improved, but resource constraints make execution impractical

Engineering Contradiction:

Improvemodel functionalityVSAvoidhardware capability requirements

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11423283B1Model adaptation

Publication Date: 2022.08.23 AMAZON TECH INC

AI summary

Techniques for model adaptation are described. For example, a method of receiving a call to provide either a model variant or a model variant profile of a deep learning model, the call including desired performance of the deep learning model, a deep learning model identifier, and current edge device characteristics; comparing the received current edge device characteristics to available model variants and profiles based on the desired performance of the deep learning model to generate or select a model variant or profile, the available model variants and profiles determined by the model identifier; and sending the generated or selected model variant or profile to the edge device to use in inference is detailed.