SLA Gateway for Multi-Hardware AI Inference Routing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In IoT device networks, existing technologies face challenges in efficiently directing inference requests to appropriate AI acceleration resources in edge clouds, leading to high software overhead, latency issues, and increased total cost of ownership due to unpredictable request arrival rates and varying performance requirements.

Innovation Solution

A service-level agreement-based multi-hardware accelerated inference system is implemented, using a gateway to manage and direct requests to suitable AI inference appliances or models based on deadline, cost, and performance requirements, with hardware mechanisms to quickly distribute requests and reject unsatisfiable requests to maintain latency and cost constraints.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If software-based request routing is used to direct inference requests to AI acceleration resources, then adaptability to different performance requirements is improved, but processing speed and latency are worsened

Engineering Contradiction:

Improveadaptability to performance requirementsVSAvoidrequest processing speed

Core Design Contradiction:

Adaptability or versatilityVSSpeed

Solution Approach 1:

The patent introduces a gateway as an intermediary component that sits between IoT devices and AI inference appliances. The gateway maintains a service level agreement (SLA) table that maps request characteristics to appropriate inference appliances, acting as a mediator that directs requests based on pre-established routing rules rather than complex software-based matching, thus improving speed while maintaining adaptability

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system performs preliminary action by pre-configuring the SLA table in the gateway with routing information for different types of inference requests. This pre-establishment of routing paths allows the system to quickly direct requests without real-time complex decision-making, resolving the contradiction between adaptability and processing speed

Inventive Principle:

Principle #10Preliminary action

2Adaptability or versatility

If multiple AI inference appliances are deployed to handle different request types, then service coverage and adaptability are improved, but device complexity and total cost of ownership are worsened

Engineering Contradiction:

Improveservice coverageVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The gateway is designed as a universal component that handles multiple types of inference requests through a single multi-functional device. Instead of requiring separate routing infrastructure for each request type, the gateway provides a unified entry point that can direct various request types to appropriate AI inference appliances based on the pre-configured SLA table, reducing overall system complexity while maintaining broad service coverage

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The gateway acts as an intermediary that abstracts the complexity of multiple AI inference appliances from IoT devices. Devices only need to communicate with the single gateway interface, while the gateway handles the complexity of routing to appropriate appliances, thus reducing perceived system complexity for end users while maintaining comprehensive service coverage

Inventive Principle:

Principle #24Intermediary (Mediator)

3Loss of energy

If centralized request management is implemented to optimize resource allocation, then cost efficiency is improved, but processing latency is worsened due to additional routing steps

Engineering Contradiction:

Improvecost efficiencyVSAvoidprocessing latency

Core Design Contradiction:

Loss of energyVSLoss of time

Solution Approach 1:

The system performs preliminary action by pre-configuring optimal routing paths in the SLA table during system setup or low-traffic periods. This allows the gateway to quickly direct requests along pre-determined efficient paths without real-time optimization overhead, maintaining cost efficiency through centralized management while minimizing additional latency from routing decisions

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent segments the system into two distinct functional layers: the gateway layer that handles intelligent routing decisions using pre-configured SLA rules, and the AI inference layer that executes computations. This segmentation allows centralized resource allocation optimization at the gateway level while keeping inference processing latency minimal by not involving the gateway in the actual computation phase

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS11743143B2Service level agreement-based multi-hardware accelerated inference

Publication Date: 2023.08.29 INTEL CORP

AI summary

Various systems and methods for implementing a service-level agreement (SLA) apparatus receive a request from a requester via a network interface of the gateway, the request comprising an inference model identifier that identifies a handler of the request, and a response time indicator. The response time indicator relates to a time within which the request is to be handled indicates an undefined time within which the request is to be handled. The apparatus determines a network location of a handler that is a platform or an inference model to handle the request consistent with the response time indicator, and routes the request to the handler at the network location.