Chain Schedule Management for ML Inference Serving
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing approaches for machine learning (ML) model processing face challenges in schedule management, particularly in edge computing environments with heterogeneous execution environments, high latency, and resource constraints, making it difficult to efficiently manage inference serving for ML models.
Innovation Solution
The implementation of a cluster-less, server-less, and model-less Inference-as-a-Service (INFaaS) platform that enables developers to deploy ML models and specify execution requirements, with the platform automatically selecting the appropriate model and execution environment based on various variables such as accuracy, cost, and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If ML model inference serving is deployed in edge computing environments with heterogeneous execution environments, then the system can process requests closer to data sources reducing latency, but the complexity of managing diverse execution environments increases significantly
Solution Approach 1:
The patent introduces a chain scheduling system as an intermediary layer between ML model deployment requests and heterogeneous execution environments. This scheduler manages the complexity by abstracting environment-specific details and providing unified scheduling algorithms that can handle diverse edge computing platforms, thus reducing the burden on developers while maintaining low latency performance
Solution Approach 2:
The patent segments the scheduling process into multiple independent algorithms that can be selectively applied to different parts of the execution chain. By dividing the complex scheduling problem into smaller, manageable algorithmic components, the system can address specific heterogeneity challenges in different execution environments without being overwhelmed by the entire system's complexity
2Reliability
If multiple scheduling algorithms are utilized to optimize ML model execution, then the quality of scheduling decisions improves, but the computational overhead and time required for decision-making increases
Solution Approach 1:
The patent implements a chain scheduling approach where multiple algorithms are applied sequentially in a chain, but not all algorithms need to be executed for every scheduling decision. The system can stop the chain early when sufficient scheduling quality is achieved, thus obtaining high-quality decisions without always incurring the maximum computational overhead of running all algorithms
Solution Approach 2:
The patent performs preliminary actions by pre-evaluating and ranking scheduling algorithms based on their performance characteristics. This allows the system to select and apply only the most appropriate algorithms for specific scheduling scenarios, reducing the number of algorithms that need to be executed in real-time while maintaining high decision quality
3Adaptability or versatility
If ML models are deployed across diverse computing environments with resource constraints, then the system achieves better scalability and flexibility, but the difficulty of managing resource allocation and execution increases
Solution Approach 1:
The patent creates a universal chain scheduling system that can manage multiple types of computing environments (edge devices, cloud platforms, hybrid systems) through a single unified framework. This multi-functional scheduler handles diverse resource constraints and execution requirements without requiring separate management systems for each environment type, thus improving deployability while controlling management complexity
Data Source
AI summary
Techniques are disclosed for chain schedule management for machine learning model-based processing in a computing environment. For example, a method receives a machine learning model-based request and determines a scheduling decision for execution of the machine learning model-based request. Determination of the scheduling decision comprises utilizing a set of one or more scheduling algorithms and comparing results of at least a portion of the set of one or more scheduling algorithms to identify execution environments of a computing environment in which the machine learning model-based request is to be executed. The identified execution environments may then be managed to execute the machine learning model-based request.