Serverless ML Inference Across Heterogeneous Device Fleets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional machine learning inference systems require significant resources and are inefficient due to dedicated fleets of devices configured for specific models, leading to underutilization of resources when demand is low and inefficiencies in handling multiple models.
Innovation Solution
A serverless machine learning inference system that dynamically allocates resources across a fleet of devices with heterogeneous physical configurations, selecting the most efficient configuration based on current resource utilization to handle multiple models and meet varying user performance requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a dedicated fleet of devices is configured specifically to support inference operations for a particular model, then inference performance and reliability are improved, but resource utilization efficiency deteriorates due to significant unused resources reserved by the fleet
Solution Approach 1:
The patent applies universality by enabling a single heterogeneous fleet of devices to support multiple different ML models through dynamic configuration. Instead of dedicating specific devices to specific models, the system allows any device in the fleet to handle any model's inference operations by dynamically allocating resources and configuring execution environments based on current demand, thereby achieving multi-functionality across the infrastructure
Solution Approach 2:
The patent implements dynamics by dynamically adjusting the configuration and allocation of devices based on real-time inference demands. The system continuously monitors workload requirements and reconfigures the fleet composition and device assignments accordingly, transforming the static dedicated fleet model into a dynamic multi-tenant architecture that adapts to varying inference needs across different models
2Speed
If large, dedicated fleets of devices are built to support inference operations, then inference speed and responsiveness are improved, but system complexity and operational cost increase significantly
Solution Approach 1:
The patent introduces an intermediary layer (the dynamic resource allocation system) between the inference requests and the physical device fleet. This intermediary manages the complexity of fleet composition and device assignment by automatically matching inference workloads with appropriate device configurations, eliminating the need for manual fleet management while maintaining high inference speeds through optimized resource allocation
3Adaptability or versatility
If a heterogeneous fleet of devices is used to support multiple ML models, then resource flexibility and adaptability are improved, but system complexity in managing diverse configurations increases
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting device configuration parameters (such as allocated memory, processor allocation, and execution environment settings) based on the specific requirements of different ML models. The system modifies these parameters on-the-fly when allocating devices to different models, enabling the heterogeneous fleet to adapt to various model requirements without manual reconfiguration
Data Source
AI summary
Systems and methods are described for providing for serverless inferences against a trained machine learning (ML) model. Rather than obtaining one or more dedicated devices to conduct inferences, users are enabled to create a task on a serverless system that, when invoked, passing input data to a trained ML model and provides a result. To satisfy varying user requirements for inference speed, the system includes a variety of hardware configurations. The system can efficiently allocate resources between different tasks by invoking the task on a particular hardware configuration that is selected based on a current availability of the selected hardware configuration to host an execution environment in which the task is implemented and an expected time to invoke the task on the individual hardware configuration. The system can therefore efficiently allocate resources among inferences using a variety of different ML models.