Asynchronous ML Model Execution via Queue Decoupling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing ML model execution systems operate synchronously, leading to idle times for applications as they wait for output data to be processed before pre-processing new input data, resulting in inefficient resource utilization.
Innovation Solution
Implementing a method that allows asynchronous ML model execution by queuing pre-processed input data, decoupling pre-processing from ML model execution, and enabling continuous input data streaming to the ML model without waiting for post-processing to finish, using an ML controller with a queueing mechanism to manage input data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If synchronous ML model execution is used, then data processing follows a straightforward sequential flow, but applications experience idle wait times during ML model execution
Solution Approach 1:
The system segments the data processing workflow into distinct independent components: pre-processing, queuing, ML model execution, and post-processing. Each component operates independently, allowing the application to submit pre-processed data to a queue and continue execution without waiting for ML model completion. This segmentation eliminates idle wait times while maintaining operational simplicity through clear separation of concerns.
Solution Approach 2:
An intermediary queue mechanism is introduced between the application and the ML model execution system. The queue acts as a buffer that decouples the application from the ML processing timeline, allowing applications to submit data and immediately proceed to subsequent tasks while the ML model processes data asynchronously. This intermediary eliminates waiting time without complicating the application's control flow.
2Reliability
If applications wait for post-processing to complete before pre-processing new data, then data dependency is maintained, but resource utilization decreases
Solution Approach 1:
The system enables continuous useful action by allowing applications to perform pre-processing of new data immediately after submitting previous data to the queue, without waiting for ML model execution or post-processing to complete. Multiple data processing operations can overlap in time, ensuring that applications and ML processing resources both remain continuously productive rather than idle.
Solution Approach 2:
The system performs preliminary pre-processing of data before ML model execution is required. Applications pre-process data in advance and submit it to the queue, so when the ML model becomes available, processed data is already ready for immediate execution. This preliminary action eliminates waiting time and ensures continuous utilization of ML processing resources.
3Productivity
If asynchronous execution with queuing is implemented, then resource utilization improves, but system complexity increases
Solution Approach 1:
A queue data structure serves as a simple intermediary component that manages the asynchronous execution flow. The queue requires minimal complex logic, primarily handling data storage and retrieval operations. This simple intermediary enables asynchronous processing and improved resource utilization without introducing significant system complexity, as queues are a fundamental and well-understood data structure.
Solution Approach 2:
The queue mechanism operates autonomously to manage the asynchronous workflow, automatically storing pre-processed data and retrieving it for ML model execution without requiring complex coordination or control logic. This self-service approach to data management improves resource utilization while maintaining relatively simple system architecture, as the queue handles timing and coordination independently.
Data Source
AI summary
A method for processing data with machine learning (ML) models. The method includes receiving, at an ML controller from at least one ML application, at least one request to run at least one ML model. The method includes queueing, at the ML controller, pre-processed input data for the at least one ML model. The method includes running, by the ML controller using an ML runtime, the at least one ML model. The running includes executing the at least one ML model by the ML runtime. The running includes starting processing of a second set of the queued input data with the at least one ML model independent from post-processing output data resulting from processing a first set of the queued input data.