Real-time ML Model Update via Worker Thread Orchestration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale real-time machine learning applications, existing mechanisms cannot update models in real time without causing data loss or system downtime, especially due to the large number and size of models that need to be managed, such as in scenarios with thousands of host models or millions of user clusters.
Innovation Solution
The solution involves an orchestrator that maintains an internal mapping data structure to redirect input data from an existing worker thread to a new worker thread during model updates, allowing the updated scorer to provide scoring capabilities while loading the new model segment by segment, thus avoiding downtime and data loss. This approach segments models into smaller parts stored across multiple libraries, enabling faster loading and resilience against network or server failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the model is updated by replacing the existing worker thread with a new worker thread, then the updated model can be executed, but system downtime and data loss occur during the transition
Solution Approach 1:
The patent creates a new worker thread before the existing worker thread completes its current scoring tasks. The new worker thread is pre-configured with the updated model and ready to take over, eliminating the need to wait for the old thread to finish naturally. This preliminary preparation ensures continuous service without downtime.
Solution Approach 2:
The patent introduces an orchestrator as an intermediary component that manages the transition between worker threads. The orchestrator coordinates the handover of scoring tasks from the existing worker thread to the new worker thread, ensuring smooth transition without data loss or system interruption.
2Measurement precision
If the model size increases to capture more specific behaviors, then prediction accuracy improves, but the time to load and update the model increases
Solution Approach 1:
The patent loads the updated model into the new worker thread in advance, before the new thread needs to start scoring. This preliminary loading allows the system to prepare the large model offline or during low-utilization periods, so when the thread activation is needed, the model is already ready, minimizing actual loading time impact on service.
Solution Approach 2:
The patent creates a copy of the worker thread with the updated model rather than modifying the existing thread in place. This copying approach allows the original thread to continue processing with the old model while the new thread is being prepared with the updated model, enabling parallel preparation and reducing overall update time.
3Productivity
If the model is segmented into smaller parts stored across multiple libraries, then loading speed and resilience improve, but system complexity increases
Solution Approach 1:
The patent divides the large model into multiple segments or chunks that are stored separately in different libraries or storage locations. This segmentation allows the system to load only the necessary segments into memory, reducing peak memory requirements and enabling faster loading by parallelizing the load operation across multiple storage sources.
Solution Approach 2:
The patent creates a universal loading mechanism that can handle models whether they are stored as a single unit or segmented across multiple libraries. The orchestrator and worker threads are designed to work with both monolithic and segmented model formats, providing flexibility without significantly increasing system complexity.
Data Source
AI summary
A mechanism is provided for implementing a model update mechanism to update new models in real time while avoiding data loss and system downtime. Responsive to receiving a request to update a scorer model currently being executed by an existing worker thread in the data processing system, the model update mechanism initializing a new worker thread. The model update mechanism loads an updated scorer model into the new worker thread and initializes a state transfer from the existing worker thread to the new worker thread. The model update mechanism executes the updated scorer model such that the updated scorer model scores the input data. The model update mechanism then outputs a prediction based on the updated scorer model processing of the input data.