Model State Update Manager for Consistent ML Across Data Centers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database and in-memory caching systems lack mechanisms to maintain consistent model states and input profiles across multiple data centers, leading to erroneous and inconsistent model results in stateful machine learning models.
Innovation Solution
A computer-implemented method that obtains and determines feature profiles and model states from a first implementation of a machine learning model, checks if a model policy is satisfied, and updates a second implementation of the model based on the satisfied policy, ensuring consistent and accurate model results across data centers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If existing database and in-memory caching systems are used to store model states and feature profiles, then data storage and retrieval are simplified, but model consistency across multiple data centers deteriorates due to lack of update control mechanisms
Solution Approach 1:
The system implements a feedback mechanism where model state updates are tracked and propagated across data centers. The model state update manager receives updates from one data center and distributes them to other data centers, ensuring that all centers maintain consistent model states. This feedback loop prevents inconsistency by continuously synchronizing state information across the distributed system.
Solution Approach 2:
A model state update manager acts as an intermediary between different data centers. This intermediary component receives model state updates from one data center and distributes them to other data centers, coordinating the update process to maintain consistency. The intermediary ensures that updates are propagated reliably without requiring direct peer-to-peer communication between data centers.
2Productivity
If model states are updated independently at each data center, then processing speed is improved, but model accuracy deteriorates due to inconsistent states across implementations
Solution Approach 1:
The system performs preliminary actions by establishing update policies and synchronization mechanisms before model state changes occur. The model state update manager pre-configures how updates should be propagated across data centers, ensuring that when updates occur, they maintain consistency. This preliminary setup enables fast processing while preserving accuracy through pre-planned coordination.
Solution Approach 2:
The system implements dynamic update propagation where model states are synchronized across data centers in real-time as changes occur. Rather than static periodic updates, the system dynamically responds to state changes by propagating them to other data centers. This dynamic approach maintains model accuracy while supporting high processing speeds through event-driven synchronization.
3Reliability
If all feature profiles are replicated across data centers, then model consistency is improved, but data transmission overhead increases
Solution Approach 1:
The system extracts and replicates only the essential model state information that is critical for consistency, rather than replicating all feature profiles. The model state update manager identifies and propagates only the necessary state changes, filtering out redundant information. This selective extraction maintains model consistency while reducing data transmission overhead by transmitting only essential updates.
Solution Approach 2:
Each data center maintains local copies of model states with quality tailored to its specific needs. Rather than uniform replication of all data, each data center receives and stores only the model state information relevant to its operations. This local quality approach ensures consistency where needed while minimizing unnecessary data transmission across the network.
Data Source
AI summary
A method, system, and computer program product for managing model updates at multiple data centers hosting a same machine learning model obtain a plurality of first feature profiles input to a first implementation of a first machine learning model and a plurality of first model states determined from processing a model input with the first implementation; determine that a first model policy associated with the first machine learning model is satisfied, based on the plurality of first model states including a set of model states defined by the first model policy and the plurality of first feature profiles including a set of feature profiles defined by the first model policy; and in response to determining that the first model policy is satisfied, provide the plurality of first model states and the plurality of first feature profiles for updating at least one second implementation of the first machine learning model.