Push-Based Metadata Synchronization for Data Staleness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems face challenges in synchronizing metadata across different data centers or cloud environments in real-time, particularly due to the staleness of data in pull-based models, where changes may not be reflected until the next pull occurs, leading to inefficiencies and delayed updates.
Innovation Solution
Implementing a push-based caching method where metadata is periodically pushed from the source data storage environment to the destination, leveraging change data capture functionality to track insert, update, and delete operations, and allowing the destination to specify filtering conditions to avoid unnecessary data transmission, ensuring near real-time access to relevant changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If pull-based metadata synchronization is used, then data retrieval simplicity is improved, but data staleness increases
Solution Approach 1:
The patent inverts the traditional pull-based synchronization model by implementing a push-based model. Instead of the destination periodically pulling metadata from the source, the source actively pushes metadata changes to the destination. This inversion resolves the contradiction by eliminating data staleness (improving time freshness) while maintaining operational simplicity through automated change capture and transmission mechanisms.
2Loss of time
If push-based caching method is implemented, then data freshness is improved, but system complexity increases
Solution Approach 1:
The source data storage environment performs self-service by automatically capturing its own changes through change data capture functionality and initiating push operations without requiring explicit requests from the destination. The system monitors its own state, identifies changes, and transmits them automatically, thereby achieving high data freshness while minimizing the complexity burden on the synchronization infrastructure.
3Loss of information
If all metadata changes are transmitted, then data completeness is improved, but data transmission volume increases
Solution Approach 1:
The patent extracts only the necessary metadata changes from the complete data set and transmits only those changes to the destination. By using change data capture to identify and isolate only the modified metadata records, the system maintains data completeness for synchronization purposes while significantly reducing the volume of data transmitted compared to sending all metadata.
Solution Approach 2:
The metadata synchronization process is segmented into change detection, change identification, and selective transmission stages. Only the changed metadata records are extracted and transmitted, rather than transmitting the entire metadata set. This segmentation approach ensures data completeness for what needs to be synchronized while minimizing transmission volume.
Data Source
AI summary
Methods, systems, and devices for data management are described. A destination data storage environment of a data management system may transmit, to a source data storage environment configured to run one or more applications, a request to synchronize metadata for the one or more applications from the source data storage environment to the destination data storage environment. In some examples, the request may include configuration information indicating one or more filtering parameters for filtering a data stream to identify a subset of a set of data records and start and stop times for pushing data to the destination data storage environment. The destination data storage environment may receive, from the source data storage environment, the subset of the set of data records based on the configuration information, where the subset of the set of data records are determined from a filtering operation at the source data storage environment.