File Metadata Replication via Persistent Message Bus
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional scale-out storage systems face challenges in quickly synchronizing data across all sites when a new site is added, particularly due to the asynchronous nature of data migration and replication processes.
Innovation Solution
A system that includes a reconstruction module to recreate transaction history for a predetermined period and a cloud-scale persistent message bus to synchronize data across consumer sites, utilizing both historical and current transactions to efficiently integrate new sites into the network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is migrated to a cloud network using conventional eventing-based synchronization, then data can be replicated across sites, but the synchronization process becomes slow and unpredictable when new sites are added
Solution Approach 1:
The system performs preliminary actions by maintaining a persistent message bus that continuously captures and stores transaction events as they occur. When a new site is added, the synchronization process leverages this pre-captured transaction history to rapidly replicate data without waiting for incremental events to propagate, thus reducing synchronization time while maintaining reliability
Solution Approach 2:
The patent introduces a persistent message bus as an intermediary component that mediates between data production and consumption across sites. This message bus retains transaction events in a persistent store, enabling new sites to subscribe and receive historical transactions efficiently, thereby accelerating synchronization while ensuring data consistency across the distributed system
2Adaptability or versatility
If eventing-based synchronization is used across multiple sites, then data can be distributed for collaboration and disaster recovery, but quickly synchronizing data with newly added sites becomes challenging
Solution Approach 1:
The system performs preliminary actions by maintaining a persistent message bus that continuously captures and stores transaction events as they occur. When a new site is added, the synchronization process leverages this pre-captured transaction history to rapidly replicate data without waiting for incremental events to propagate, thus reducing synchronization time while maintaining reliability
Solution Approach 2:
The persistent message bus serves multiple functions: it acts as an event stream for real-time synchronization, a historical transaction store for catch-up synchronization, and a coordination mechanism for site management. This multi-functional design enables the system to efficiently handle both existing and new site synchronization needs, improving both scalability and synchronization speed
Data Source
AI summary
Methods that can accelerate replication of file metadata on different sites are disclosed herein. One method includes recreating, by a processor, a transaction history for a predetermined period of time for a set of consumer sites of a first storage network and synchronizing data in the set of consumer sites and a new consumer site added to the first storage network utilizing a cloud scale persistent message bus. In this method, the data is synchronized based on the recreated transaction history during the predetermined period of time and current transactions performed on each consumer site in the set of consumer sites. Apparatus, systems, and computer program products that can include, perform, and/or implement the methods are also disclosed herein.