Distributed Storage Data Replication via Segmented Publish-Subscribe
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Replicating large quantities of data across a distributed database system is inefficient due to limited bandwidth in long-haul network paths and the need for manual intervention during network outages, especially when relying on static network topologies.
Innovation Solution
Data changes are tracked as deltas and transmitted only to other instances, utilizing a network tree structure to reduce data transmission across long-haul links, and chunks of data are distributed efficiently across multiple chunk stores with load balancing and cryptographic signatures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is transmitted to all database instances through long-haul network paths, then data replication is achieved, but network bandwidth is consumed inefficiently and replication speed is reduced
Solution Approach 1:
The patent segments the data transmission process by introducing intermediate nodes (publishers, subscribers, message queues) that divide the replication path into smaller segments. Instead of direct point-to-point transmission to all instances, data flows through segmented routes using publish/subscribe mechanisms and message queues, reducing the bandwidth burden on long-haul links while maintaining replication reliability.
2Device complexity
If static network topology is used for replication, then system simplicity is maintained, but adaptability to network outages and changes is reduced
Solution Approach 1:
The patent implements dynamic network topology adaptation through event-driven architecture where publishers and subscribers dynamically establish connections based on network conditions. The system uses message queues that can route data through alternative paths when outages occur, and the publish/subscribe mechanism allows flexible routing decisions that adapt to changing network topology without requiring static configuration changes.
3Ease of operation
If manual intervention is required during network outages, then system simplicity is maintained, but operational complexity and downtime increase
Solution Approach 1:
The patent implements self-service fault tolerance through automatic failover mechanisms where the message queue system dynamically redirects data flow without manual intervention. When network outages occur, the system automatically detects the failure and routes data through alternative paths or waits for recovery, eliminating the need for manual intervention while maintaining system reliability and reducing operational complexity.
Data Source
AI summary
A method for uploading an object into a distributed storage system is implemented at a computing device The computing device splits an object into one or more chunks and uploads the one or more chunks into the distributed storage system. For each uploaded chunk, the computing device receives a write token from the distributed storage system, inserts an entry into an extents table of the object for the chunk in accordance with the received write token and the chunk ID, chunk offset, and chunk size of the chunk, generates a digest of the extents table, the digest representing the one or more chunks that the client expects to be within the distributed storage system, and sends the digest of the extents table to the distributed storage system. The distributed storage system is configured to use the digest to determine whether it has each of the one or more client-expected chunks.