Reverse Replication for Distributed Storage Traffic Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional storage area network (SAN) and network-attached storage (NAS) architectures face challenges in supporting large multi-petabyte capacity levels and efficient data replication across geographical zones, leading to high inter-zone network traffic.
Innovation Solution
The system employs a scale-out, cluster-based, shared-nothing object storage architecture with a microservices pattern, utilizing erasure coding for data protection and implementing reverse replication by transferring location information instead of data itself, reducing inter-zone network traffic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data replication schemes are used to store backup copies across geographical zones, then data durability and availability are improved, but inter-zone network traffic increases significantly
Solution Approach 1:
The patent extracts only the essential metadata (location information) from the complete data set and transmits only this extracted information across zones, rather than replicating entire data chunks. This selective extraction maintains data durability while minimizing network traffic.
Solution Approach 2:
Instead of the traditional approach where data is replicated from primary to backup zones, the patent inverts the process by having the primary zone query the backup zone for location information after data is initially stored. This reverse information flow reduces redundant data transmission.
2Reliability
If data is replicated across multiple geographical zones for disaster recovery, then system reliability is improved, but storage costs and complexity increase
Solution Approach 1:
The patent creates a simplified copy of data location information rather than copying the actual data. The primary zone stores only metadata pointing to the backup location, which is a minimal representation that maintains disaster recovery capability while reducing complexity.
Solution Approach 2:
The patent introduces location information as an intermediary between the primary and backup zones. This mediator element enables disaster recovery coordination without requiring direct complex interactions between zones, simplifying the overall system architecture.
3Reliability
If traditional mirroring or parity protection schemes are used, then data protection is achieved, but scalability to multi-petabyte capacity is limited
Solution Approach 1:
The patent segments data protection into two independent layers: (1) object-level erasure coding for primary data protection, and (2) zone-level location information replication for disaster recovery. This segmentation enables independent scaling of each layer to multi-petabyte capacities without traditional replication constraints.
Solution Approach 2:
The patent changes the fundamental parameter being replicated from data content to location metadata. This parameter transformation enables scalability to multi-petabyte capacities because metadata size remains constant regardless of data volume, unlike traditional mirroring where protected data size must be replicated.
Data Source
AI summary
Inter-zone data replication in a distributed storage system can be organized. In one aspect, a replication destination zone can send a new data portion to a replication source zone to reduce inter-zone traffic. The new data portion can be stored within one or more chunks within the replication source zone, which can then provide the replication destination zone with location information (e.g., chunk identifier (ID), offset within chunk, segment size, etc.) indicative of a location of the new data portion within one or more chunks. The location information can be utilized by the replication destination zone to store a backup copy of the new data portion. Since the backup copy is stored without data replication from the replication source zone to the replication destination zone, inter-zone network traffic is significantly reduced.