Distributed Content Addressable Storage De-Duplication Across Replicas
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face challenges in efficiently managing redundant replicas across distributed data replication systems, leading to inefficiencies in storage space and network resources.
Innovation Solution
Implementing an asynchronous distributed de-duplication algorithm that utilizes a global index with unique content handles to identify and remove redundant replicas, ensuring availability, liveness, and consistency of immutable objects across distributed multi-master data replication systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across multiple geographical locations to increase availability and reduce network distance, then data availability and access speed are improved, but storage space is wasted due to redundant replicas
Solution Approach 1:
The patent extracts the index metadata from the actual data objects and stores it centrally in a code repository. This allows the build system to track and identify redundant replicas without storing duplicate data, thereby reducing storage space while maintaining availability through distributed caching of only the index information.
Solution Approach 2:
The patent creates lightweight copies of index metadata (content handles, timestamps, locations) rather than copying actual data objects. These index copies are stored in a centralized repository and distributed build caches, enabling redundancy detection without duplicating expensive data storage.
2Productivity
If a centralized build system is used to manage dependencies and caching, then build efficiency and resource utilization are improved, but the system complexity and network coordination overhead increase
Solution Approach 1:
The patent segments the build system into two independent parts: a centralized build server that manages the code repository and dependency graph, and distributed build agents that perform local caching and execution. This segmentation reduces coordination overhead by allowing autonomous operation at the agent level while maintaining centralized intelligence for dependency resolution.
Solution Approach 2:
The patent introduces an intermediary index metadata layer that mediates between the centralized build system and distributed caches. This intermediary stores content handles and dependency information, allowing build agents to independently determine what to cache and execute without constant communication with the central server, thereby reducing network overhead.
3Quantity of substance
If an index of objects is replicated across distributed devices to enable de-duplication, then storage efficiency is improved, but network bandwidth is consumed during index synchronization
Solution Approach 1:
The patent implements partial index replication where only the necessary portions of the index (content handles, timestamps, locations) are replicated to distributed build agents, not the complete index. This selective replication reduces network bandwidth consumption while providing sufficient information for local de-duplication and caching decisions.
Data Source
AI summary
A method is performed by a device of a group of devices in a distributed data replication system. The method includes storing an index of objects in the distributed data replication system, the index being replicated while the objects are stored locally by the plurality of devices in the distributed data replication system. The method also includes conducting a scan of at least a portion of the index and identifying a redundant replica(s) of the at least one of the objects based on the scan of the index. The method further includes de-duplicating the redundant replica(s), and updating the index to reflect the status of the redundant replica.