Erasure-Coded Object Deduplication Across Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in efficiently managing duplicate data across geographically distributed storage locations, leading to increased computational load and storage requirements, especially when using erasure coding and deduplication techniques.
Innovation Solution
A distributed storage topology is implemented where object data is divided into chunks and a parity chunk is generated, with deterministic hashing algorithms ensuring that duplicate chunks are stored on the same system, allowing for local deduplication and balanced computational load by dynamically assigning chunk responsibilities across systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is divided into chunks and distributed across geographically dispersed storage locations with erasure coding, then data reliability and redundancy are improved, but storage overhead and computational load increase
Solution Approach 1:
The patent divides data into multiple chunks and distributes them across different storage locations. Each chunk is independently stored with erasure coding parity information, allowing the system to reconstruct the original data even if some chunks are lost. This segmentation approach improves reliability while managing storage overhead through localized parity calculation.
Solution Approach 2:
The patent implements local deduplication at each storage node rather than global deduplication. Each node independently identifies and deduplicates chunks within its local storage, reducing the computational complexity and communication overhead associated with global deduplication while still achieving significant storage savings. This local quality approach balances reliability with reduced storage overhead.
2Quantity of substance
If global deduplication is implemented across distributed storage locations, then storage efficiency is improved, but computational load and management complexity increase
Solution Approach 1:
The patent segments the deduplication process into local operations at each storage node. Instead of requiring centralized global deduplication computation, each node independently processes its local chunks, identifying duplicates within its own storage. This segmentation dramatically reduces the computational load and management complexity while maintaining storage efficiency through local deduplication ratios.
Solution Approach 2:
Each storage node autonomously performs deduplication operations on its local chunks without requiring coordination with other nodes. The nodes self-manage their own deduplication processes, generating local deduplication metadata and maintaining chunk indexes independently. This self-service approach eliminates the need for complex global management while achieving practical storage efficiency improvements.
3Reliability
If duplicate chunks are distributed across different storage locations, then data redundancy is improved, but deduplication efficiency decreases
Solution Approach 1:
The patent implements local deduplication at each storage node, where duplicate chunks are identified and consolidated within the local storage context. Each node maintains a local chunk index and compares incoming chunks against its existing chunks, achieving deduplication without requiring global coordination. This local quality approach maintains data redundancy across distributed locations while improving deduplication efficiency through localized processing.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In some examples, a computer system may receive a data object including object data. For example, the system may be one of three or more systems and each system may be located at a different respective geographic location. The system may determine a value representative of the object data content. The system may further determine a plurality of chunks by dividing the object data into a plurality of data chunks based on a total number of the systems and determining a parity chunk for the plurality of data chunks. The system may determine a respective role value corresponding to each of the systems, and may send individual ones of the chunks to respective ones of the systems based on the role value and the value representative of the object data content. In some examples, each system may perform data deduplication independently of the other systems based on received chunks.