Scalable De-duplication via Distributed Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data de-duplication schemes face scalability issues as the amount of data stored continues to grow, leading to performance problems due to large indexes that cannot adequately scale to handle large data sets, particularly in enterprise environments with petabytes of data.
Innovation Solution
The method involves accessing initial partitions of files, determining indicators with high metadata similarity ratios to identify duplicates, and generating optimized partitions for de-duplication, which allows for a more efficient de-duplication process by distributing de-duplication indexes across multiple network-connected computer systems, enabling scalable de-duplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional de-duplication schemes are used, then de-duplication functionality is provided, but scalability fails as data volume grows to petabytes
Solution Approach 1:
The patent divides the storage system into multiple partitions and distributes de-duplication indexes across multiple network-connected computer systems. Each partition is handled independently, allowing the system to scale by adding more partitions and computer systems rather than requiring a single monolithic index to handle all petabytes of data.
2Reliability
If indexes are made large enough to track all data, then de-duplication coverage is complete, but lookup performance deteriorates due to disk accesses
Solution Approach 1:
The de-duplication index is segmented and distributed across multiple computer systems. Each system maintains a portion of the index in memory, enabling fast local lookups without requiring access to a monolithic large index on disk. This segmentation allows complete coverage while maintaining performance.
Solution Approach 2:
The patent introduces a distributed index structure where metadata and indexing information are stored in memory across multiple computer systems rather than on disk. This intermediary memory layer enables fast lookups while the distributed architecture provides complete coverage for petabyte-scale data.
3Productivity
If block and bit de-duplication are used, then de-duplication efficiency is improved, but processing power requirements increase significantly
Solution Approach 1:
The de-duplication process is segmented into partition-based operations distributed across multiple computer systems. Each system processes only its assigned partition, reducing the processing burden on any single system while maintaining overall efficiency through parallel processing of multiple partitions simultaneously.
Data Source
AI summary
A method for performing storage system de-duplication. The method includes accessing a plurality of initial partitions of files of a storage system and performing a de-duplication on each of the initial partitions. For each duplicate found, an indicator comprising the metadata that is similar across said each duplicate is determined. For each indicator, indicators are determined that infer a likelihood that data objects with said indicators contain duplicate data is high. Optimized partitions are generated in accordance with the chosen indicators. A de-duplication process is subsequently performed on each of the optimized partitions.