Deduplication Digest Segmentation for Storage Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication systems face inefficiencies due to random access patterns and the need for extensive index tables, leading to increased access times and resource consumption as the number of data blocks grows, especially when dealing with large datasets.
Innovation Solution
The proposed method organizes digests into digest segments based on temporal and spatial locality, using a rapid lookup digest segment stored in fast access media and applying a sampling criteria to reduce the number of indexed digests, thereby improving deduplication efficiency and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional data deduplication systems use extensive index tables to store all digests, then deduplication accuracy is improved, but access time and resource consumption increase significantly
Solution Approach 1:
The patent divides the index table into multiple segments organized as a trie data structure with prefix nodes and suffix nodes. Digests are distributed across different segments based on their prefix patterns, allowing the system to search only relevant segments rather than scanning the entire index table. This segmentation reduces access time while maintaining complete deduplication coverage.
Solution Approach 2:
The patent introduces a hierarchical dimension to the index structure by organizing digests into prefix nodes and suffix nodes across multiple levels. This multi-dimensional organization allows the system to navigate to specific digest segments efficiently using prefix matching, transforming a linear search problem into a logarithmic navigation problem.
2Quantity of substance
If the number of data blocks increases in conventional deduplication systems, then storage capacity is improved, but access time and resource consumption increase
Solution Approach 1:
The patent segments the index table into multiple partitions based on prefix patterns of digests. As storage capacity grows and more data blocks are added, new digests are automatically routed to appropriate segments based on their prefix values. This maintains efficient search performance regardless of the total number of data blocks, as each segment remains manageable in size.
Solution Approach 2:
The patent applies local quality by making different segments of the index table have specialized characteristics optimized for their specific prefix patterns. Each segment can be independently optimized and cached, allowing the system to handle large storage capacities while maintaining high deduplication performance through localized efficient access patterns.
3Adaptability or versatility
If random access patterns are used in conventional deduplication systems, then data retrieval flexibility is improved, but access time increases
Solution Approach 1:
The patent performs preliminary organization of digests into a structured trie-based index table before deduplication operations. Digests are pre-sorted and distributed into segments based on their prefix patterns, creating an optimized access structure in advance. This preliminary action transforms random access patterns into systematic navigational access, reducing access time while preserving retrieval flexibility.
Data Source
AI summary
A method is used in managing deduplication of data in storage systems. A digest is determined for a data object received for deduplication. A candidate data object is identified for deduplicating the data object. A digest associated with the candidate data object matches the digest associated with the data object. The digest in a digest segment is maintained based on identification of the candidate data object. The digest segment includes a set of digests associated with a set of data blocks identified for deduplication in an ordered arrangement.