De-duplication Storage System with Multiple Indices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale backup storage systems face performance issues due to the inefficiency of managing large indices for de-duplication, as the index size exceeds memory limits and disk access speeds become slow, especially when dealing with billions of segments.
Innovation Solution
Implementing multiple indices, where frequently used segments are stored in fast storage like RAM or SSD, and less frequently used segments are stored on disk drives, allowing for efficient lookup and storage of file segments, with the ability to move indices from RAM to disk when full and maintaining a base index for frequent access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the index is stored in RAM to make lookup fast, then lookup speed is improved, but the system cannot scale to large capacities because the index size exceeds memory limits
Solution Approach 1:
The patent divides the single large index into multiple smaller indices that are distributed across different storage devices. Each index is stored on a separate storage device, allowing the system to handle large capacities by adding more storage devices rather than being limited by the capacity of a single memory or disk. This segmentation resolves the contradiction by enabling scalability while maintaining fast lookup speeds through distributed access.
Solution Approach 2:
The patent transitions from storing the entire index in a single dimension (either in RAM or on a single disk) to distributing the index across multiple dimensions (multiple storage devices). This dimensional expansion allows the system to scale horizontally by adding more storage devices, effectively solving the memory capacity limitation while maintaining performance through parallel access to multiple indices.
2Adaptability or versatility
If the index is stored on disk to handle large capacities, then scalability is improved, but lookup speed becomes slow due to poor disk access performance
Solution Approach 1:
The patent segments the index across multiple storage devices, allowing the system to achieve scalability by adding more devices while maintaining fast lookup speeds through distributed access. Each storage device holds a portion of the index, and the system can parallelize lookup operations across multiple devices, effectively resolving the speed-scalability tradeoff.
Solution Approach 2:
The patent merges multiple storage devices into a unified index management system where indices are distributed across different devices. This combination allows the system to leverage the high-speed access of multiple devices simultaneously, achieving both scalability and fast lookup performance by combining the capabilities of individual storage devices.
3Device complexity
If a single large index is used to manage all segments, then the system structure is simple, but performance deteriorates due to slow disk access and memory constraints
Solution Approach 1:
The patent segments the single large index into multiple smaller indices distributed across different storage devices. This segmentation increases structural complexity slightly but dramatically improves productivity by enabling parallel index operations and eliminating the bottleneck of single-device access. The distributed architecture allows multiple index operations to occur simultaneously, resolving the contradiction between simplicity and performance.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A de-duplication storage system which uses multiple indices is described. A first group of one or more indices may be stored in random access memory (RAM) or another type of fast storage. A second group of one or more indices may be stored on one or more disk drives or another type of storage where large amounts of data can be stored inexpensively. The first group of indices may be used when adding new files to the de-duplication storage system in order to determine whether the file segments of the new files are already stored. The second group of indices may be used when restoring files in order to lookup the segments of the files.