Block-Level Single-Instance Storage Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face inefficiencies due to repeated data blocks, which occupy additional storage space and are not effectively eliminated by existing file-level single instance storage systems, especially in cases where files have only slight differences.
Innovation Solution
Implementing a block-level single instance storage system using a pair of lookup tables, a hash lookup table, and a physical-address lookup table, where data blocks are identified by signatures (hashes) rather than full block contents, allowing for the elimination of redundant data blocks during storage and retrieval operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If file-level single instance storage systems are used, then some duplicate files can be eliminated, but repeated data blocks within files or between slightly different files cannot be effectively eliminated
Solution Approach 1:
The patent divides files into fixed-size data blocks and applies hashing to each block individually. This segmentation allows the system to identify and eliminate duplicate blocks even when files are only partially similar, rather than treating entire files as single units. The lookup tables store hash values of individual blocks, enabling fine-grained deduplication.
Solution Approach 2:
The patent introduces hash values as an intermediary representation of data blocks. Instead of comparing actual block contents directly, the system compares compact hash signatures. This intermediary approach enables efficient duplicate detection with minimal storage overhead, as only hash values (not full block contents) need to be stored in the lookup tables.
2Quantity of substance
If all data blocks are stored to ensure data availability, then data retrieval reliability is maintained, but storage space is wasted due to repeated blocks
Solution Approach 1:
The patent uses hash values as copies or representations of the actual data blocks. Instead of storing multiple physical copies of identical data blocks, the system stores a single copy and uses its hash value as a reference. The lookup tables map logical addresses to physical block locations, allowing the system to reference existing blocks rather than duplicating them, thereby saving storage space while maintaining data availability.
3Productivity
If block-level hashing is implemented to eliminate duplicates, then storage efficiency is improved, but additional lookup tables and processing overhead are introduced
Solution Approach 1:
The patent designs the lookup tables to serve multiple functions: they map logical addresses to physical block locations, store hash values for duplicate detection, and enable both read and write operations. The same lookup table structure supports both deduplication during writes and verification during reads, reducing the need for separate specialized components and minimizing overall system complexity.
Data Source
AI summary
A method for reading data in a block-level single-instance storage system may involve receiving a first address of a data block, retrieving a signature corresponding to the first address, and reading data from a second address corresponding to the signature. A storage system may include a storage manager and first and second lookup tables. The storage manager may interface with an application (such as a database system or a file system) that uses a first set of identifiers for data blocks. The storage manager may use a second set of identifiers for the data blocks, and translates between the first and second identifiers using the lookup tables. The first lookup table indexes data block signatures according to the first set of identifiers. The second lookup table indexes the second set of identifiers according to the data block signatures. The second lookup table may be pruned to provide single instance storage.