On-the-fly Data Deduplication via Hash Table Lookup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional storage systems face inefficiencies in data deduplication due to the need for extensive hashing calculations and increased storage space requirements, which can lead to performance issues and high costs from storing duplicate data.
Innovation Solution
A system and method for 'on-the-fly' data deduplication that uses a de-duplication module cooperating with protocol servers and a file system to generate a block store, index file, and hash table, allowing for real-time identification and elimination of duplicate data blocks before storage, utilizing a Rabin-Karp hash function to determine block fingerprints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored before deduplication in conventional systems, then storage space must be allocated for all data including duplicates, but this increases storage space consumption and costs
Solution Approach 1:
The patent applies preliminary action by performing deduplication processing on incoming data blocks before they are stored in the storage system. The system calculates fingerprints of data blocks, checks against existing fingerprints in the hash table, and determines whether to store the block or return a reference to an existing block before the data is physically written to disk. This prevents duplicate data from occupying storage space in the first place.
Solution Approach 2:
The patent introduces a hash table as an intermediary data structure that stores fingerprints of previously seen data blocks. Before storing a new data block, the system queries the hash table to check if an identical block already exists. This intermediary mechanism enables fast duplicate detection without scanning the entire storage system, resolving the contradiction between storage efficiency and processing time.
2Reliability
If extensive hashing calculations are performed for deduplication, then duplicate data can be identified, but this increases processing load and reduces system performance
Solution Approach 1:
The patent segments the data storage process into distinct phases: fingerprint calculation for each data block, hash table lookup, and storage decision. By segmenting the process and using a hash table for O(1) average-time complexity lookups, the system achieves accurate duplicate detection without extensive linear searching, thereby maintaining high data storage throughput while ensuring reliable duplicate identification.
Solution Approach 2:
The patent changes the parameter of duplicate detection from comparing entire data blocks (which would be computationally expensive) to comparing cryptographic fingerprints or hash values of the blocks. This parameter change maintains high reliability in duplicate detection while dramatically reducing the processing load and time required, thus preserving system productivity.
3Reliability
If all data blocks are stored initially, then data availability is ensured, but this increases storage costs and reduces storage efficiency
Solution Approach 1:
The patent uses copying by creating reference pointers to existing data blocks instead of storing duplicate copies. When a duplicate block is detected, the system stores a reference or pointer to the original block's location rather than writing the duplicate data to disk. This ensures data availability through multiple references to the same physical storage location while maximizing storage capacity utilization by eliminating redundant data.
Data Source
AI summary
A system and method for “on-the-fly” de-duplication of data before storing the data in a storage system. A data de-duplication module illustratively cooperates with protocol servers and a file system of a storage operating system executing on the storage system to implement the novel de-duplication technique. The de-duplication module illustratively generates a block store, an index file and a hash table on storage space provided by the storage system. The hash table is utilized for tracking fingerprints and locations of blocks within the block store. The index file is utilized for storing directory information identifying the contents of data containers stored on the storage system, while the block store is utilized to store raw data blocks that comprise the data containers.