iSCSI Data De-duplication via Hash Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As data managed in networks implementing the iSCSI protocol increases, so does the amount of redundant data, reducing available storage capacity, necessitating methods to identify and eliminate redundant data efficiently.
Innovation Solution
The implementation of data de-duplication systems that analyze write commands to determine duplicate data by deriving probabilistically unique identifiers and using routing records to re-route requests to existing data instances, reducing redundant storage across iSCSI networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored in iSCSI networks without de-duplication, then data management is simple, but storage capacity is wasted due to redundant data
Solution Approach 1:
The system performs preliminary de-duplication by deriving unique identifiers from data blocks before storage operations complete. Hash values are calculated and stored in routing tables in advance, allowing duplicate detection and elimination to occur proactively rather than reactively, thus preserving storage capacity without significantly increasing operational complexity
Solution Approach 2:
A routing table acts as an intermediary data structure that maps data block addresses to their unique identifiers. This intermediary layer enables the system to track and identify duplicate data blocks efficiently without requiring complex direct comparison of all stored data, thus reducing storage waste while maintaining manageable complexity through structured organization
2Quantity of substance
If de-duplication is implemented to eliminate redundant data, then storage capacity is optimized, but network traffic increases due to data analysis and identifier derivation
Solution Approach 1:
The system extracts only the essential identifying特征 of data blocks (hash values) rather than transmitting or storing entire duplicate data blocks. By deriving and comparing unique identifiers instead of full data content, the system eliminates redundant storage while minimizing additional network traffic to only the lightweight identifier data
Solution Approach 2:
The system transforms data blocks into a different parameter representation (hash values/unique identifiers) that are much smaller and more efficient for comparison and storage. This parameter transformation allows the system to identify and eliminate duplicates through compact identifier manipulation rather than bulky data comparison, optimizing storage while keeping network overhead low
3Quantity of substance
If de-duplication systems analyze write commands to identify duplicates, then redundant storage is reduced, but processing time increases
Solution Approach 1:
Unique identifiers are derived and routing table entries are created in advance during the write command processing phase, before actual data storage occurs. This preliminary processing allows duplicate detection to happen proactively, preventing redundant storage operations and reducing overall processing time by avoiding subsequent de-duplication steps
Solution Approach 2:
The system replaces complex mechanical or algorithmic data comparison processes with mathematical hash function computations. Deriving unique identifiers through hash functions is computationally more efficient than traditional byte-by-byte data comparison, thus reducing processing time while effectively identifying duplicate data blocks for storage optimization
Data Source
AI summary
Redundant data is identified and eliminated in a network that implements the iSCSI protocol either in-band at the source, in-band at the target, or out-of-band at the target. For in-band de-duplication, a data block included with a write command is assigned a unique identifier that is compared to a database of unique identifiers corresponding to previously written data. If the unique identifier is identical to an existing unique identifier, this indicates that the data block is redundant and has previously been stored elsewhere, in which case it is not stored again. Instead, the storage address specified in the write command may be added to a routing table showing the equivalence of unique identifiers, actual storage addresses, and duplicate storage addresses. When a read request specifying a duplicate storage address is received, the duplicate storage address can be translated to a corresponding unique identifier which points to the actual storage address.