Distributed Data De-duplication Using Timestamp Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data de-duplication systems face scalability and integrity issues due to reliance on reference counts for managing unreferenced blocks, leading to performance degradation and potential data loss, especially in large and dynamic environments where millions of blocks are referenced by millions of files.
Innovation Solution
A method that identifies and removes unreferenced blocks without relying on reference counts, using a system with 'LastFCSValidationTime' and 'LastBlockExistCheckTime' attributes, and metadata storage containers like 'BlocksContainer' and 'RecycleBinForBlocks', allowing online integrity testing and self-healing capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reference counts are maintained for each block to track unreferenced blocks, then data integrity can be monitored, but system complexity and performance degradation occur due to expensive maintenance operations
Solution Approach 1:
The patent extracts the reference count maintenance mechanism from the system and replaces it with a validation time-based approach. Instead of tracking each block's reference count, the system uses 'LastFCSValidationTime' and 'LastBlockExistCheckTime' attributes to determine block validity, eliminating the complex reference counting operations while maintaining data integrity
Solution Approach 2:
The patent replaces the mechanical reference counting system with a time-based validation mechanism. By substituting the manual tracking of reference counts with automated timestamp comparisons, the system reduces complexity while preserving the ability to identify unreferenced blocks
2Reliability
If reference counts are updated for every file operation, then data integrity is maintained, but productivity decreases due to performance degradation
Solution Approach 1:
The patent enables continuous data operations without interruption for reference count updates. By using validation timestamps that are set during normal file operations rather than requiring separate update operations, the system maintains both data integrity and continuous productivity
Solution Approach 2:
The system performs self-validation through timestamp comparisons without requiring external reference count management. The validation mechanism automatically determines block integrity by comparing timestamps, eliminating the need for additional productivity-costing operations
3Reliability
If the system is taken offline for integrity testing, then data integrity can be verified, but system availability is reduced
Solution Approach 1:
The patent performs integrity validation continuously in the background using pre-established timestamp attributes. By conducting validation checks as a preliminary and ongoing process rather than a periodic offline operation, the system maintains both integrity verification capability and continuous availability
Solution Approach 2:
The validation timestamp attributes serve as intermediaries that enable integrity verification without direct system intervention. These timestamps mediate between data operations and integrity checks, allowing verification to proceed independently without taking the system offline
4Quantity of substance
If distributed machines are added to increase storage capacity, then storage scalability improves, but maintaining referential integrity across all machines increases complexity
Solution Approach 1:
The patent segments the integrity verification process into independent timestamp comparisons that can be performed locally on each distributed machine. By dividing the global integrity problem into local timestamp validation tasks, the system achieves storage scalability without proportionally increasing coordination complexity
Solution Approach 2:
The validation timestamp mechanism serves as a universal solution applicable across all distributed machines. This single approach handles integrity verification, block validation, and unreferenced block identification uniformly across the distributed system, preventing complexity multiplication
Data Source
Figure 1
Figure 2
Figure 3
AI summary
This disclosure relates to systems and methods for both maintaining referential integrity within a data storage system, and freeing unused storage in the system, without the need to maintain reference counts to the blocks of storage used to represent and store the data.