Metadata-Separated Object Storage for File Segment Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage methods are inefficient in reducing storage requirements, as they do not effectively deduplicate data before storage, leading to unnecessary storage space usage.
Innovation Solution
The system employs an object storage architecture that deduplicates data by segmenting files, identifying and deleting duplicate segments, and storing the remaining segments along with their meta-data, using fingerprints to manage storage efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is stored without deduplication, then storage operations are simple and fast, but storage space is wasted due to duplicate data
Solution Approach 1:
The patent segments files into fixed-size chunks before storage. Each chunk is independently hashed and stored, allowing efficient deduplication at the chunk level rather than processing entire files. This segmentation enables the system to identify and eliminate duplicate portions while maintaining simple storage operations for unique data.
Solution Approach 2:
The patent uses cryptographic hashing to create unique identifiers (fingerprints) for each data chunk. Instead of storing actual duplicate data, the system stores references to existing chunks based on their hash values. This copying approach eliminates waste while maintaining data integrity and enabling fast deduplication operations.
2Quantity of substance
If data is deduplicated using traditional methods, then storage space is reduced, but disk I/O operations and bandwidth usage increase
Solution Approach 1:
The patent performs deduplication during the initial data ingestion phase rather than as a separate post-processing step. Data chunks are hashed, deduplicated, and stored in a single pass through the system. This preliminary action eliminates the need for additional disk I/O and bandwidth consumption that would occur with traditional two-stage deduplication approaches.
Solution Approach 2:
The patent introduces an in-memory hash table as an intermediary structure to store chunk fingerprints during ingestion. This intermediary allows fast O(1) lookup operations to detect duplicates without requiring disk I/O or network bandwidth, significantly reducing energy loss while achieving effective deduplication.
3Quantity of substance
If files are segmented into chunks for deduplication, then duplicate detection is improved, but processing time and computational overhead increase
Solution Approach 1:
The patent applies different processing strategies to different data characteristics. Frequently accessed or recently ingested data uses in-memory hash tables for O(1) lookup, while less frequent data may use persistent storage. This local quality approach optimizes processing time for hot data while maintaining deduplication effectiveness across the entire dataset.
Solution Approach 2:
The patent uses cryptographic hashing functions that transform variable-length input data into fixed-size hash values. This parameter change from variable to fixed size enables efficient storage and comparison of chunk identifiers, reducing processing time while maintaining high deduplication effectiveness through collision-resistant hash functions.
Data Source
Figure 1A
Figure 1B
Figure 2A~2B
AI summary
A data management device includes a persistent storage and a processor. The persistent storage includes an object storage. The processor segments a file into file segments. The processor generates meta-data of the file segments. The processor stores a portion of the file segments in a data object of the object storage. The processor stores a portion of the meta-data of the file segments in a meta-data object of the object storage.