Separated Metadata Container Format for File Segment Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage methods are inefficient in reducing storage requirements, as they do not effectively deduplicate data before storage, leading to unnecessary storage space usage.
Innovation Solution
A data management system that segments files into deduplicated file segments and stores them in data objects, while storing meta-data in separate meta-data objects, utilizing object storage architecture to reduce storage needs by identifying and removing duplicate segments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If files are stored without segmentation and deduplication, then storage operations are simple, but storage space is wasted due to duplicate data
Solution Approach 1:
The patent divides files into fixed-size segments (e.g., 4KB each) before storage. Each segment is treated as an independent unit that can be deduplicated separately. This segmentation enables efficient identification and elimination of duplicate data portions while maintaining simple storage operations for unique segments.
Solution Approach 2:
The system performs deduplication operations before final storage by comparing segment hashes against existing segments. This preliminary action identifies duplicates early in the process, preventing wasted storage space from the beginning rather than requiring later cleanup operations.
2Quantity of substance
If all file segments are stored in a single data object, then data retrieval is straightforward, but storage efficiency is reduced due to inability to deduplicate across files
Solution Approach 1:
The patent segments files into uniform units and stores them in a centralized segment storage area where deduplication can occur across different files. This allows the system to recognize that identical segments from different files represent the same data, thereby improving storage efficiency while maintaining organized retrieval through segment identifiers.
Solution Approach 2:
The segment storage structure serves multiple functions: it stores actual data segments, maintains deduplication indexes through hashing, and enables cross-file deduplication. This multi-functional approach improves storage efficiency without proportionally increasing complexity.
3Quantity of substance
If deduplication is performed on entire files, then storage space is optimized, but processing time increases significantly
Solution Approach 1:
By dividing files into smaller segments, the system reduces the computational burden of deduplication. Instead of hashing and comparing entire large files, the system processes smaller segment units independently and in parallel, significantly reducing processing time while achieving the same storage optimization through identification of duplicate segments.
Solution Approach 2:
The system performs deduplication on segments rather than complete files, applying partial action to achieve sufficient storage optimization. This approach processes only the necessary granular units needed for effective deduplication without the overhead of handling entire files, balancing processing time with storage space reduction.
Data Source
AI summary
A data management device includes a persistent storage and a processor. The persistent storage includes an object storage. The processor segments a file into file segments. The processor generates meta-data of the file segments. The processor stores a portion of the file segments in a data object of the object storage. The processor stores a portion of the meta-data of the file segments in a meta-data object of the object storage.