Hierarchical Metadata for Distributed Data De-duplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in efficiently managing I/O access and de-duplication across multiple node apparatuses, particularly in scaling out and rebalancing data distribution while maintaining data location independence.
Innovation Solution
A data storage system utilizing metadata structures with root, direct, and indirect metadata nodes to manage de-duplicated data blocks and handle I/O requests, including a method to determine duplicate data blocks and update metadata references, and a distribution algorithm for data segments across node apparatuses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is distributed across multiple node apparatuses in a cluster system, then scalability and storage capacity are improved, but managing I/O access and performing de-duplication efficiently becomes more complex
Solution Approach 1:
The metadata structure is segmented into multiple levels (root metadata node, direct metadata nodes, indirect metadata nodes) that are distributed across different node apparatuses. Each segment manages specific data blocks, allowing independent processing and reducing the complexity of global I/O access management while maintaining de-duplication capabilities across the entire cluster.
2Adaptability or versatility
If data is distributed across multiple node apparatuses, then scalability is improved, but maintaining data location independence and efficient de-duplication becomes more difficult
Solution Approach 1:
Metadata nodes serve as intermediaries between client I/O requests and the actual distributed data blocks. The metadata structure maintains references to data block locations across nodes, enabling de-duplication operations to be performed through the metadata layer without requiring direct knowledge of physical data locations, thus maintaining data location independence while enabling efficient de-duplication.
3Ease of operation
If metadata structures are used to manage I/O access to distributed data objects, then I/O access control and management are improved, but the system complexity and overhead increase
Solution Approach 1:
The metadata structure is divided into hierarchical levels (root, direct, indirect nodes) that can be independently managed and cached. This segmentation allows the system to load only necessary metadata portions into memory, reducing overhead while maintaining comprehensive I/O access control. Each metadata node manages a specific subset of data blocks, simplifying the overall management complexity.
Data Source
AI summary
A data de-duplication in a distributed storage of data objects in a cluster system, in which plural data objects are distributed across a group of node apparatuses and stored in units of data blocks. Each metadata structure including a root metadata node and one or more direct metadata nodes, and optionally including one or more indirect metadata nodes; and a metadata object is stored for managing de-duplicated data blocks based on a metadata structure of the metadata object wherein at least one direct metadata node of the metadata structure of the metadata object includes a block reference pointing to a de-duplicated data block being associated with two or more data objects. Preferably, each of the metadata structures of the two or more data objects being associated with the de-duplicated data block includes a respective direct metadata node including an object reference to the metadata structure of the metadata object.