Distributed Metadata Structures for Scalable Cluster Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face challenges in efficiently managing I/O access across a cluster of node apparatuses, particularly in distributing data and rebalancing loads while maintaining scalability and reliability, especially when adding or removing nodes.
Innovation Solution
A data storage system that uses metadata structures with a parent object and child objects, where metadata is distributed across multiple node apparatuses, allowing for efficient I/O access management through deterministic distribution algorithms and dynamic rebalancing of data segments across nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is distributed across multiple node apparatuses in a cluster system, then scalability and reliability are improved, but I/O access management complexity increases
Solution Approach 1:
The metadata of a data object is segmented into multiple metadata structures distributed across different node apparatuses. Each metadata structure contains a portion of the metadata and references to other metadata structures, enabling scalable I/O access management without centralizing control
Solution Approach 2:
Metadata structures act as intermediaries between I/O access requests and distributed data blocks. The metadata structures contain references (pointers) that enable indirect access to data blocks through a chain of metadata nodes, simplifying the access path across distributed nodes
2Ease of operation
If metadata structures are distributed across node apparatuses, then I/O access independence from data location is improved, but system complexity increases
Solution Approach 1:
Each metadata structure serves multiple functions: storing metadata for its associated data blocks, containing references to other metadata structures, and enabling I/O access regardless of which node receives the request. This multi-functionality reduces the need for separate location-tracking mechanisms
Solution Approach 2:
Metadata structures are nested in a hierarchical manner where metadata structures can contain references to other metadata structures (root metadata nodes, direct metadata nodes, indirect metadata nodes). This nesting enables indirect access paths that work independently of data location
3Productivity
If data is rebalanced across node apparatuses for load balancing, then system performance is improved, but data relocation overhead increases
Solution Approach 1:
The system enables dynamic rebalancing of data across node apparatuses by updating references in metadata structures. When data is moved between nodes, only the reference pointers in the distributed metadata structures need to be updated, not the entire metadata, enabling flexible load balancing with minimal overhead
Data Source
AI summary
In a distributed storage of data objects in a cluster system, a data object is distributed across a group of node apparatuses in units of data blocks. For each child object, the metadata structure of the respective child object is stored to one of the node apparatuses of the group of node apparatuses for managing locations of data blocks of the data object and includes a root metadata node and one or more direct metadata nodes, optionally further including one or more indirect metadata nodes; and the metadata structure of the parent object is distributed across the plural node apparatuses of the group of node apparatuses for managing locations of child objects of the data object and includes, on each node apparatus of the group of node apparatuses, a root metadata node and one or more direct metadata nodes, optionally further including one or more indirect metadata nodes.