Scalable Data Storage Pools Metadata Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage pool management techniques are not scalable due to limitations in metadata storage, making it difficult to efficiently manage and distribute data across multiple devices in a data storage hierarchy.
Innovation Solution
The implementation of scalable data storage techniques that assign metadata distribution based on fault domains and operational characteristics of devices within the storage hierarchy, allowing for the addition and removal of data storage devices in parallel and rebalancing metadata according to resiliency constraints, thereby enhancing the pool's scalability and reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional data storage pool management techniques are used, then simplicity of implementation is maintained, but scalability is limited due to metadata storage constraints
Solution Approach 1:
The patent segments metadata management by introducing a hierarchical structure with metadata servers that manage metadata for specific groups of storage devices. This divides the monolithic metadata management task into smaller, manageable segments that can be independently scaled and maintained, resolving the contradiction between scalability and complexity.
Solution Approach 2:
The patent introduces a new dimensional layer in the storage architecture by adding metadata servers as an intermediate management layer between storage devices and clients. This dimensional change enables scalable metadata management without increasing the complexity of individual components, as the metadata server layer handles the complexity centrally.
2Ease of operation
If metadata is distributed across all storage devices, then data accessibility is improved, but system complexity and management overhead increase
Solution Approach 1:
The patent extracts metadata management functionality from individual storage devices and consolidates it into dedicated metadata servers. This extraction maintains data accessibility by keeping metadata centrally managed and readily available, while removing the complexity of metadata management from storage devices themselves.
Solution Approach 2:
The patent introduces metadata servers as intermediary components between storage devices and clients. These intermediaries handle metadata distribution and management, enabling easy data accessibility through centralized metadata lookup while preventing complexity from propagating to storage devices and clients.
3Reliability
If multiple metadata copies are maintained for reliability, then data reliability is improved, but storage overhead and management complexity increase
Solution Approach 1:
The patent applies local quality by distributing metadata copies strategically across different metadata servers and storage devices based on local requirements and fault domain considerations. This selective replication maintains reliability by ensuring metadata availability in appropriate locations while minimizing overall storage overhead through targeted rather than universal replication.
Data Source
AI summary
Scalable data storage techniques are described. In one or more implementations, data is obtained by one or more computing devices that describes fault domains in a storage hierarchy and available storage resources in a data storage pool. Operational characteristics are ascertained, by the one or more computing devices, of devices associated with the available storage resources within one or more levels of the storage hierarchy. Distribution of metadata is assigned by the one or more computing devices to one or more particular data storage devices within the data storage pool based on the described fault domains and the ascertained operational characteristics of devices within one or more levels of the storage hierarchy.