Distributed Metadata Server Architecture for Scalable Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional filesystems face scalability issues due to limited metadata server numbers, leading to inefficiencies in handling large loads and failures, especially in distributed storage systems.
Innovation Solution
The proposed solution involves a distributed filesystem architecture that effectively spreads metadata functionality across many servers, utilizing non-volatile memory and distributed erasure coding to manage data and metadata, allowing for load distribution and efficient scaling to thousands of nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If traditional centralized metadata servers are used, then system simplicity is maintained, but scalability and fault tolerance deteriorate when cluster size increases
Solution Approach 1:
The patent segments the centralized metadata server functionality into distributed metadata server instances across multiple nodes. Each metadata server instance handles a specific subset of metadata operations, allowing the system to scale horizontally while maintaining fault tolerance through distribution. This resolves the contradiction by transforming the single-point-of-failure architecture into a resilient distributed architecture.
Solution Approach 2:
The patent introduces a new dimensional approach by organizing metadata servers in a hierarchical structure with primary and secondary instances across different nodes. This multi-dimensional arrangement allows the system to maintain simplicity at the application level while achieving scalability and fault tolerance through the underlying distributed architecture.
2Productivity
If more metadata servers are added to handle large loads, then system capacity increases, but system complexity and difficulty of management increase
Solution Approach 1:
The patent implements metadata servers that can function as both primary and secondary instances depending on their operational state. Each metadata server instance is designed to be multi-functional, capable of handling read operations, write operations, and failover operations. This universality allows the system to scale capacity without proportionally increasing management complexity, as the same component type performs multiple roles.
3Reliability
If metadata functionality is distributed across many servers, then scalability improves, but system complexity increases
Solution Approach 1:
The patent employs homogeneous metadata server instances across the distributed system, where each instance implements the same core functionality and interface. This homogeneity allows for simplified management and operation, as the same component type can be deployed, monitored, and maintained uniformly across all nodes, thereby reducing the complexity that would otherwise arise from heterogeneous distributed components.
4Ease of operation
If centralized metadata control is used, then ease of operation is maintained, but scalability to thousands of nodes is limited
Solution Approach 1:
The patent introduces client-side metadata caching as an intermediary layer between the distributed metadata servers and clients. This caching mechanism abstracts the complexity of the distributed architecture from clients, allowing them to access metadata with the same simplicity as in centralized systems while the backend distributed infrastructure handles scalability and fault tolerance.
Data Source
AI summary
A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more of a plurality of storage devices. A plurality of failure resilient stripes is distributed across the plurality of storage devices such that each of the plurality of failure resilient stripes spans a plurality of the storage devices. The plurality of computing devices maintains each failure resilient stripe using a bucket. These buckets are operably split and transferred between the computing devices to balance available computing power and storage access.