Consistent Hashing Global Namespace Storage Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional storage systems face bottlenecks in scaling storage capacity without impacting performance, particularly due to metadata servers becoming a bottleneck in processing I/O requests, leading to latency issues.
Innovation Solution
Implementing a system that uses consistent hashing in combination with a global namespace layout and per-object object layouts, distributing object layouts across storage servers, eliminating the need for dedicated metadata servers, allowing for unlimited scaling without latency penalties.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If dedicated metadata servers are used to manage storage capacity, then storage capacity can be scaled, but the metadata servers become a bottleneck in processing I/O requests, leading to increased latency
Solution Approach 1:
The patent extracts the metadata storage function from dedicated metadata servers and distributes it across all storage servers in the pool. Each storage server maintains its own metadata locally, eliminating the centralized metadata server bottleneck while preserving the ability to scale storage capacity across the distributed pool.
Solution Approach 2:
Storage servers are given universal functionality to handle both data storage and metadata management. Each storage server can independently manage its own metadata and serve I/O requests without requiring centralized metadata server coordination, allowing the system to scale without performance degradation.
2Quantity of substance
If the number of storage servers is increased to scale storage capacity, then storage capacity increases, but the metadata servers become overwhelmed, impacting overall system performance
Solution Approach 1:
The centralized metadata server is segmented into distributed metadata storage across multiple storage servers. Each storage server maintains a portion of the metadata locally, allowing parallel processing of I/O requests without a single point of contention. This segmentation enables the system to handle increased I/O throughput as storage capacity scales.
3Quantity of substance
If traditional metadata server architecture is used, then storage capacity can be managed, but the system complexity increases with the need for dedicated metadata servers
Solution Approach 1:
The patent merges the metadata storage function with data storage by allowing storage servers to maintain their own metadata locally. This consolidation eliminates the need for separate dedicated metadata servers, reducing system architecture complexity while maintaining the ability to manage large storage capacities.
Data Source
Figure 1
Figure 2A~2B
Figure 2C
AI summary
A method for writing data to a storage pool. The method includes receiving a virtual identifier (ID) and an offset for an object, extracting a node identifier (ID) that identifies a first storage server in the storage pool from the virtual ID, obtaining an object layout (OL) for the object from the first storage server, hashing an object ID and an offset ID to obtain a hashed value, where the virtual ID comprises the object ID and where the offset ID is generated from the offset, identifying a second storage server in the storage pool using a global namespace layout (GL), the OL, and the hashed value, and issuing a write request to write data to the object in the second storage server, where the write request comprises the object ID and the offset.