Distributed Directory Hashing for Scalable Storage Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face challenges in efficiently managing large-scale distributed storage environments, particularly in handling metadata distribution and directory structures, leading to performance bottlenecks and inefficiencies in load balancing across multiple nodes.
Innovation Solution
A distributed storage system utilizing a directory structure with on-disk hashing (ODH) and distributed failure-resilient address spaces (DFRAS) that allows for independent operation of buckets, enabling load distribution and high-performance parallel commits across thousands of nodes without central coordination, combined with a scalable directory structure implemented using ODH to handle metadata across multiple servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If metadata is centralized in conventional storage systems, then management is simplified, but scalability and performance deteriorate due to bottlenecks in large clusters
Solution Approach 1:
The patent divides the centralized metadata namespace into distributed directory slices, with each slice independently managed by different nodes in the cluster. This segmentation eliminates the single-point bottleneck while maintaining manageable complexity through modular organization of metadata across multiple nodes.
2Productivity
If directory operations are distributed across multiple nodes, then scalability improves, but coordination complexity increases
Solution Approach 1:
Each node autonomously manages its own directory slices without requiring centralized coordination. Nodes independently perform directory operations on their assigned slices, eliminating complex inter-node coordination while achieving horizontal scalability across the cluster.
3Ease of operation
If load is concentrated on individual nodes, then operation simplicity is maintained, but performance degrades under high workload
Solution Approach 1:
The patent segments the directory namespace into multiple slices distributed across different nodes, allowing workload to be parallelized across the cluster. Each node handles a portion of the total directory operations, increasing overall throughput while maintaining simple operation interfaces through the unified VFS layer.
Data Source
AI summary
A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more flash storage devices. Each computing device is operable to access one or more memory blocks within the flash storage devices and maintain a directory structure for managing access to the memory. The directory structure may be adaptively resized according to the addition or removal of one or more associated files stored in memory.