Flash Registry On-Disk Hashing for Distributed Metadata Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face challenges in efficiently managing large-scale data storage and retrieval, particularly in distributed environments, leading to performance bottlenecks and inefficiencies in load balancing and metadata management.
Innovation Solution
A flash registry system utilizing on-disk hashing (ODH) is implemented, which splits and merges blocks efficiently, allowing for distributed failure-resilient address spaces (DFRAS) and dynamic load balancing across multiple nodes, using cryptographic hashing for uniform data distribution and minimizing coordination overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional data storage systems are used for large-scale data storage and retrieval, then data can be stored, but performance bottlenecks occur and load balancing becomes inefficient
Solution Approach 1:
The patent segments the storage system into multiple independent nodes, each maintaining its own registry and hash table. This segmentation allows parallel operations across nodes, eliminating the single point of bottleneck while distributing the computational load. Each node operates autonomously with local metadata, enabling simultaneous data storage and retrieval operations without interfering with other nodes.
2Productivity
If centralized coordination is used for load balancing, then load distribution can be achieved, but coordination overhead increases and scalability decreases
Solution Approach 1:
Each node in the distributed system autonomously manages its own load balancing and registry maintenance without requiring centralized coordination. Nodes independently compute hash values for data blocks, determine their own storage locations, and maintain local metadata. This self-service approach eliminates coordination overhead and allows the system to scale horizontally by simply adding more independent nodes.
3Ease of operation
If metadata is centralized in a single registry, then data management is simplified, but performance bottlenecks and single points of failure occur
Solution Approach 1:
The centralized registry is segmented into multiple distributed registries, one per node. Each registry manages metadata for data blocks stored on its local node, eliminating the single point of failure. The segmentation maintains operational simplicity through consistent hash-based lookup mechanisms while distributing reliability across multiple independent registries. Data blocks are routed to the appropriate node registry using hash functions, preserving ease of operation.
4Productivity
If cryptographic hashing is used for uniform data distribution, then load balancing improves, but computational overhead increases
Solution Approach 1:
The system applies cryptographic hashing selectively at critical decision points rather than continuously. Hash functions are computed only when determining data block locations, resolving conflicts in the hash table, or routing data to nodes. This localized application of hashing achieves uniform data distribution while minimizing computational overhead by avoiding unnecessary hash computations during routine operations.
Data Source
AI summary
A plurality of computing devices are communicatively coupled to each other via a network, and each of the plurality of computing devices is operably coupled to one or more of a plurality of storage devices. Each computing device is operable to access one or more memory blocks within the storage devices and maintain a registry over the same one or more memory blocks. The registry may be adaptively resized according to the access of the one or more memory blocks.