Hash-Based Traffic Distribution for Storage Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As storage systems scale, communication bottlenecks and data throughput limitations arise when connecting newer storage system architectures to legacy networks, necessitating effective load-balancing solutions to enhance data storage and retrieval efficiency.
Innovation Solution
The implementation of a storage system architecture that includes multiple storage nodes with non-volatile solid state storage units, erasure coding, and a distributed authority system for data redundancy and load-balancing, utilizing hash algorithms for efficient data distribution across multiple storage nodes and switches to manage data throughput and availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If storage systems scale with more storage nodes and network connections, then storage capacity and data redundancy improve, but communication bottlenecks and data throughput limitations worsen
Solution Approach 1:
The patent segments data into multiple shards distributed across different storage nodes using hash algorithms. This segmentation allows parallel data access and transmission across multiple network paths, thereby maintaining data throughput as storage capacity scales by adding more nodes.
Solution Approach 2:
The patent introduces a distributed hash table dimension that maps data keys to storage nodes, creating an additional organizational layer. This dimensional approach enables efficient data routing and load distribution across the network, preventing communication bottlenecks even as the system scales to numerous nodes.
2Adaptability or versatility
If newer storage system architectures are connected to legacy networks, then storage modernization and scalability improve, but communication bottlenecks worsen
Solution Approach 1:
The patent implements a distributed hash table as an intermediary layer between legacy networks and modern storage nodes. This intermediary abstracts the complexity of distributed storage from legacy systems while enabling efficient data routing, allowing modern storage architectures to integrate with legacy networks without suffering from communication bottlenecks.
Solution Approach 2:
The patent dynamically adjusts hash algorithm parameters and data sharding configurations to optimize network utilization. By changing parameters such as hash function selection and shard distribution, the system adapts to legacy network constraints while maintaining modern storage scalability and throughput performance.
3Reliability
If data is distributed across multiple storage nodes, then load-balancing and availability improve, but system complexity increases
Solution Approach 1:
The patent implements self-service mechanisms where each storage node independently computes data placement using distributed hash algorithms. Nodes autonomously determine where to store and retrieve data based on hash key calculations, eliminating the need for centralized coordination and reducing system complexity while maintaining high availability through distributed redundancy.
4Productivity
If hash algorithms are used for data distribution, then load-balancing efficiency improves, but computational overhead increases
Solution Approach 1:
The patent applies partial hashing where only necessary portions of data keys are processed through hash algorithms for data placement decisions. This partial action approach achieves sufficient load-balancing efficiency for distributed storage while minimizing unnecessary computational overhead compared to full hashing of all data elements.
Data Source
AI summary
A storage system determines source addresses, and destination addresses in a storage system, for network traffic. The storage system determines a hash algorithm, from a plurality of hash algorithms. The hash algorithm is to be used across the source addresses for load-balancing the network traffic to the destination addresses. The storage system determines that the hash algorithm more closely meets one or more load-balancing criteria than at least one other hash algorithm, of the plurality of hash algorithms. The storage system distributes the network traffic from the source addresses to the destination addresses in the storage system, with load-balancing according to the determined hash algorithm.