Hash-Based Partitioning for Distributed Storage Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing systems face challenges with unpredictable downtime and load distribution issues due to range-based partitioning, which can impact tenant applications and infrastructure, and existing solutions involve data replication during load balancing operations.
Innovation Solution
A hash-based partitioning system is introduced, built on a range-based architecture, providing dynamic load balancing without replication, using a hybrid range-hash storage model that supports key-value store functionality and allows for flexible load distribution by leveraging hash-based partitioning to manage IOPS and partitioning operations dynamically.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If range-based partitioning is used for load balancing, then dynamic load balancing capability is achieved, but unpredictable downtime and load distribution issues occur
Solution Approach 1:
The patent changes the partitioning parameter from range-based to hash-based. This parameter change eliminates the unpredictable downtime associated with range-based partitioning while maintaining dynamic load balancing capability. The hash-based approach provides deterministic partition assignment based on partition key hashes, ensuring stable and predictable system behavior.
2Adaptability or versatility
If range-based partitioning is used, then partitioning flexibility is provided, but load distribution issues directly impact tenant applications
Solution Approach 1:
The patent changes the partitioning parameter from range-based to hash-based, which fundamentally alters the load distribution characteristics. Hash-based partitioning provides more uniform and predictable load distribution across nodes, eliminating the load distribution issues that directly impact tenant applications while retaining partitioning flexibility through hash function selection and partition key configuration.
3Productivity
If conventional hash-based load balancing is used, then load distribution is improved, but data replication is required
Solution Approach 1:
The patent extracts the data replication requirement from the load balancing process. By implementing hash-based partitioning at the storage layer with a catalog that maps partition keys to nodes, the system achieves efficient load distribution without requiring data replication during load balancing operations. The catalog serves as a metadata layer that enables direct routing without copying data.
Data Source
AI summary
In various embodiments, methods and systems for implementing hash-based partitioning in distributed computing systems are provided. At a high level, a distributed computing system having an underlying range-based partitioning architecture for storage may be configured as a hash-based partitioning system, for example, a hybrid range-hash table storage. An operations engine of the hash-based partitioning system receives a tenant request to provision input/output operations per second (IOPS). The tenant request comprises a requested number of IOPS. Based on the tenant request, a provisioning operation to provision IOPS in a hybrid range-hash table storage with hash-based partitioning is determined. The provisioning operation is selected from one of the following: a table creation provisioning operation, an IOPS increase provisioning operation, and an IOPS decrease provisioning operation. The selected provisioning operation is executed for a corresponding table. A user request for data is processed using the table associated with the requested number of IOPS.