Secondary Hashing for Balanced Data Partition Redistribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed systems face performance issues due to imbalanced data distribution across partitions, leading to delayed access and migration challenges, as some partitions become overburdened while others remain underutilized, necessitating techniques to optimize data redistribution without impacting user access.
Innovation Solution
Implementing a secondary hashing scheme that generates modified key values to redistribute data sets across partitions, ensuring a balanced distribution by applying a different hash function or modifying key values to achieve optimal partitioning, allowing incremental migration and maintaining mappings for seamless access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is redistributed across partitions to balance load, then resource utilization is improved, but user access performance deteriorates due to migration blocking
Solution Approach 1:
The system pre-computes target partition locations using secondary hash values before actual data migration begins. This preliminary calculation enables seamless redirection of access requests to new locations without interruption, resolving the contradiction by preparing migration paths in advance so that user access continues uninterrupted while redistribution occurs
Solution Approach 2:
The patent introduces a request routing component that acts as an intermediary between users and data partitions. This mediator transparently redirects access requests from old partition locations to new ones using pre-computed secondary hash values, allowing data migration to proceed without blocking user access and thus resolving the performance contradiction
2Stability of the object's composition
If data migration is performed to rebalance partitions, then distribution balance is improved, but system availability deteriorates due to access blocking
Solution Approach 1:
The system performs preliminary computation of secondary hash values and target partition mappings before migration executes. This advance preparation ensures that when data moves between partitions, the routing infrastructure already knows where to direct requests, maintaining system availability while achieving better distribution balance
Solution Approach 2:
The patent maintains multiple copies of routing information (primary and secondary hash mappings) to ensure continuity during migration. By having redundant routing paths pre-computed and stored, the system can switch between old and new partition locations without interruption, preserving availability while improving distribution balance
Data Source
AI summary
Items within a data set can be redistributed to different partitions of a distributed data store according to modified hash values. A redistribution event may, in some embodiments, trigger the redistribution of a data set. Modified hash values for items in the data set may be generated according to a secondary hashing scheme to be used for identifying the location of a new partition to store the item instead of hash values based on current key values that identify a current partition that stores the item. Access can be provided to items based on the modified hash values to determine the new partition that stores the item after redistribution.