Hypergraph Partitioning for Dynamic Shard Reassignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional approaches for allocating shards in distributed databases fail to account for changes in workload over time, leading to suboptimal performance and imbalance in workload patterns, particularly in OLTP scenarios where minimizing distributed queries is crucial for maximizing transaction throughput.
Innovation Solution
A workload-aware reassignment framework that uses hypergraph partitioning to dynamically reassign shards based on real-time workload data, splitting the workload into time slices and applying a sliding window approach to optimize shard placement and balance load across hosts, thereby reacting to changes in workload patterns and peaks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional graph orhypergraph partitioning is used to allocate shards to hosts, then the allocation can be optimized for a static workload model, but the system cannot adapt to changes in workload intensity or mixture over time
Solution Approach 1:
The patent implements dynamic shard allocation by continuously monitoring workload characteristics and re-running hypergraph partitioning algorithms when workload patterns change. The system transitions from static allocation to dynamic adaptation by detecting changes in workload intensity and mixture, and re-optimizing shard-to-host mappings accordingly to maintain high transaction throughput under varying conditions
Solution Approach 2:
The system incorporates feedback mechanisms by monitoring query workloads and detecting changes in workload patterns. Based on this feedback, the system dynamically adjusts shard allocation decisions, creating a closed-loop control system that adapts to changing conditions while maintaining optimal performance
2Productivity
If shards are placed to minimize distributed queries, then transaction throughput is improved, but load imbalance across hosts occurs when workload patterns change
Solution Approach 1:
The system dynamically balances the contradiction between minimizing distributed queries and maintaining load balance by continuously monitoring workload patterns and re-optimizing shard allocations. When workload patterns change, the system re-runs the hypergraph partitioning algorithm to find new allocations that maintain both low distributed query counts and balanced host loads under the new conditions
3Device complexity
If a single shard allocation is created for all workload patterns, then the allocation is simple to manage, but it becomes non-optimal for multiple different workload patterns over time
Solution Approach 1:
The patent segments the workload into different patterns and creates multiple specialized shard allocations, each optimized for specific workload characteristics. Instead of using a single allocation for all patterns, the system divides the allocation space into pattern-specific allocations, allowing each to be optimized independently for its target workload pattern
Solution Approach 2:
The system creates a universal framework that generates multiple allocations serving different workload patterns. Each allocation in the set serves a specific function for its target pattern, while the overall system provides multi-functionality by handling diverse workload types through the appropriate allocation selection
Data Source
AI summary
Methods, systems, and computer-readable storage media for receiving workload data, the workload data including queries executed within a distributed database system over a period of time, defining windows, each window including a time slice within the period of time, generating a hypergraph for each window, each hypergraph including vertices and hyperedges and being generated based on a sub-set of queries and weight functions, partitioning each hypergraph into blocks, for each shard in a set of shards, determining a set of ratings, each rating in the set of ratings being based on a weight of a respective share with respect to a respective block, and assigning each shard in the set of shards to a block in the set of blocks based on the set of ratings for the respective shard, the shard being assigned to a block, for which a maximum rating is provided in the set of ratings.