Fan-Out Partitioning for Distributed Database Range Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed database management systems face inefficiencies in range queries due to random distribution of items across computing nodes, leading to hotspots and uneven workload distribution, which affects performance and scalability.
Innovation Solution
Implementing a fan-out system that assigns items to partitions based on a composite primary key, using a partition function to distribute write operations and a fan-out group selection function for range queries, along with metadata catalogs and predictive models to dynamically adjust partitioning and replication, thereby balancing workload and improving query efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If items are randomly distributed across computing nodes, then write operations are evenly distributed, but range queries become inefficient
Solution Approach 1:
The system segments the key space into multiple partitions and uses a fan-out group structure where each partition is served by a primary node and multiple replica nodes. This segmentation allows write operations to be distributed across multiple nodes while keeping items with the same hash key co-located for efficient range queries.
Solution Approach 2:
The patent implements local quality by ensuring that items with the same hash key are co-located on the same computing node (primary or replica). This local grouping optimizes range query performance by allowing queries to be executed on a single node rather than scanning across multiple distributed nodes.
2Loss of time
If items are non-randomly distributed to improve range queries, then hotspots are created with concentrated workload
Solution Approach 1:
The patent merges the benefits of random distribution and non-random distribution by combining a hash-based primary key (for even distribution) with a range key (for efficient range queries). The fan-out group structure merges multiple replica nodes to handle write operations while maintaining co-location for range queries.
Solution Approach 2:
The system dynamically adjusts the fan-out group configuration based on workload patterns. The primary key hash value determines the partition, while the range key enables dynamic range queries. This dynamic structure allows the system to adapt to different query patterns without creating persistent hotspots.
3Loss of time
If items are co-located by hash key for range queries, then write operations concentrate on specific nodes
Solution Approach 1:
The patent uses replication by creating multiple copies (replicas) of each partition. Each partition has one primary node and multiple replica nodes. Write operations are distributed across all replicas, while range queries can be executed on any replica, eliminating write hotspots while maintaining range query efficiency.
Solution Approach 2:
The system adds a replica dimension to the partition structure. Instead of a single node per partition, each partition exists across multiple nodes in parallel. This dimensional expansion allows write operations to be distributed across the replica dimension while maintaining data co-location for range queries.
Data Source
AI summary
Distributed database management systems may perform range queries over the leading portion of a primary key. Non-random distribution of data may improve performance related to the processing of range queries, but may tend to cause workload to be concentrated on particular partitions. Groups of partitions may be expanded and collapsed based on detection of disproportionate workload. Disproportionate write workload may be distributed among a group of partitions that can subsequently be queried using a federated approach. Disproportionate read workload may be distributed among a group of read-only replicated partitions.