Disjoint Table Sets for Distributed Database Workload Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed database systems, the execution of queries across multiple nodes leads to network overhead due to data transfer, and maximizing data locality can result in workload imbalance across nodes, where a few nodes handle a disproportionate number of queries while others remain idle.
Innovation Solution
An allocation engine identifies disjoint table sets based on query frequency and usage patterns, determining key performance indicators like execution time and count, and splits these sets across multiple nodes to balance workload and minimize data transfer, ensuring no node is excessively loaded or idle.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored across multiple nodes in a distributed database system, then data distribution and fault tolerance are improved, but network overhead and data transfer requirements increase
Solution Approach 1:
The system segments tables into disjoint table sets based on query access patterns. Tables that are frequently accessed together are grouped into the same set and co-located on the same node, while tables accessed independently are placed in separate sets. This segmentation strategy reduces cross-node data transfers by keeping related data local, thereby decreasing network overhead while preserving the distributed storage architecture's fault tolerance benefits.
2Productivity
If data locality is maximized by storing related tables on the same node, then query execution efficiency is improved, but workload imbalance occurs across nodes
Solution Approach 1:
The system applies local quality by creating distinct storage policies for different table sets based on their specific access patterns. High-frequency tables that benefit from locality are co-located on nodes optimized for fast access, while lower-frequency tables are distributed to balance the workload. The allocation engine assigns table sets to nodes based on both locality considerations and workload balancing metrics, ensuring that no single node becomes a bottleneck while still maintaining efficient query execution for frequently accessed data.
3Adaptability or versatility
If tables are distributed across multiple nodes, then system scalability is improved, but query execution complexity increases
Solution Approach 1:
The allocation engine performs preliminary action by pre-analyzing query workloads and determining optimal table set assignments before actual query execution. It identifies access patterns, groups related tables into disjoint sets, and pre-allocates these sets to specific nodes based on predicted workload distribution. This upfront planning reduces the complexity of query execution at runtime, as the system no longer needs to dynamically determine data locations or perform complex coordination across nodes for each query.
Data Source
AI summary
In one respect, there is provided a method. The method can include identifying, based on a plurality of queries executed at a distributed database, a disjoint table set. The identifying of the disjoint table set can include: identifying a first table used in executing a first query; identifying a second query also using the first table used in executing the first query; identifying a second table used in executing the second query but not in executing the first query; and including, in the disjoint table set, the first table and the second table. The method can further include allocating, based at least on the first disjoint table set, a storage and/or management of the first disjoint table set such that the first disjoint table set is stored at and/or managed by at least one node in the distributed database. Related systems and articles of manufacture are also disclosed.