Distributed Database Table Allocation Engine
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed database systems, the execution of queries across multiple nodes leads to increased network overhead due to data transfer, and maximizing data locality can result in workload imbalance across nodes, where a few nodes handle a disproportionate number of queries while others remain idle.
Innovation Solution
An allocation engine identifies disjoint table sets based on frequently used queries and determines key performance indicators (KPIs) to split table sets across multiple nodes, balancing workload and minimizing data transfer by distributing tables across nodes to maintain KPIs within threshold values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is stored across multiple nodes in a distributed database system, then data distribution and scalability are improved, but network overhead increases due to data transfer between nodes
Solution Approach 1:
The patent segments tables into different subsets and distributes them across multiple nodes. By dividing the table set into manageable segments that can be independently allocated, the system achieves better data distribution while controlling network overhead through intelligent placement of related table segments on the same or nearby nodes.
Solution Approach 2:
The patent applies local quality by optimizing data placement at the node level. Tables that are frequently accessed together are placed on the same node or closely connected nodes, creating local data clusters that reduce cross-node data transfer and network overhead while maintaining overall distributed architecture benefits.
2Speed
If data locality is maximized by storing related tables on the same node, then query execution speed is improved, but workload becomes imbalanced across nodes
Solution Approach 1:
The patent implements dynamic table allocation where the system continuously monitors workload metrics and cross-KPI values, then adjusts table placement accordingly. This dynamic approach allows the system to maintain optimal data locality for query performance while automatically redistributing tables to balance workload across nodes when imbalances are detected.
Solution Approach 2:
The patent employs feedback mechanisms by calculating cross-KPI values that measure the impact of table placement on query performance and workload distribution. This feedback is used to iteratively optimize table allocation, ensuring that related tables are co-located for fast query execution while preventing any single node from becoming overloaded.
3Productivity
If tables are distributed across different nodes, then workload balance is improved, but data transfer between nodes increases
Solution Approach 1:
The patent segments the table set into subsets that are distributed across nodes to achieve workload balance. By carefully controlling which table segments are placed together and which are separated, the system minimizes unnecessary data transfer while maintaining balanced workload distribution across the distributed database nodes.
4Loss of energy
If cross-KPI threshold is set low to minimize data transfer, then network overhead is reduced, but workload balance deteriorates
Solution Approach 1:
The patent dynamically adjusts the cross-KPI threshold parameter based on system conditions and workload characteristics. By changing this parameter, the system can optimize the trade-off between minimizing data transfer and maintaining workload balance, allowing flexible adaptation to different operational scenarios and query patterns.
Data Source
AI summary
In one respect, there is provided a method. The method can include identifying, based on a plurality of queries executed at a distributed database, a disjoint table set. The identifying of the disjoint table set can include: identifying a first table that is used in executing a first query; identifying a second query that also uses the first table used in executing the first query; identifying a second table that is used in executing the second query but not in executing the first query; and including, in the disjoint table set, the first table and the second table. The method can further include allocating, based on the disjoint table set, a storage and/or management of the disjoint table set such that subsets of the disjoint table set are stored at and/or managed by different nodes in the distributed database system. Related methods and articles of manufacture are also disclosed.