Distributed Database Partition Boundaries for Related Data Locality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed database partitioning techniques fail to account for specific relationships between data, leading to increased latency when querying the database due to separation of child table data from related parent table data during horizontal partitioning.
Innovation Solution
Embedding horizontal partition boundaries in data tables to guide the distribution of related data, ensuring that parent and child data remain on the same node, thereby preventing splits and reducing latency during queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If horizontal partitioning is performed without considering data relationships, then database scalability is improved, but query latency increases due to separation of parent and child data
Solution Approach 1:
The patent applies preliminary action by embedding partition boundary indicators in parent table data before the actual partitioning occurs. This allows the distributed database system to pre-identify which child table data must remain with which parent table data, ensuring that related data stays together during subsequent partitioning operations without requiring complex real-time coordination during queries.
Solution Approach 2:
The patent introduces an intermediary mechanism - the partition boundary indicator - that acts as a mediator between parent and child table data. This indicator serves as a marker that guides the partitioning process to maintain data relationships, allowing the system to achieve both scalability through partitioning and low latency by preserving data locality through the intermediary indicator.
2Adaptability or versatility
If distributed database partitioning separates child table data from parent table data, then data distribution across nodes is improved, but data locality is worsened causing increased query latency
Solution Approach 1:
The patent applies local quality by making each parent table data record have a unique characteristic - the embedded partition boundary indicator - that specifies its local data requirements. This allows different portions of the database to have different partitioning behaviors, with related parent-child data pairs maintaining their locality while unrelated data can be distributed independently, achieving both good data distribution and data locality.
3Speed
If related data is kept on the same node, then query performance is improved, but database scalability is limited
Solution Approach 1:
The patent applies segmentation by dividing the database into independently manageable segments (nodes) while using the partition boundary indicator to ensure that related data segments remain co-located. This allows the database to be segmented for scalability while maintaining the performance benefit of data locality, as the segmentation is guided by the embedded indicators rather than being purely arbitrary.
Data Source
AI summary
A method includes receiving, by a security analytics platform, first data associated with a computing resource, storing the first data in a first database table associated with the computing resource, and generating a first set of indicators associated with the first database table. Each indicator of the first set of indicators identifies a corresponding horizontal partition associated with the first database table. The method further includes receiving second data associated with the computing resource, storing the second data in a second database table associated with the first database table, and generating a second set of indicators associated with the second database table. The method further includes storing, based on the first and second set of indicators, a first partition of the first database table and a corresponding partition of the second database table, on a same database node.