Distributed Database Joiner Using Pluggable Sorting Index
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for sorting and joining data in distributed databases are inefficient, consuming large amounts of memory and processing resources, and often result in data duplication or alteration, leading to reliability issues and increased risk of data loss.
Innovation Solution
A system comprising a database server that uses a sorter to generate a pluggable sorting index for efficient data sorting and a joiner to join datasets in a stream-wise fashion, reducing memory consumption and processing costs while maintaining data integrity by avoiding duplication and alteration of input data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional sorting methods duplicate and alter data tables to sort and join data, then sorting and joining operations can be performed, but large amounts of memory and processing resources are consumed
Solution Approach 1:
The patent creates a sorted copy of the index rather than duplicating and altering the entire data table. The index structure is copied and sorted in memory, then used to efficiently retrieve sorted data without modifying the original tables or requiring full table duplication.
Solution Approach 2:
The patent applies sorting only to the index structure locally rather than sorting the entire data table. By sorting just the index (a small subset of the data), the system achieves efficient data access with minimal resource consumption.
2Productivity
If conventional methods alter input data to achieve desired sorted order, then sorting can be performed, but data reliability decreases and risk of data loss increases
Solution Approach 1:
The patent creates a copy of the index for sorting operations, leaving the original data tables unchanged. This ensures data integrity and reliability while still enabling efficient sorting and joining operations through the sorted index copy.
3Productivity
If large amounts of memory are allocated for joining datasets, then joining operations can be performed, but system resource efficiency decreases
Solution Approach 1:
The patent performs joining operations by sorting and comparing only the index structures locally in memory, rather than loading and processing entire datasets. This localized approach to joining requires minimal memory allocation while maintaining joining capability.
Data Source
AI summary
A joiner accesses a first sorted dataset and a second sorted dataset. Each dataset includes a corresponding plurality of data blocks, each including a set of records. Each record is associated with a corresponding record key. A set of first records for each first data block of the first dataset is arranged based on values of the first record keys. A set of second records for each second data block of the second dataset is arranged based on values of the second record keys. A first root element is extracted from the first sorted dataset. A second root element is extracted from the second sorted dataset. In response to determining that the first and second root elements match, an output is generated by joining the first record associated with the first root element with the second record associated with the second root element.