Pluggable Sorting Index for Distributed Database Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for sorting and joining data in distributed databases require significant memory and processing resources, often leading to data duplication and alteration, which can result in data loss and decreased reliability.
Innovation Solution
A system that includes a database server capable of generating pluggable sorting indices for efficient sorting and joining of datasets, allowing for sorting and joining operations without duplicating or altering the input data, using a sorter to arrange records based on sortable properties and a joiner to generate output as a data stream.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional sorting methods are used to sort and join data tables, then the data can be sorted and joined as desired, but large amounts of memory and processing resources are consumed
Solution Approach 1:
The patent introduces an intermediary indexing structure that mediates between the raw data and the sorting/joining operations. Instead of directly manipulating data tables, the system creates indexes that serve as intermediaries to enable efficient sorting and joining without requiring large amounts of memory to hold duplicate data structures.
Solution Approach 2:
The patent creates lightweight index copies rather than full data table copies. These indexes contain only the necessary sorting keys and pointers to original data, dramatically reducing memory consumption compared to conventional approaches that duplicate entire data tables for sorting operations.
2Productivity
If data tables are duplicated and altered to achieve sorting, then the desired sorted order is achieved, but data reliability decreases and data loss risk increases
Solution Approach 1:
The index structure serves as an intermediary that provides sorting capability without directly altering the original data tables. The indexes reference the original data without modifying it, ensuring data integrity while enabling efficient sorting operations through the intermediate indexing layer.
Solution Approach 2:
The patent creates read-only index copies that reference original data without allowing modifications to the source tables. These index copies enable sorting operations while the original data remains unchanged and protected, preventing data loss and maintaining reliability.
3Productivity
If large amounts of memory are allocated for joining datasets, then the joining operation can be performed, but resource consumption increases significantly
Solution Approach 1:
The patent uses indexes as intermediary structures that enable joining operations without requiring large memory allocations. The indexes act as compact representations that facilitate efficient matching and joining of datasets from multiple sources without holding entire datasets in memory simultaneously.
Solution Approach 2:
The patent extracts only the essential joining keys and necessary data portions into index structures, leaving the bulk of the data on disk or in distributed storage. This extraction approach enables joining operations with minimal memory allocation while maintaining the ability to access and process large datasets.
Data Source
AI summary
A database server is configured to receive input data which includes a data table. Each record of the table includes a record key and one or more entries. Each of the one or more entries is associated with a corresponding column key of the data table. Instructions are received for sorting the input data according to the column key. The instructions indicate an order in which to sort the input data. Based on the order indicated by the instructions, a pluggable sorting index associated with a column key of the data table is generated. The pluggable sorting index includes the record keys arranged in a sorted order according to sortable properties of entries associated with the column key. A sorted dataset which includes the entries associated with the column key arranged according to the sorted order of the pluggable sorting index is generating using the pluggable sorting index.