Database Permutation Sorting for Large Scale Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large database systems face performance penalties and increased computational costs as data sets grow, making it challenging to efficiently sort and query large amounts of data, especially when indexing and sorting become impractical due to the vast number of rows requiring significant computational resources.
Innovation Solution
The implementation of a method that uses permutations to impose a sorted order on database tables, allowing for incremental or on-demand sorting of subsets of columns without using traditional sorting algorithms, thereby reducing computational costs and enabling efficient sorting of large datasets by rearranging rows in linear time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional sorting algorithms are used on large database tables, then the data can be sorted, but the computational cost and processing time increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing permutation data that represents the sorted order of rows. Instead of executing sorting algorithms at query time, the sorting permutation is calculated in advance and stored in the database, allowing instant retrieval of sorted data without repeated computational overhead
Solution Approach 2:
The patent creates a copy of the sorting permutation data and stores it separately from the original data. This permutation copy can be applied to multiple queries without re-computing the sort order, significantly reducing processing time for repeated access patterns
2Productivity
If traditional sorting algorithms are used on large datasets, then the data can be sorted, but memory usage increases significantly
Solution Approach 1:
The patent extracts the essential sorting information (permutation data) from the full dataset and stores only this compact representation. The permutation data structure requires significantly less memory than storing complete sorted copies of large datasets, enabling sorting of billions of rows with limited memory resources
3Ease of operation
If the entire database table is sorted upfront, then all columns are sorted, but the up-front computational cost is high
Solution Approach 1:
The patent applies partial action by sorting only the necessary columns and storing permutation data selectively. Instead of forcing a complete table sort, the system computes and stores permutations only for columns that require sorting, reducing up-front computational cost while maintaining ease of access for commonly queried columns
Solution Approach 2:
The patent segments the sorting process by column, allowing different permutation data to be stored and applied to different columns independently. This enables selective sorting where only specific columns are pre-computed and stored, rather than sorting the entire table structure
Data Source
AI summary
A system such as a database management system determines a rule to impose sorted order upon a column without applying a sorting algorithm to the column and stores the rule to be applied to sort the column. In an embodiment, a permutation is the rule. The column, in an embodiment, is a column of a database table. In some cases, the rule is used to efficiently impose a correct sorted order on a subset of unsorted columns without the use of a sorting algorithm. In some cases, the rule is generated based at least in part on another column that is sorted using a sorting algorithm such as quicksort.