Distributed Database Data Loading with Dual-Key Sorting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current information technology systems fail to optimally configure and distribute large volumes of data among resources for rapid searching, leading to inefficient data storage and retrieval in high-volume database systems.
Innovation Solution
A method where a computer processor selects and sorts data records using two keys to generate candidate files, determining the most efficient distribution by counting groups, and loads the candidate file with the greater number of groups into a database management system, ensuring even data distribution across resources for improved search efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If data is loaded into a distributed database using conventional methods, then the loading process is simple, but the data distribution becomes uneven across storage resources
Solution Approach 1:
The patent applies preliminary action by sorting data records by key values before loading them into the distributed database. The system generates candidate files with pre-sorted data and pre-calculated group counts, selecting the optimal candidate file before the actual loading process. This pre-processing ensures even data distribution across storage resources without adding complexity during the loading phase itself.
2Productivity
If data is distributed unevenly across storage resources, then the loading process is faster, but the searching efficiency decreases
Solution Approach 1:
The system applies self-service by automatically evaluating multiple candidate files and selecting the optimal one based on group count metrics. The database management system autonomously determines which candidate file will provide the most even data distribution, eliminating the need for manual configuration or external intervention. This self-optimizing approach ensures high searching efficiency without requiring additional time for manual data distribution tuning.
3Manufacturing precision
If multiple candidate files are generated and evaluated, then the data distribution optimizes, but the processing time increases
Solution Approach 1:
The patent applies partial action by generating multiple candidate files with different sorting configurations, but only evaluating them based on the critical metric of group count. The system generates sufficient candidate files to ensure optimal data distribution, then selectively evaluates them using a streamlined process that counts groups without performing exhaustive analysis. This approach achieves high data distribution quality while limiting processing time through focused evaluation criteria.
Data Source
AI summary
Presented is a method by which large amounts of data may be sorted and downloaded. In the presented invention, two keys are selected which may be associated with all records in a designated distributed, relational, object-oriented, graph or other database, and the records are sorted first by means of the first and second keys to generate a first file, and subsequently by means of the second and first keys to generate a second file. It is determined which of the resultant files contains a larger number of groups, and the file containing the larger number of groups is selected for loading, preferably by means of a plurality of data load streams. Further presented is means by which the data load streams may be executed in parallel with one another while minimizing contention for shared resources. The method additionally helps to increase the amount of parallelism that may be deployed in support of database search operations.


