Distributed Database Data Loading with Dual-Key Sorting

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Current information technology systems fail to optimally configure and distribute large volumes of data among resources for rapid searching, leading to inefficient data storage and retrieval in high-volume database systems.

Innovation Solution

A method where a computer processor selects and sorts data records using two keys to generate candidate files, determining the most efficient distribution by counting groups, and loads the candidate file with the greater number of groups into a database management system, ensuring even data distribution across resources for improved search efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If data is loaded into a distributed database using conventional methods, then the loading process is simple, but the data distribution becomes uneven across storage resources

Engineering Contradiction:
Improvedata distribution evennessVSAvoidloading process complexity
Core Design Contradiction:
Manufacturing precisionVSDevice complexity

Solution Approach 1:

The patent applies preliminary action by sorting data records by key values before loading them into the distributed database. The system generates candidate files with pre-sorted data and pre-calculated group counts, selecting the optimal candidate file before the actual loading process. This pre-processing ensures even data distribution across storage resources without adding complexity during the loading phase itself.

Inventive Principle:
Principle #10Preliminary action

2Productivity

If data is distributed unevenly across storage resources, then the loading process is faster, but the searching efficiency decreases

Engineering Contradiction:
Improvesearching efficiencyVSAvoiddata loading time
Core Design Contradiction:
ProductivityVSLoss of time

Solution Approach 1:

The system applies self-service by automatically evaluating multiple candidate files and selecting the optimal one based on group count metrics. The database management system autonomously determines which candidate file will provide the most even data distribution, eliminating the need for manual configuration or external intervention. This self-optimizing approach ensures high searching efficiency without requiring additional time for manual data distribution tuning.

Inventive Principle:
Principle #25Self-service

3Manufacturing precision

If multiple candidate files are generated and evaluated, then the data distribution optimizes, but the processing time increases

Engineering Contradiction:
Improvedata distribution qualityVSAvoidprocessing time
Core Design Contradiction:
Manufacturing precisionVSLoss of time

Solution Approach 1:

The patent applies partial action by generating multiple candidate files with different sorting configurations, but only evaluating them based on the critical metric of group count. The system generates sufficient candidate files to ensure optimal data distribution, then selectively evaluates them using a streamlined process that counts groups without performing exhaustive analysis. This approach achieves high data distribution quality while limiting processing time through focused evaluation criteria.

Inventive Principle:
Principle #16Partial or excessive action

Data Source

PatentUS10346423B2Minimizing resource contention while loading graph structures into a distributed database
Publication Date: 2019.07.09 OBJECTIVITY INC
  • US10346423B2 patent drawing
  • US10346423B2 patent drawing
  • US10346423B2 patent drawing

AI summary

Presented is a method by which large amounts of data may be sorted and downloaded. In the presented invention, two keys are selected which may be associated with all records in a designated distributed, relational, object-oriented, graph or other database, and the records are sorted first by means of the first and second keys to generate a first file, and subsequently by means of the second and first keys to generate a second file. It is determined which of the resultant files contains a larger number of groups, and the file containing the larger number of groups is selected for loading, preferably by means of a plurality of data load streams. Further presented is means by which the data load streams may be executed in parallel with one another while minimizing contention for shared resources. The method additionally helps to increase the amount of parallelism that may be deployed in support of database search operations.