Independent Data Loading via Hash-Based Key Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems face challenges in efficiently loading related data into data storages, particularly due to dependencies between tables that require sequenced data loads, increasing join complexity and processing time.
Innovation Solution
A data management system that uses column definitions to generate unique values for identifying records in one table based on values from another table, employing hash functions to break dependencies between tables and allow independent data loading.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional database management systems load related data into tables with dependencies, then data integrity is maintained through referential constraints, but data loading complexity increases and processing time extends due to sequenced loading requirements
Solution Approach 1:
The patent segments the data loading process into independent operations for each table. By generating unique identifier values using hash functions before actual data insertion, the system allows each table to be loaded independently without waiting for related tables, thus reducing data loading complexity while maintaining referential integrity through the pre-generated unique values.
Solution Approach 2:
The patent applies preliminary action by generating unique identifier values (primary keys and foreign keys) before the actual data loading process. The hash function computes these unique values in advance based on the data to be loaded, allowing tables to be prepared and loaded independently without requiring sequential processing to ensure referential constraints are met.
2Reliability
If traditional database systems enforce referential constraints between tables, then data consistency is ensured, but processing time increases due to the need for sequenced data loads and join operations
Solution Approach 1:
The system performs preliminary computation of unique identifier values using hash functions before data loading. This allows all necessary primary and foreign key values to be predetermined, eliminating the need for time-consuming sequential loading and reducing join operations during query processing, thus decreasing processing time while maintaining data consistency.
Solution Approach 2:
The patent replaces the mechanical system of sequential data loading and constraint enforcement with a computational approach using hash functions. Instead of physically ensuring referential integrity through ordered loading operations, the system computationally generates unique values that inherently satisfy referential constraints, thereby reducing processing time.
3Reliability
If data is loaded into related tables with foreign key dependencies, then referential integrity is maintained, but data loading flexibility decreases due to mandatory loading sequences
Solution Approach 1:
The patent segments the data loading process into independent, parallelizable operations. Each table can be loaded separately with pre-computed unique identifiers, allowing flexible loading sequences and parallel processing without compromising referential integrity. This segmentation enables adaptability in data loading strategies while maintaining reliability.
Solution Approach 2:
The hash function acts as an intermediary that mediates between the data to be loaded and the unique identifier requirements. It transforms the input data into unique values that satisfy referential constraints without requiring direct coordination between tables during loading, thus enhancing data loading flexibility while preserving referential integrity.
4Ease of operation
If traditional systems use join operations to relate data across tables, then comprehensive data access is achieved, but computing time increases due to complex join processing
Solution Approach 1:
The system performs preliminary computation of unique identifier values using hash functions during the data loading phase. By pre-establishing these unique values that inherently encode relationships, the system reduces the need for complex join operations during query processing, thereby decreasing computing time while maintaining comprehensive data access capability.
Data Source
AI summary
Some embodiments provide a program that receives a set of data for a first record in a first table. The set of data includes a set of values for a set of attributes. In a data loading process configured to load a subset of the set of data into a subset of a first set of columns in the first table, the program determines that a first column in a first set of columns does not belong in the subset of the first set of columns. The program generates the first record in the first table. The program generates a value for the first column in the first set of columns that refers to a second record in the second table configured to represent a defined type of record. The program stores the value in the first column in the first set of columns of the first record.