Database Bulk Load Method Using New Data Area and File Control Block
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing bulk load methods into databases face issues with data consistency and performance, particularly during large data insertions, where transactional logging is slow and non-transactional methods risk database corruption and accessibility issues.
Innovation Solution
A method that allocates a new data area for bulk loads, locks it for writing to prevent interference, transfers data internally bypassing transaction logs, builds a new index, and updates the file control block without affecting active data, allowing for fast, non-corrupt data addition and maintaining database accessibility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transactional logging is used for bulk load, then data consistency is ensured, but loading speed becomes very slow
Solution Approach 1:
The patent segments the bulk load process into two distinct phases: a fast non-transactional loading phase that writes data without logging, and a subsequent validation/commit phase that ensures consistency. This segmentation allows the system to achieve both high loading speed and data consistency by performing these functions at different times rather than requiring transactional logging during the entire load process.
Solution Approach 2:
The patent performs preliminary data validation and formatting before the bulk load operation begins. By preparing and validating data in advance, the system can then perform the actual bulk insertion without transactional overhead, achieving both speed and consistency. The preliminary action includes checking data integrity and preparing the target structure before the fast load occurs.
2Productivity
If non-transactional bulk load is used, then loading speed increases, but database corruption risk increases on failure
Solution Approach 1:
The patent implements beforehand cushioning by creating validation mechanisms and checksums before the bulk load operation. It performs preliminary data validation, creates backup references, and establishes integrity checks that will detect any corruption. This preparatory cushioning allows the system to safely perform fast non-transactional loading while having pre-positioned safeguards against potential corruption.
Solution Approach 2:
The patent implements feedback mechanisms that continuously monitor the bulk load process and validate data integrity during and after loading. Checksum verification, referential integrity checks, and consistency validation provide feedback loops that can detect corruption early and trigger recovery procedures, allowing fast non-transactional loading while maintaining reliability through active monitoring and validation.
3Productivity
If non-transactional bulk load is used, then loading speed increases, but database accessibility decreases during load
Solution Approach 1:
The patent segments database accessibility by implementing read-only access during the bulk load phase while allowing write operations after completion. This temporal segmentation allows the bulk load to proceed at high speed without transactional overhead, while the database remains partially accessible for reading during the process and fully accessible for writing once completed, thus balancing speed and accessibility.
Solution Approach 2:
The patent introduces an intermediary layer or mechanism that manages access rights during bulk load operations. This intermediary controls which operations are permitted (typically read-only) while the load occurs, and automatically transitions to full access mode upon successful completion. This mediator enables fast non-transactional loading while maintaining controlled accessibility rather than complete lockdown.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
The present invention relates to a method for performing a bulk load into a data partition (12) of the database (10), which does not compromise the database in case of a failure during the bulk load process and allows read access to the affected data while the bulk load is performed. This is achieved by storing the bulk load data (24) in a new data area (22) while locking the old data for writing; building a new index (26) which covers the old and new data (22); building a new file control block (28) which includes the storage location of the new index (26) and the old (16) and new data (22) in the storage device (14); and then in the last step appending the new data to the data of the database by replacing the active file control block (20) with the new file control block (28).