Database Bulk Load Tool Using Control Files to Reduce Resource Consumption
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing amount of data in databases leads to resource-intensive loading and indexing processes, which can render databases unusable when many users concurrently load and index data, causing high processor and memory consumption.
Innovation Solution
A database load tool comprising a user interface, naming engine, load engine, and index engine that bulk loads and indexes files by renaming files, generating control files, and loading them into the database, reducing the need for individual user loading and indexing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If multiple users concurrently load and index data individually, then each user can access and load data independently, but processor and memory resources are excessively consumed rendering the database unusable
Solution Approach 1:
The patent combines multiple individual data loading and indexing operations into a single bulk load process. Instead of allowing each user to load data separately which consumes excessive resources, the system merges these operations into one coordinated bulk load that shares database connections and processing resources, thereby reducing overall processor and memory consumption while maintaining the ability to handle multiple users' data.
Solution Approach 2:
The system performs preliminary actions by pre-naming files with sequential identifiers and pre-generating control files before the actual bulk load operation. This preparation work is done outside the database locking period, so when the bulk load executes, the files are already ready to be processed in sequence without requiring repeated database access for naming or control file creation, thus reducing resource consumption during the critical loading phase.
2Use of energy by moving object
If bulk load is performed with database locking, then resource consumption is reduced, but database access is blocked for other operations
Solution Approach 1:
The bulk load process is segmented into distinct phases: a preparation phase where files are pre-named and control files are generated (performed outside database locking), and an execution phase where the actual data loading occurs with minimal locking. This segmentation allows the system to reduce the duration and scope of database locks, maintaining better availability while still achieving resource efficiency in the critical loading phase.
Solution Approach 2:
Control files serve as intermediaries between the bulk load process and the database. These control files contain all necessary metadata and loading instructions, allowing the bulk load engine to process data efficiently with minimal database interaction. The intermediary control files enable the system to maintain database availability by reducing the time the database is locked during the loading process.
3Adaptability or versatility
If individual file loading is performed, then each file can be processed independently, but the loading process is time-consuming and resource-intensive
Solution Approach 1:
The patent merges multiple individual file loading operations into a single bulk load operation that processes multiple files simultaneously or in sequence with shared resources. The bulk load engine combines the processing of numerous files into one coordinated operation, maintaining the ability to handle each file's specific requirements while achieving much higher throughput and speed compared to individual loading operations.
Data Source
AI summary
A database load tool includes a user interface, a naming engine, a load engine, and an index engine. The user interface receives a path of a folder storing a plurality of files and an index. The naming engine renames each file of the plurality of files in response to the user interface receiving the path. The load engine loads the plurality of files to a database in response to the user interface receiving the path. The index engine generates a control file for each file of the plurality of files in response to the user interface receiving the path. Each control file includes a file type of the file for the control file and the index. The load engine further loads the control file to the database. Each file of the plurality of files is indexed according to its control file by the database.