Direct Data Loading Middleware Records DBMS
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Database Management Systems (DBMS) face bottlenecks due to data loading being treated as a 'one-time' offline process, which becomes inefficient with explosive data growth, leading to increased overhead and latency, especially when middleware is involved.
Innovation Solution
The method involves sending external-writes to a memory-based DBMS for vertices, exporting all external-writes to a disk-based DBMS as an export file, and sending external-reads to an in-memory DBMS if required, while generating raw data files from the export file and sending them to the disk-based DBMS, thereby optimizing data loading processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If data loading is treated as a one-time offline process in conventional DBMS, then hardware cost is minimized, but data processing speed and transaction throughput deteriorate with explosive data growth
Solution Approach 1:
The patent segments data loading into two distinct phases: (1) bulk data loading phase that loads majority of data in one go, and (2) incremental data loading phase that handles new/changed data. This segmentation allows the system to optimize for both bulk loading efficiency and incremental update performance, resolving the contradiction between minimizing hardware cost and maintaining high transaction throughput with growing data volumes.
Solution Approach 2:
The patent performs preliminary data loading of the majority of data before the critical analysis phase. By pre-loading bulk data into the DBMS in advance, the system eliminates the need for repeated data loading during analysis, thereby improving data processing speed and transaction throughput without requiring excessive hardware resources.
2Ease of operation
If middleware is used to provide simple API for analytics, then application development complexity is reduced, but data loading overhead increases
Solution Approach 1:
The patent introduces an export extension as an intermediary component between the middleware and the DBMS. This export extension intercepts write operations, exports data to files, and manages the dual-phase loading process. By placing this intermediary layer, the system maintains the simplicity of the middleware API for application developers while internally optimizing data loading to reduce overhead through batch processing and incremental updates.
3Reliability
If data loading is performed in one go before queries, then data consistency is ensured, but data loading becomes a bottleneck in the data analysis pipeline
Solution Approach 1:
The patent segments the data loading process into bulk loading of majority data followed by incremental loading of new/changed data. This segmentation allows queries to be executed on the already-loaded bulk data while new data is progressively loaded, thereby maintaining data consistency for queried portions while eliminating the bottleneck effect of requiring complete data loading before any analysis can proceed.
Solution Approach 2:
The patent enables continuous data loading and querying operations to proceed simultaneously through incremental updates. Once bulk data is loaded, the system continues to load new data incrementally without halting query operations, ensuring that useful analysis actions continue uninterrupted while data consistency is maintained for the loaded portions.
Data Source
AI summary
A computer-implemented method is presented for reducing data loading overhead of middleware to facilitate direct data loading to a database management system (DBMS). The method includes, while data loading, sending external-writes to a memory-based DBMS if corresponding internal-writes are for vertices, exporting all the external-writes to a disk-based DBMS as an export file, and sending an external-read for vertices to an in-memory DBMS if the middleware requests data. The method further includes, at the end of data loading, generating files for raw data of the disk-based DBMS from the export file and sending the generated raw files to the disk-based DBMS.