Direct Storage Loading for Database Data Ingestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face performance issues when adding large amounts of data, as this process consumes significant system resources, hindering other workloads and disrupting performance.
Innovation Solution
Implementing direct storage loading, where the database engine offloads the work to separate resources, such as a data loading cluster, to prepare and store data without utilizing the database engine's resources, allowing for parallelization and reducing the time required to load data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is added to a database using traditional methods through the database engine, then data can be loaded into the database, but system resources are significantly consumed and performance of other workloads is hindered
Solution Approach 1:
The patent extracts the data loading function from the database engine by introducing a separate data loading cluster. This cluster independently loads data directly into the database storage without involving the database engine, thereby separating the resource-intensive loading operation from the database system and eliminating the performance degradation and resource consumption issues.
2Productivity
If data is added to a database using traditional methods, then data can be loaded, but the loading process disrupts performance of other database operations
Solution Approach 1:
By extracting the data loading operation into a separate data loading cluster, the patent isolates the disruptive loading process from the database engine. This ensures that database operations continue to execute with stable performance since they are not competing for resources or contending with the loading process.
Solution Approach 2:
The patent segments the database system into distinct functional components: the database engine for processing queries and operations, and a separate data loading cluster for data ingestion. This segmentation allows each component to operate independently without interfering with the other, maintaining performance stability.
3Productivity
If traditional data loading methods are used, then data can be added to the database, but the process is slow and consumes significant time
Solution Approach 1:
By extracting the data loading function to a dedicated data loading cluster, the patent enables optimized loading processes that do not contend with database operations for resources. This results in significantly faster data loading times, reducing the loss of time while maintaining high productivity.
Data Source
AI summary
Direct storage loading may be used to add data to a database. New data may be added to a database, using nodes different than a database engine to access a database. The addition of the new data may be assigned to different nodes. The nodes may obtain the data and store the data to storage locations according allocated space in the database by the database engine. The new data can then be made available for access at the database engine.