Database Mass Insert Using Preliminary Pages
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of inserting multiple records into pages loaded into memory in an in-memory database can be resource intensive and slow down database performance, particularly due to the inefficiencies in handling partially filled pages and managing free space during mass insert operations.
Innovation Solution
The database system employs a method where preliminary pages are iteratively filled and registered, with the last partially filled page being either deleted or kept for future mass insert operations, and entries can be moved to different pages within the container to optimize resource usage and reduce bottlenecks during mass insert operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If records are inserted into pages loaded into memory to enable more rapid operations, then database operation speed is improved, but resource consumption increases and performance slows down during mass insert operations
Solution Approach 1:
The mass insert operation is segmented into multiple phases: preliminary page creation, bulk entry insertion into preliminary pages, registration of full pages, and handling of the last partially filled page. This segmentation allows the system to optimize resource usage by processing entries in batches and managing memory pages systematically, reducing overall resource consumption while maintaining high insertion speed
Solution Approach 2:
Preliminary pages are created and allocated in advance before the actual mass insert operation begins. These preliminary pages are pre-configured with appropriate data structures and pointers, allowing the insertion process to proceed without the overhead of dynamic page allocation and management during the actual insertion, thus reducing resource consumption while maintaining speed
2Productivity
If multiple records are inserted into pages loaded into memory, then insertion efficiency is improved, but handling partially filled pages becomes complex and resource intensive
Solution Approach 1:
The complexity of managing partially filled pages is extracted and isolated into a separate handling mechanism. The last partially filled page is identified and separated from the bulk processing flow, allowing it to be handled through a simplified path that involves either moving entries to a different page or deleting the preliminary page structure. This extraction reduces the overall management complexity while maintaining high insertion efficiency
Solution Approach 2:
The system discards the preliminary page structure for the last partially filled page by either moving its entries to a different page or deleting it entirely. This discarding approach eliminates the complexity of maintaining inconsistent page structures, and the necessary data is recovered through the move or insert operations, simplifying the overall page management process while preserving insertion efficiency
3Adaptability or versatility
If the last partially filled page is kept for future mass insert operations, then resource utilization is improved, but page management complexity increases
Solution Approach 1:
The last partially filled page is given multi-functionality: it can serve as a storage container for remaining entries from the current mass insert operation, and simultaneously be prepared for reuse in future mass insert operations. By registering this page with the container and maintaining it in a reusable state, the system improves resource utilization without significantly increasing management complexity, as the same page management mechanisms handle both its current and future roles
Data Source
AI summary
A database system receives a mass insert operation command specifying a plurality of entries. Thereafter, the entries are iteratively inserted into a plurality of preliminary pages associated with a container provided by a persistence layer of the database system. Thereafter, the preliminary pages are registered to the container once they are full.