Parallel Load of Database Mapping Containers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face prolonged start and restart times due to the single-threaded loading of mapping containers, particularly file ID and LOB file ID mapping, which can take multiple hours in systems with billions of LOBs, leading to high load times.
Innovation Solution
Implementing a parallel load operation of file ID and LOB file ID mapping using multiple scanning engines and buffers, where each scanning engine processes a portion of the page chain and transfers data to mapping engines via queues, allowing for simultaneous loading into in-memory storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If single-threaded loading of mapping containers is used, then system simplicity is maintained, but start and restart times are prolonged
Solution Approach 1:
The patent divides the mapping container loading operation into multiple segments by creating multiple scanning engines (first scanning engine, second scanning engine, etc.) that each process different portions of the page chain simultaneously. This segmentation enables parallel loading operations, reducing start and restart times while maintaining manageable complexity through modular engine design.
Solution Approach 2:
The patent transitions from single-threaded sequential loading to multi-threaded parallel loading by introducing multiple scanning engines operating concurrently. This dimensional change from one-dimensional sequential processing to multi-dimensional parallel processing significantly reduces loading time while the modular engine architecture keeps system complexity controlled.
2Loss of time
If parallel loading with multiple scanning engines is implemented, then start and restart times are reduced, but system complexity increases
Solution Approach 1:
The patent introduces memory queues as intermediary components between scanning engines and mapping engines. These queues buffer data from multiple scanning engines and regulate flow to mapping engines, simplifying the coordination complexity of parallel operations while enabling significant speedup in start and restart times through efficient parallel processing.
Solution Approach 2:
Each scanning engine independently processes its assigned portion of the page chain and autonomously transfers data to memory queues without requiring complex inter-engine coordination. This self-service approach reduces system complexity by eliminating the need for centralized control mechanisms while achieving efficient parallel loading.
3Ease of operation
If mapping data is loaded sequentially into in-memory storage, then memory access simplicity is maintained, but loading speed is reduced
Solution Approach 1:
The patent segments the mapping data loading process by assigning different scanning engines to process different portions of the page chain simultaneously. Each engine loads its segment into memory independently, maintaining simple memory access patterns within each segment while achieving high overall loading speed through parallel execution of multiple segments.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Aspects of the current subject matter are directed to an approach in which a parallel load operation of file ID mapping containers is accomplished at start and/or restart of a database system. Parallel load operation of file ID mapping and/or large binary object (LOB) file ID mapping is done among a plurality of scanning engines into a plurality of data buffers that are associated with each of the plurality of scanning engines. Each scanning engine operates on a certain path of a page chain of a page structure including the mapping, causing the page chain to be split among scanning engines to process maps. Contents of the data buffers are pushed to mapping engines via a queue. The mapping engines load the file ID mapping and the LOB file ID mapping into maps for in-system access.