In-Memory Database Copy Generation Without Blocking DMLs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face interruptions and inefficiencies due to the need to lock entire chunks during pre-loading into volatile memory, which blocks concurrent data manipulation language (DML) operations, especially in high-volume applications like online transaction processing.
Innovation Solution
The system generates an in-memory copy (IMC) of a database object without blocking DMLs by creating in-memory journals to track changes, identifying and invalidating straddling transactions, and allowing DMLs to proceed while ensuring data consistency by reading from the journal or disk when necessary, and repopulating the IMC when it becomes stale.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system obtains an exclusive lock to load a chunk from disk into volatile memory, then data consistency within the in-memory copy is ensured, but concurrent data manipulation language operations are blocked causing significant interruptions
Solution Approach 1:
The patent segments the chunk loading process into multiple phases: (1) obtain shared lock and read chunk data, (2) convert data format in memory, (3) obtain exclusive lock only for the brief moment when writing to in-memory storage, (4) release lock. This segmentation minimizes the duration of exclusive locking while ensuring consistency, allowing DML operations to proceed concurrently during the non-locked phases.
Solution Approach 2:
The patent performs preliminary actions before obtaining the exclusive lock: obtaining a shared lock first to read the chunk data from disk, converting the data format in memory from row-major to column-major format. This preliminary processing reduces the time the exclusive lock needs to be held, thereby minimizing blocking of DML operations while still ensuring data consistency when the in-memory copy is finalized.
2Productivity
If the system grants DML operations higher priority than chunk loading, then application performance is improved, but the load operation may never finish
Solution Approach 1:
The patent implements dynamic lock management where the exclusive lock is acquired only for the specific duration needed to write the in-memory copy to storage, not for the entire loading process. The lock is released immediately after the write operation completes, allowing DML operations to dynamically proceed without unnecessary blocking, thus balancing loading completion with operational responsiveness.
Solution Approach 2:
The patent ensures continuous useful action by allowing DML operations to proceed concurrently during the chunk loading process. While the exclusive lock is briefly held for consistency, DML operations continue uninterrupted during the shared lock phase and format conversion, ensuring both loading and DML throughput maintain continuous productive action without one blocking the other indefinitely.
3Reliability
If the system locks entire chunks during pre-loading, then data consistency is maintained, but applications experience significant interruptions in high-volume DML processing
Solution Approach 1:
The patent applies local quality by obtaining the exclusive lock only for the specific local operation of writing the in-memory copy to storage, rather than locking the entire chunk for the duration of the loading process. This localized locking approach maintains data consistency for the in-memory copy while minimizing the impact on DML operations that can proceed concurrently in other parts of the system.
Solution Approach 2:
The patent performs preliminary actions (obtaining shared lock, reading data, format conversion) before acquiring the exclusive lock, thereby reducing the time the exclusive lock must be held. This preliminary processing ensures data consistency is established before the brief exclusive lock period, minimizing interruption to DML operations while maintaining reliability.
Data Source
AI summary
Techniques are herein described for loading a portion of a database object into volatile memory without blocking database manipulation language transactions. The techniques involve invalidating data items loaded from blocks affected by a transaction, referred to as a straddling transaction that started before the load time and committed after the load time. Identifying these straddling transactions involves reviewing one or more transaction lists associated with the set of data items loaded in memory. The transaction list may be read in reverse temporal order of commit to identify a transaction meeting the criteria of starting before the load start, not committing before the load time, and affecting a data item loaded in memory.