Database Post-Commit Processing Delegation via Background Jobs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in minimizing commit latency and ensuring consistent visibility of records during concurrent transactions, particularly in handling temporary timestamps and post-commit processing.
Innovation Solution
The solution involves assigning temporary timestamps to records based on transactions, delegating post-commit processing to a background job, and using a visibility function that considers transaction states and timestamps to determine record visibility, allowing for efficient re-stamping of commit timestamps and reducing latency by processing commits with O(1) complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional commit processing is used where all records are re-stamped synchronously, then timestamp consistency is ensured, but commit latency increases significantly
Solution Approach 1:
The system performs preliminary actions by setting a temporary commit timestamp during the commit process and immediately delegating the actual re-stamping operation to a background job. This allows the commit transaction to complete quickly with O(1) complexity while the background process handles the time-consuming re-stamping of all affected records afterward, thus resolving the contradiction between fast commits and timestamp consistency.
Solution Approach 2:
A background job acts as an intermediary between the commit transaction and the actual re-stamping operation. The background job receives the commit timestamp and list of affected records, then performs the re-stamping asynchronously. This intermediary mechanism decouples the commit latency from the re-stamping time, allowing commits to be fast while ensuring eventual consistency through the background process.
2Productivity
If post-commit processing is handled synchronously in the main transaction thread, then timestamp accuracy is maintained, but transaction throughput decreases
Solution Approach 1:
The commit processing is segmented into two independent parts: (1) the main transaction thread that sets the temporary commit timestamp and completes the commit with O(1) complexity, and (2) a background job that handles the time-consuming re-stamping of all affected records. This segmentation allows the main thread to proceed quickly without waiting for re-stamping, thereby improving transaction throughput while the background job ensures timestamp accuracy.
Solution Approach 2:
The background job serves as an intermediary that takes over the time-consuming re-stamping task from the main transaction thread. It receives the commit timestamp and affected records list, then processes them asynchronously to update all affected records with the correct commit timestamp, ensuring timestamp accuracy without blocking the main transaction flow and thus improving throughput.
3Speed
If a simple visibility function is used that only checks timestamps, then processing speed increases, but visibility correctness during concurrent transactions deteriorates
Solution Approach 1:
The visibility function implements local quality by treating different transaction states differently. For transactions in active or committing state, the function performs a simple timestamp comparison for fast processing. For transactions in committed state, the function checks both the timestamp and the post-commit processing status to ensure correctness. This differentiated approach maintains high speed for most cases while ensuring correctness when needed.
Solution Approach 2:
The visibility function is made dynamic by adapting its checking behavior based on the transaction state and the presence of temporary timestamps. When a record has a temporary commit timestamp, the visibility function performs additional checks to determine if the background re-stamping has completed. This dynamic adjustment ensures correctness for committed transactions while maintaining fast processing for active transactions, resolving the contradiction between speed and correctness.
Data Source
AI summary
A plurality of transactions are handled in a database. Each transaction comprises a plurality of operations on at least one record in the database with at least two of the transactions being handled concurrently. Thereafter, a temporary timestamp is assigned to each record that is based, at least in part, on the corresponding transaction. A first transaction among the plurality of transactions is subsequently committed. Afterwards, re-stamping of at least one commit timestamp modified by the first transaction is delegated. Related apparatus, systems, techniques and articles are also described.