Thread-Local Transaction Log Buffering for Faster Data Persistence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The contention for invoking an insert thread in a processor by multiple threads in a data processing system increases transaction execution delays and occupies computing resources, affecting performance.
Innovation Solution
A data processing system with a first processor and a second processor, where the second processor actively writes transaction logs from data buffers into a buffer and persists them to storage, offloading the need for the first processor to contend for an insert thread, thereby reducing resource occupation and transaction delays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple threads contend for invoking an insert thread in the processor, then transaction logs can be persisted to storage, but transaction execution delay increases and computing resources are occupied
Solution Approach 1:
The patent extracts the insert thread invocation process from the first processor to a dedicated second processor. The second processor independently manages transaction log persistence to storage, eliminating the contention bottleneck in the first processor while ensuring data durability through WAL mechanisms.
Solution Approach 2:
The patent introduces a buffer as an intermediary between the first processor and storage device. The buffer temporarily stores transaction logs, allowing the first processor to continue executing transactions without waiting for persistent storage, thus reducing execution delay while maintaining data durability.
2Reliability
If multiple threads contend for invoking an insert thread in the processor, then transaction logs can be persisted to storage, but computing resources are occupied
Solution Approach 1:
The patent segments the data processing system into two independent processors: the first processor dedicated to transaction execution and the second processor dedicated to log persistence. This segmentation allows parallel operation, reducing computing resource contention and improving overall system efficiency while maintaining data durability.
Solution Approach 2:
The second processor independently manages the persistence of transaction logs without requiring resources from the first processor. By self-managing the WAL buffer and storage operations, it eliminates resource occupation in the transaction processing path while ensuring data durability.
3Reliability
If the first processor writes transaction logs to data buffers and contends for insert thread, then data modification can be tracked, but transaction execution efficiency decreases
Solution Approach 1:
The patent extracts the log persistence function from the first processor to the second processor. The first processor only needs to write logs to its local data buffer, while the second processor handles the contentious insert thread operations and storage persistence, thereby improving transaction execution efficiency while maintaining atomicity through proper log tracking.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
This application provides a data processing system. The data processing system includes a first processor, a storage device, and a second processor. The first processor includes a plurality of threads, and each thread corresponds to one data buffer. The first processor is configured to execute the plurality of threads, to separately write a plurality of transaction logs into a plurality of data buffers respectively corresponding to the plurality of threads. The second processor includes a buffer. The second processor is configured to: separately obtain the plurality of transaction logs from the plurality of data buffers, store the plurality of transaction logs to the buffer, and persistently store the plurality of transaction logs from the buffer to the storage device. In this way, in a process of executing the plurality of threads, the first processor does not need to contend for invoking an insert thread, and does not need to execute a process of copying a transaction log, so that the first processor can complete processing of a transaction in a shorter time period. This can effectively improve performance of the data processing system. In addition, this application further provides a corresponding data processing method and a related device.