Dynamic Log Pipeline Splitting for Database Write Contention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database system performance is hindered by high contention in writing data to log pipelines, leading to slowed update, delete, and insert operations, as well as delayed flushing of modified data from memory to storage due to log materialization and serialization bottlenecks.
Innovation Solution
The system dynamically splits log pipelines when contention exceeds a threshold, creating additional pipelines and deploying additional log write engines to reduce contention and increase writing speed, while also merging pipelines when contention is low to conserve resources, allowing for concurrent writing of data and improved system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single log pipeline is used for writing data, then device complexity is reduced, but contention increases under high concurrency leading to reduced productivity
Solution Approach 1:
The patent divides a single log pipeline into multiple log pipelines (e.g., log pipeline 0, log pipeline 1, log pipeline 2) to distribute the writing load. Each pipeline handles a subset of write operations, reducing contention and improving throughput. The system dynamically selects which pipeline to use based on current load conditions.
Solution Approach 2:
The patent implements dynamic log pipeline selection where the system can switch between different pipelines based on real-time contention levels. The log manager monitors write operations and dynamically routes them to appropriate pipelines, allowing the system to adapt to varying workloads and optimize performance.
2Reliability
If log data is written to storage before modified data is flushed, then data integrity is improved, but system performance degrades due to serialization bottleneck
Solution Approach 1:
The patent segments the log writing process into multiple parallel pipelines, each capable of writing log data to storage independently. This allows multiple write operations to proceed concurrently rather than serially, maintaining data integrity through proper logging while improving system throughput by eliminating the serialization bottleneck.
Solution Approach 2:
The patent ensures continuous progress of write operations by maintaining multiple active log pipelines that can process data concurrently. The log manager coordinates these pipelines to ensure that log data is written to storage before modified data is flushed, maintaining data integrity while keeping the writing process continuous and non-blocking.
3Productivity
If additional log pipelines are created to reduce contention, then data writing speed increases, but device complexity and resource consumption increase
Solution Approach 1:
The patent implements a dynamic approach where the system creates additional log pipelines only when contention exceeds a threshold level. The log manager monitors write operation performance and dynamically allocates new pipelines as needed, rather than maintaining a fixed large number of pipelines. This allows the system to scale resources based on actual demand.
Solution Approach 2:
The patent changes the parameter of pipeline quantity dynamically based on contention levels. When contention is low, the system uses fewer pipelines to minimize resource consumption. When contention increases, the system creates additional pipelines to improve throughput. This adaptive parameter adjustment optimizes the balance between performance and resource usage.
Data Source
AI summary
Database log writing is based on log pipeline contention. A determination is made as to whether contention in writing data to a log pipeline, which is used in writing data from memory to storage, is at a prespecified level. Based on determining that the contention in writing the data to the log pipeline is at the prespecified level, a split operation is automatically performed to create a new log pipeline.