Log-Structured Storage Coalescing for Reduced Transaction I/O
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Databases are challenging to distribute while maintaining ACID properties, leading to costly and complex deployment and maintenance of 'shared nothing' and 'shared disk' models, which are not amenable to efficient scaling.
Innovation Solution
A distributed database system is implemented with a separate storage layer that handles durability and space management, reducing network traffic and costs by offloading operations like backup, restore, and crash recovery, and using log-structured storage to coalesce redo log records for efficient data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If databases are distributed using shared nothing or shared disk models, then fault tolerance and durability are improved, but device complexity and deployment cost increase significantly
Solution Approach 1:
The database system is segmented into independent compute nodes and storage nodes. Compute nodes handle query processing while storage nodes manage data persistence and durability. This segmentation allows the system to achieve fault tolerance through distribution without requiring complex coordination between full database instances, thereby reducing deployment complexity while maintaining reliability.
Solution Approach 2:
Storage functionality is extracted from compute nodes and placed in dedicated storage nodes. This extraction separates the complexity of data management from query processing, allowing compute nodes to focus on computation while storage nodes handle durability guarantees. The result is a simpler deployment model that achieves fault tolerance without replicating full database instances across all nodes.
2Reliability
If log records are written to persistent storage for each transaction, then durability is improved, but I/O bandwidth consumption increases
Solution Approach 1:
Multiple log records are merged into a single persistent storage write operation. Instead of writing each log record individually to disk, the system batches and combines multiple log records into consolidated write operations. This merging maintains durability guarantees while significantly reducing the total number of I/O operations and bandwidth consumption.
Solution Approach 2:
Log records are buffered in memory before being written to persistent storage. The system performs preliminary buffering of log records, allowing multiple transactions to accumulate their log records in memory before a coordinated write to disk. This preliminary action reduces immediate I/O bandwidth requirements while ensuring durability through eventual persistence.
3Reliability
If data is replicated across multiple nodes, then availability is improved, but network bandwidth requirements increase
Solution Approach 1:
Data and log records are stored locally on storage nodes rather than being replicated across all compute nodes. Each storage node maintains local copies of the data it manages, eliminating the need for network-based replication. This local quality approach ensures availability through distributed storage while minimizing network bandwidth consumption by avoiding redundant data transmission.
Data Source
AI summary
A distributed database system may implement reduced transaction I/O. In various embodiments, a plurality of log records may be stored in persistent storage for a plurality of pages of a log-structured storage system. A coalesce operation may be performed for at least some of the plurality of data pages for the log-structured storage system. The coalesce operation may, in some embodiments, coalesce two or more log records for one of the plurality of pages to generate an instance of the page and store the instance of the page to a new location in the persistent storage. Log records for one or more of the plurality of pages may be excluded from the coalesce operation. These excluded log records may not be dependent on a prior version of the page or portion of the page corresponding to the log record. Excluded log records may include one or more of redo records, undo records, or transaction table records.