Distributed Database Storage Using Redo Logs for Fast Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in distributing database components while ensuring ACID properties and are costly and complex to deploy and maintain, with crash recovery not scaling well to large databases.
Innovation Solution
A database system is partitioned into tiers, with durability and space management functions offloaded to a separate distributed storage layer, using redo log records for asynchronous, lazy crash recovery and log-structured storage to minimize network traffic and improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If database systems use traditional direct attached disks with snapshot-based crash recovery, then data durability is ensured, but the system does not scale well to large databases and recovery time increases
Solution Approach 1:
The patent segments the database system into distinct layers: compute nodes that handle query processing and a separate distributed storage service that handles data persistence. This segmentation allows independent optimization of each layer, enabling the storage layer to use efficient log-structured storage while compute nodes focus on ACID transaction processing, thereby reducing recovery time for large databases while maintaining durability
Solution Approach 2:
The patent implements preliminary actions through continuous logging of all data modifications to a distributed log-structured storage service. By pre-recording all changes in a sequential log format with version vectors, the system prepares recovery information in advance, allowing for rapid crash recovery by simply replaying the log from the last checkpoint rather than restoring from snapshots, thus reducing recovery time while ensuring data durability
2Reliability
If database systems are distributed using shared nothing or shared disk models with glue logic, then fault tolerance and durability are improved, but deployment cost and system complexity increase
Solution Approach 1:
The patent extracts the storage management functionality from the traditional database system and places it in a separate distributed storage service. This extraction eliminates the need for complex glue logic at compute nodes, as the storage service independently manages data persistence, replication, and recovery. The compute nodes simply interact with the storage service through standard interfaces, significantly reducing system complexity while maintaining fault tolerance through the distributed storage architecture
Solution Approach 2:
The distributed storage service implements multiple functions within a single unified system: it provides data persistence, handles replication across nodes, manages crash recovery, and supports version control through version vectors. This multi-functionality eliminates the need for separate components for each function, reducing overall system complexity while maintaining robust fault tolerance through its distributed architecture
3Reliability
If databases are scaled out using replicated monolithic instances with dispatcher glue logic, then ACID properties are maintained, but deployment cost increases and the system over-serves many use cases
Solution Approach 1:
The patent segments the database architecture into compute nodes that handle ACID transaction processing and a separate distributed storage service that handles data persistence. This segmentation allows each component to be optimized independently: compute nodes maintain ACID properties through standard database protocols while the storage service uses efficient log-structured storage, reducing deployment costs by eliminating redundant functionality in replicated monolithic instances
4Reliability
If crash recovery uses snapshot restoration and log replay, then data durability is ensured, but the approach does not scale well to large databases
Solution Approach 1:
The patent inverts the traditional crash recovery approach by using log-structured storage where the log is the primary storage format rather than using snapshots as the primary storage and log as incremental updates. This inversion allows the system to scale efficiently because the log can be stored in a distributed manner across multiple nodes with version vectors, enabling parallel recovery operations and eliminating the bottleneck of restoring large snapshot images, thus improving both scalability and data durability
Data Source
AI summary
A database system may include a database service and a separate distributed storage service. The database service (or a database engine head node thereof) may be responsible for query parsing, optimization, and execution, transactionality, and consistency, while the storage service may be responsible for generating data pages from redo log records and for durability of those data pages. For example, in response to a write request directed to a particular data page, the database engine head node may generate a redo log record and send it, but not the data page, to a storage service node. The storage service node may store the redo log record and return a write acknowledgement to the database service prior to applying the redo log record. The server node may apply the redo log record and other redo log records to a previously stored version of the data page to create a current version.