In-place Database Snapshots via Log Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Databases are challenging to distribute while maintaining ACID properties, leading to costly and complex deployment and maintenance of 'shared nothing' and 'shared disk' models, which may not be suitable for all database use cases.
Innovation Solution
A distributed database system is implemented with a separate database engine and storage service, where log records are managed and snapshots are created without reading or writing data pages, reducing network traffic and allowing for asynchronous operations across multiple nodes, thus distributing only subsets of functional components for scaling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If databases are distributed using shared nothing or shared disk models, then fault tolerance and durability are improved, but deployment cost and system complexity increase significantly
Solution Approach 1:
The database system is segmented into independent compute nodes that can be distributed across multiple machines. Each node maintains a portion of the database functionality, allowing the system to be scaled out horizontally while maintaining ACID properties through coordinated segmentation of database operations across nodes.
Solution Approach 2:
A distributed transaction coordinator acts as an intermediary to manage transactions across multiple compute nodes. This coordinator ensures ACID compliance by orchestrating commit protocols and maintaining consistency without requiring complex point-to-point communication between all nodes.
2Reliability
If monolithic database instances are replicated across multiple nodes, then fault tolerance is improved, but resource cost and maintenance complexity increase
Solution Approach 1:
Database functionality is extracted from monolithic instances and distributed as smaller compute node processes. This extraction allows the same database service to run on multiple smaller, less expensive nodes rather than requiring replication of full monolithic instances on expensive hardware.
Solution Approach 2:
The system uses multiple small, inexpensive compute nodes that can be easily provisioned and deprovisioned. These nodes can be replaced or scaled without the high cost associated with replicating large monolithic database instances, enabling cost-effective fault tolerance through horizontal scaling.
3Loss of time
If snapshots are created by reading and writing data pages across the network, then backup operations are performed, but network traffic increases and performance decreases
Solution Approach 1:
Snapshot data is prepared and staged in advance within the distributed storage system before actual snapshot creation is requested. This preliminary action allows snapshot operations to complete quickly without requiring extensive network data transfer at the moment of snapshot creation, thereby reducing both time and network traffic.
Solution Approach 2:
The distributed storage system automatically manages snapshot creation by utilizing its own internal data structures and metadata. Instead of requiring data to be read from and written to network storage, the storage system serves its own snapshot needs through self-service mechanisms that operate locally within the distributed storage architecture.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A database system may maintain a plurality of log records at a distributed storage system. Each of the plurality of log records may be associated with a respective change to a data page. A snapshot may be generated that is usable to read the data as of a state corresponding to the snapshot. Generating the snapshot may include generating metadata that is indicative of a particular log identifier of a particular one of the log records. Generating the snapshot may be performed without additional reading, copying, or writing of the data.