Log-Structured Storage Tail and Stream Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in designing a durable log data storage system that supports low-latency writes and high-throughput reads while preventing duplicate writes and ensuring data durability even in the event of failures.
Innovation Solution
The system employs a dual-data store approach with a 'tail store' for low-latency writes to a raw block device and a 'stream store' for high-throughput scans, using a filesystem for sorted data storage, ensuring write-once semantics and asynchronous persistence to guarantee data durability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a single storage system is used for log data, then device complexity is reduced, but it cannot simultaneously support low-latency writes and high-throughput reads
Solution Approach 1:
The storage system is segmented into two independent components: a tail store optimized for low-latency writes using raw block devices, and a stream store optimized for high-throughput reads using filesystems. This segmentation allows each component to be specialized for its specific function without compromising the other performance metric.
2Reliability
If data is written to a filesystem for durability, then data durability is improved, but write latency increases
Solution Approach 1:
The tail store acts as an intermediary for write operations, providing low-latency writes to raw block devices without the overhead of filesystem operations. The stream store serves as the intermediary for durability, using filesystems to ensure data persistence. This intermediary arrangement allows the system to achieve both low write latency and data durability through different pathways.
3Productivity
If log data is stored in sorted order by key, then scan efficiency is improved, but write complexity increases
Solution Approach 1:
The system segments storage responsibilities: the stream store maintains sorted data by key to enable efficient scans, while the tail store handles unsorted writes to raw block devices. This segmentation allows scan-optimized storage without burdening the write path with sorting complexity.
4Reliability
If synchronous persistence is used, then data durability is improved, but write latency increases
Solution Approach 1:
The system performs preliminary writes to the tail store in memory before flushing to persistent storage. This preliminary action in volatile memory provides low-latency write acknowledgment, while the actual persistence to the stream store occurs asynchronously, ensuring durability without blocking the write path.
5Productivity
If a dual-data store approach is used, then write performance and read performance are both improved, but device complexity increases
Solution Approach 1:
Each storage component is given local quality optimized for its specific function: the tail store uses raw block devices with minimal abstraction for maximum write speed, while the stream store uses filesystems with sorting and indexing for maximum read efficiency. This local optimization justifies the dual-store architecture by providing specialized performance characteristics.
Data Source
AI summary
A system includes reception of a first write request from a client including a first key and a first one or more stream, payload pairs associated with the first key, copying of the first key and the first one or more stream, payload pairs to a first buffer of a volatile memory, storage of data of the first buffer in one or more blocks of a raw block non-volatile memory device, providing of the first buffer to a stream store server, reception of the first buffer at the stream store server, adding of the first key and the first one or more stream, payload pairs to a second buffer of the volatile memory, in key-order, storage of the data of the second buffer in a filesystem storage device, according to stream, and transmission of an indication of the durability of the key to the tail store server.