Individual Write Quorums for Log-Structured Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face inefficiencies due to high complexity and operational costs in maintaining consistent and durable data versions, particularly when performing updates or changes, as they require multiple copies of data across storage nodes, leading to increased latency and congestion.
Innovation Solution
Implementing individual write quorums in a log-structured distributed storage system, where a write quorum requirement is satisfied by acknowledging writes across a subset of storage nodes, allowing for balanced workload distribution and reduced latency, even if not all nodes complete writes immediately, ensuring durability through a quorum set.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple copies or replicas of data are stored on multiple storage nodes, then data availability is improved, but system complexity and operational costs increase
Solution Approach 1:
The patent segments the write acknowledgment process by introducing individual write quorums for different data types. Metadata writes require acknowledgment from a first quorum of storage nodes, while log record writes require acknowledgment from a second quorum. This segmentation allows the system to maintain multiple replicas for reliability while managing complexity through differentiated consistency requirements.
2Reliability
If durability requirements enforce that each copy completes updates, then data durability is improved, but update latency increases
Solution Approach 1:
The patent applies partial action by requiring write acknowledgments from only a quorum subset of storage nodes rather than all replicas. For metadata updates, a first quorum must acknowledge, and for log records, a second quorum must acknowledge. This partial acknowledgment approach ensures durability while reducing update latency by not requiring all copies to complete before acknowledgment.
Solution Approach 2:
The patent implements local quality by applying different durability requirements to different data types. Metadata writes have stricter durability requirements (first write quorum) compared to log record writes (second write quorum). This allows the system to optimize for both durability and performance by tailoring consistency requirements to the specific data type and its criticality.
3Stability of the object's composition
If all storage nodes must complete writes, then data consistency is improved, but write performance and throughput decrease
Solution Approach 1:
The patent segments the consistency requirement into two independent quorums: one for metadata and one for log records. This allows the system to maintain data consistency through quorum acknowledgment while improving write performance by not requiring all storage nodes to complete both types of writes. The segmented approach enables parallel processing and reduces the critical path for write operations.
Data Source
AI summary
A log-structured distributed storage system may implement individual write quorums. Log records may be sent to different storage nodes of a quorum set storing data for a storage client sufficient to satisfy a write quorum requirement. For each log record, acknowledgments from storage nodes are received, and a determination is made whether the write quorum requirement is satisfied for the log record. Different log records may be maintained at different storage nodes, and still satisfy the write quorum requirement such that in some embodiments no one storage node may maintain all of the log records sent to storage nodes in the quorum set.