Log-Structured Distributed Storage Using Single LSN Space
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in maintaining consistent views of data across multiple storage nodes while reducing latency and operational costs, as existing synchronization protocols often increase complexity and resource utilization.
Innovation Solution
Implementing a log-structured distributed storage system using a single log sequence number space, where log records are segmented across protection groups and metadata is used to maintain consistency and enable independent recovery, reducing the need for complex synchronization protocols and minimizing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronization protocols are used to ensure consistency across storage nodes, then data consistency is improved, but time consumption and resource utilization increase
Solution Approach 1:
The patent segments the log structure into multiple chains distributed across different storage nodes, with each node maintaining a portion of the log chains. This segmentation allows parallel processing of write operations across nodes without requiring centralized synchronization, thereby reducing time consumption while maintaining consistency through the shared log sequence number space.
Solution Approach 2:
The patent implements a universal log sequence number space that is shared across all storage nodes in the distributed system. This single LSN space serves multiple functions: it provides global ordering of log records, enables consistency verification across nodes, and allows any node to process any log chain without coordination. This multi-functionality eliminates the need for complex synchronization protocols while ensuring data consistency.
2Reliability
If synchronization protocols are used to ensure consistency across storage nodes, then data consistency is improved, but system complexity increases
Solution Approach 1:
By segmenting logs into distributed chains and assigning different chains to different storage nodes, the system eliminates the need for centralized synchronization protocols. Each node independently manages its assigned chains using the shared LSN space for consistency verification, significantly reducing system complexity while maintaining data consistency across the distributed system.
Solution Approach 2:
Each storage node autonomously processes write operations on its assigned log chains by generating and verifying LSNs against the shared LSN space without requiring external coordination or synchronization protocols. This self-service capability reduces system complexity by eliminating inter-node communication overhead for consistency management.
3Reliability
If multiple versions or replicas of data are stored on multiple storage nodes, then data availability is improved, but operational costs increase
Solution Approach 1:
The patent segments the log structure into multiple chains distributed across storage nodes, with each node storing a portion of the log and associated data. This segmentation provides data availability through distribution while reducing operational costs by eliminating the need to maintain multiple full replicas of the entire dataset across all nodes, thereby reducing storage and synchronization overhead.
Solution Approach 2:
The shared log sequence number space serves as a universal mechanism that enables multiple storage nodes to maintain data availability independently while ensuring consistency. Each node can serve read and write requests for its assigned data portions without requiring coordination with other nodes, reducing operational costs associated with synchronization and replica management.
Data Source
AI summary
A distributed database system may implement log-structured distributed storage using a single log sequence number space. A log for a data volume may be maintained in a log-structured distributed storage system. The log may be segmented across multiple protection groups according to a partitioning of user data for the data volume. Updates to the log may be assigned a log sequence number from a log sequence number space for the data volume. A protection group may be determined for an update according to which partition of user data space the update pertains. Metadata to be included with the log record may indicate a previous log sequence number of a log record maintained at the protection group. The log record may be sent to the protection group and identified as committed based on acknowledgments received from storage nodes implementing the protection group.