Data Flush Mechanism for Distributed Storage Durability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing block-based storage systems face challenges in maintaining data durability and low latency due to single-point failures in server nodes or control planes, leading to significant losses in storage capacity and increased recovery latency when data is distributed across multiple locations.
Innovation Solution
A data storage system comprising head nodes and data storage sleds, where data is replicated across multiple head nodes and mass storage devices, allowing for independent operation without a zonal control plane, with a local control plane managing failovers and data distribution across redundant networks and power sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored across multiple devices in multiple locations using a common control plane, then data durability is improved, but system complexity increases and a single point of failure can impact large quantities of storage capacity
Solution Approach 1:
The system divides the storage infrastructure into independent distributed head nodes, each capable of autonomous operation. Each head node manages its own data and can operate independently without relying on a centralized control plane, thereby segmenting the system to eliminate single points of failure while maintaining data durability through distribution across multiple locations.
Solution Approach 2:
Head nodes are designed to be self-sufficient, performing data management operations autonomously without requiring a common control plane. Each head node can independently handle data storage, retrieval, and replication tasks, eliminating the bottleneck and failure risk associated with centralized control while preserving data durability through distributed operations.
2Reliability
If data is distributed across multiple geographic locations, then data durability is improved, but data recovery latency increases due to extensive networks required to move data between locations
Solution Approach 1:
The system segments data storage into distributed head nodes across multiple geographic locations, with each node maintaining independent data copies. This segmentation enables local data recovery operations at each site without requiring data transfer across extensive networks, thereby reducing recovery latency while preserving the durability benefits of geographic distribution.
3Productivity
If a server node services multiple storage nodes in a SAN architecture, then storage capacity utilization is improved, but a failure of the server node renders large amounts of storage capacity unusable
Solution Approach 1:
The system segments the traditional server-dependent storage architecture into independent head nodes, where each head node directly manages its associated storage nodes without requiring a central server. This segmentation eliminates the single point of failure represented by the server node, allowing storage capacity to remain accessible even when individual head nodes fail, thereby maintaining both high capacity utilization and reliability.
Solution Approach 2:
Each head node operates autonomously to service its own storage nodes, eliminating dependence on a central server. This self-service capability ensures that storage capacity remains accessible and manageable even when other parts of the system fail, maintaining high availability while preserving efficient capacity utilization through distributed management.
Data Source
AI summary
A data storage system includes a head node and mass storage devices. The head node is configured to flush data stored in a storage of the head node, based at least in part on one or more triggers being met, from the storage of the head node to a set of the mass storage devices of the data storage system. The flushed data is written to a segment of free storage space across the set of the mass storage devices allocated for the given data flush operation. In some embodiments, a head node may flush both current version data and point-in-time version data to the set of mass storage devices. Also, the data storage system maintains an index that indicates storage locations of data for particular portions of a volume before and after the data is flushed to the set of mass storage devices.