Multi-writer Log-Structured File System for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, storage nodes with limited processing power struggle to handle input/output traffic efficiently due to write amplification, which occurs when more data is written to physical media than intended, often due to parity blocks for error detection and correction, limiting the ability to execute virtualization logic directly on storage nodes, especially with multi-writer log-structured file systems.
Innovation Solution
Implementing a global multi-writer log-structured file system (LFS) that allows concurrent writing from multiple objects on different physical nodes, using local and master segment usage tables (SUTs) to manage free segments, and leveraging storage virtualization to allocate and manage storage resources efficiently, thereby reducing write amplification by writing full segments in an erasure-encoded manner.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple objects simultaneously write to the LFS from different physical nodes, then storage throughput and parallelism are improved, but write amplification increases due to parity block calculations and coordination overhead
Solution Approach 1:
The patent divides the storage system into multiple independent segments that can be written to in parallel. Each segment is managed independently with its own metadata, allowing multiple objects to write simultaneously without interfering with each other. This segmentation reduces the coordination overhead and parity block calculations needed for multi-writer operations, thereby improving throughput while controlling write amplification.
Solution Approach 2:
The patent introduces an intermediary layer (the LFS metadata management system) that coordinates writes from multiple objects to different segments. This intermediary manages the allocation of segments, tracks free space, and handles the necessary parity block operations centrally, allowing parallel writes while minimizing overall write amplification through efficient resource management.
2Device complexity
If storage nodes have minimal processing power, then hardware cost and complexity are reduced, but the ability to handle I/O traffic and execute virtualization logic is limited
Solution Approach 1:
The patent extracts the complex processing requirements from individual storage nodes and consolidates them into the LFS metadata management system. Storage nodes with minimal processing power only need to handle simple I/O operations and segment writes, while the metadata system handles complex tasks like parity calculations, free space management, and coordination of multi-writer operations. This extraction enables simple hardware to achieve high I/O throughput.
Solution Approach 2:
The patent creates a universal LFS metadata management system that handles multiple functions: coordinating writes from multiple objects, managing free segments across all storage nodes, calculating parity blocks, and tracking segment usage. This single multi-functional system serves all storage nodes, allowing each node to remain simple while the collective system achieves high productivity and handles complex I/O traffic efficiently.
Data Source
AI summary
Solutions for supporting storage using a multi-writer log-structured file system (LFS) are disclosed that include receiving incoming data from an object of a plurality of objects that are configured to simultaneously write to the LFS from different nodes; based at least on receiving the incoming data, determining whether sufficient free segments are available in a local segment usage table (SUT) for writing the incoming data; based at least on determining that insufficient free segments are available, requesting allocation of new free segments; writing the incoming data to a log; acknowledging the writing to the object; determining whether the log has accumulated a full segment of data; based at least on determining that the log has accumulated a full segment of data, writing the full segment of data to a first segment of the free segments; and updating the local SUT to mark the first segment as no longer free.