Append-Only Data Log for Fast Random Access Erasure Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face challenges in achieving high availability and low costs while maintaining performance, particularly in managing data redundancy and handling write operations efficiently, especially with erasure coding techniques that require complex atomic operations across multiple storage nodes.
Innovation Solution
The system employs an append-only data log with a log-structure merge tree for efficient data storage and retrieval, using a mapping structure to manage data chunks and parity blocks across multiple disks, allowing for fast random access and garbage collection to maintain storage efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If erasure coding is implemented to provide data redundancy and high availability, then data reliability is improved, but storage capacity requirements and operational complexity increase
Solution Approach 1:
The data is segmented into blocks and distributed across multiple storage nodes. Each write operation is divided into independent block operations, allowing parallel processing and reducing the impact of any single block failure. The mapping structure segments the address space into blocks that can be independently managed and tracked.
Solution Approach 2:
A mapping structure acts as an intermediary layer between the logical file system and physical storage blocks. This mapping structure tracks which blocks are valid, which are stale, and their locations, enabling efficient garbage collection and reducing the need for excessive redundancy while maintaining data availability.
2Reliability
If erasure coding with multiple parity blocks is used to survive multiple block losses, then data reliability is improved, but write operation complexity and processing time increase
Solution Approach 1:
The system pre-allocates blocks and pre-computes mapping relationships before actual data writes. The mapping structure is prepared in advance with block assignments, so when data needs to be written, the system can immediately proceed with the pre-planned block operations without complex real-time computation, reducing write operation time.
Solution Approach 2:
The mapping structure automatically tracks and manages block validity states, enabling self-service garbage collection. When blocks become stale, the system automatically identifies and reuses them without requiring complex external coordination, reducing processing overhead for write operations.
3Stability of the object's composition
If atomic operations are enforced across multiple storage nodes for erasure coding, then data consistency is improved, but operation complexity and coordination overhead increase
Solution Approach 1:
The atomic operation requirement is segmented and applied independently to each block rather than requiring coordinated atomic operations across all blocks simultaneously. Each block operation is independent and can proceed without blocking other block operations, reducing coordination complexity while maintaining overall data consistency through the mapping structure.
Solution Approach 2:
The mapping structure maintains copies of block assignment information and validity states. These copies enable the system to track data consistency without requiring complex real-time coordination between storage nodes, as each node can independently consult its local mapping information to determine the correct state.
4Reliability
If data is spread across multiple disks with parity bits for error detection and reconstruction, then data reliability is improved, but storage efficiency and access performance deteriorate
Solution Approach 1:
Error detection and parity operations are segmented to operate on individual blocks independently rather than requiring processing of entire files or large data sets. This block-level segmentation enables efficient parallel processing and improves storage efficiency by allowing selective operations on only the affected blocks.
Solution Approach 2:
The mapping structure serves as an intermediary that optimizes access patterns to parity and data blocks. It tracks the locations and states of all blocks, enabling efficient retrieval of only the necessary blocks for error detection or reconstruction, improving access performance while maintaining reliability.
Data Source
AI summary
A system and method is provided for fast random access erasure encoded storage. An exemplary method includes writing data to an append-only data log that includes data log extents that are each associated with data that is mapped to corresponding offset range of a virtual file of a client and storing the append-only data log as a sequence of data chunks each allocated on one or more one storage disks. Moreover, the method determines an amount of useful data in one or more data chunks and, when the amount of useful data in the data chunk is less than a predetermined threshold, appending the useful data from the data chunk to an end of the append-only data log. Finally, the data log is cleaned by releasing the one or more data chunk from the append-only data log after the useful data is appended to the append-only data log.