Append-Only Storage Extent State Management for Lock-Free Repairs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based storage systems face performance issues during repair operations due to the need for locks to prevent updates, which can complicate data block retrieval and replication, leading to inefficiencies in accessing and managing data blocks across multiple storage devices.
Innovation Solution
The implementation of an append-only data storage system that manages extents in either an open or closed state, allowing or disallowing specific operations based on the state, and performing synchronization to ensure data consistency across copies, thereby eliminating the need for locks and enhancing operational efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If locks are used to prevent updates during repair operations, then data integrity is maintained, but performance degradation occurs due to blocked access
Solution Approach 1:
The system dynamically changes the state of extents between open and closed based on operational needs. Open extents allow concurrent appends while closed extents enable repair operations without locks. This dynamic state transition resolves the contradiction by allowing the system to adapt its behavior to current operational requirements rather than using static locking mechanisms.
Solution Approach 2:
The system changes the state parameter of extents from open to closed to alter their operational characteristics. When an extent is closed, it prevents new appends and enables repair operations. This parameter change allows the system to maintain data integrity during repairs without requiring locks that would block application access.
2Reliability
If multiple copies of an extent are maintained for fault tolerance, then reliability is improved, but synchronization complexity increases
Solution Approach 1:
The system performs synchronization as a preliminary action when transitioning from open to closed state. By synchronizing all copies before closing the extent, the system ensures consistency across replicas before repair operations begin. This preliminary synchronization simplifies subsequent repair operations and avoids complex coordination during critical operations.
Solution Approach 2:
The system maintains multiple copies of extents for fault tolerance and uses the closed state mechanism to manage these copies. When an extent is closed, all copies are synchronized and then repaired independently without requiring complex inter-copy coordination during repair operations, reducing synchronization complexity.
3Adaptability or versatility
If repair operations are performed on open extents, then operational flexibility is maintained, but data consistency cannot be guaranteed
Solution Approach 1:
The system dynamically transitions extents from open to closed state when repair operations are needed. This dynamic state change allows the system to maintain operational flexibility during normal operations while ensuring data consistency during repair operations by preventing new appends to closed extents.
Solution Approach 2:
The system performs a preliminary action of closing the extent before repair operations begin. This preliminary anti-action (preventing new appends) ensures that no new data is written during repair operations, guaranteeing data consistency while still allowing the system to be flexible about when and how repairs are performed.
Data Source
AI summary
The disclosed embodiments relate to the design of an append-only data storage system that stores sets of data blocks in extents that are located in storage devices in the system. During operation of the system, when an extent is in an open state, the system allows data blocks to be appended to the extent, and disallows operations to be performed on the extent that are incompatible with data being concurrently appended to the extent. When the extent becomes full, the system changes the extent from the open state to a closed state. Then, while the extent is in the closed state, the system disallows data blocks to be appended to the extent, and allows operations to be performed on the extent that are incompatible with data being concurrently appended to the extent.