Container Flush Ownership Assignment in Dual Node Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in efficiently assigning flush ownership of containers for logged writes, particularly in non-ALUA based configurations, which can lead to suboptimal cache locality and concurrent flushing issues.
Innovation Solution
A method is introduced that dynamically determines flush ownership based on observed write operation counts and metadata mapping, ensuring that the node with the highest count of ingested write operations is designated as the flush owner for each container, optimizing cache locality and avoiding concurrent flushing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If flush ownership is assigned in non-ALUA based configurations, then data storage system can handle multiple host systems accessing single data storage system, but cache locality becomes suboptimal and concurrent flushing issues occur
Solution Approach 1:
The system dynamically determines flush ownership based on observed write operation counts rather than static configuration. Each node maintains counters for write operations received for each container, and the node with the highest counter value becomes the flush owner. This dynamic approach adapts to actual data access patterns, optimizing cache locality while maintaining multi-host access capability.
Solution Approach 2:
The system implements feedback mechanisms where nodes continuously monitor and report write operation counts to a quorum manager. The quorum manager uses this feedback information to make informed decisions about flush ownership assignment. This feedback loop ensures that the system maintains synchronized ownership views across all nodes, preventing concurrent flushing issues.
2Productivity
If multiple nodes maintain write counters and determine flush ownership dynamically, then cache locality is optimized, but system complexity increases due to counter management and synchronization requirements
Solution Approach 1:
The system segments the flush ownership determination process into distinct components: individual nodes maintain local write counters for their received operations, a quorum manager collects and processes counter information from multiple nodes, and the quorum manager makes the final ownership assignment. This segmentation distributes the complexity across multiple specialized components rather than requiring a single complex mechanism.
Solution Approach 2:
The quorum manager acts as an intermediary between individual nodes and the flush ownership determination process. Nodes report their write counters to the quorum manager, which then determines the flush owner based on the highest counter value. This intermediary simplifies the overall system by centralizing the decision-making logic and providing a clear coordination point that reduces synchronization complexity.
3Reliability
If the system uses quorum-based ownership determination with hash functions, then ownership assignment becomes deterministic and concurrent access is prevented, but processing overhead increases
Solution Approach 1:
The system performs preliminary actions by maintaining write counters continuously as write operations are received, before flush operations are needed. This way, when flush ownership determination is required, the system already has up-to-date counter information available, eliminating the need for additional counting operations at the moment of decision-making and reducing processing time.
Solution Approach 2:
The system uses hash functions to create deterministic copies of ownership assignments based on container identifiers. This allows nodes to independently verify ownership assignments without requiring continuous communication, reducing the processing overhead associated with consensus mechanisms while maintaining reliability and preventing concurrent access conflicts.
Data Source
AI summary
A protocol for processing write operations can include recording each write operation in a log using a PDESC (page descriptor)-PB (page block) pair. The log entry for the write operation can be included in a container of logged writes. In a dual node system, the protocol when processing the write operation, that writes first data, can include incrementing a corresponding one of two counters of the container, where the corresponding counter is associated with one of the system's nodes which received the write operation and and caches the first data. Each container can be associated with an logical block address (LBA) range of a logical device, where logged writes that write to target addresses in the particular LBA range are included in the container. Nodes can independently determine flush ownership using the container's counters and can flush containers based on the flush ownership.