Shared Layered Storage Write Request Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems face challenges in managing a large number of write requests efficiently due to high latency and traffic congestion within the internal communication lines between control nodes, especially when acknowledging write requests to hosts before data is fully processed and stored in slower secondary storage layers.
Innovation Solution
Implementing a method where a receiving control node initially stores data in a fast, shared first-layer storage, while a destaging control node handles the slower second-layer storage, with dynamic load balancing and virtual address-based allocation to distribute destaging responsibilities among control nodes, reducing the need for immediate notification between nodes and optimizing data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored directly in second layer storage before acknowledgment, then data durability is improved, but write latency increases
Solution Approach 1:
The system performs preliminary storage of data units in the first layer storage before the write operation is complete. This allows the system to acknowledge the write request to the host earlier, reducing write latency, while still ensuring data durability through subsequent destaging to the second layer storage.
Solution Approach 2:
The storage system is divided into two layers: first layer storage for fast temporary storage and second layer storage for permanent durable storage. This segmentation allows different parts of the storage system to serve different purposes - the first layer provides speed for acknowledgment while the second layer provides durability.
2Device complexity
If all control nodes handle both receiving and destaging operations, then system simplicity is maintained, but internal communication traffic increases
Solution Approach 1:
Multiple control nodes are merged into a collaborative system where they share the destaging workload. Instead of each node handling all its own destaging operations independently, the system combines their efforts by allowing any control node to perform destaging for any data unit, reducing redundant internal communication.
Solution Approach 2:
Control nodes are designed with multi-functionality, capable of both receiving write requests and performing destaging operations. This universality allows the system to flexibly assign destaging tasks to any available control node, optimizing resource utilization and reducing internal communication overhead.
3Reliability
If notification between receiving and destaging control nodes is performed for each data unit, then data consistency is ensured, but processing overhead increases
Solution Approach 1:
Instead of notifying the destaging control node for every single data unit, the system implements periodic or batched notification. The receiving control node accumulates multiple data units before triggering a destaging operation, reducing the frequency of notifications and improving processing throughput while maintaining data consistency.
Solution Approach 2:
The system performs partial notification by only notifying the destaging control node when certain conditions are met, such as when a threshold number of data units are ready for destaging. This partial action approach reduces unnecessary notifications while ensuring that destaging occurs when sufficient data is available.
Data Source
AI summary
A method for managing write requests in a storage system, the method may include receiving, by a receiving control node out of a group of control nodes of the storage system, a write request for writing a data unit to the storage system; storing the data unit in a first layer storage that is shared among the control nodes of the group, by the receiving control node; retrieving the data unit from the first layer storage and destaging the data unit to a second layer storage by a destaging control node that belongs to the group and is responsible for destaging the data unit; wherein at a time of the destaging of the data unit to the second layer storage another control node of the group is responsible for destaging another data unit to the second layer storage; wherein the storing of the data unit in the first layer storage is faster than the destaging of the data unit to the second layer storage; and when the destaging control node differs from the receiving control node then each one of the receiving and the storing is not preceded by notifying the destaging control node about the receiving and the storing, respectively.