Dynamic Queue Slot Allocation for Delete I/O Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems with a tiered architecture, the efficient processing of delete I/O operations is challenging due to the need to free up storage spaces quickly, which is delayed when delete I/Os are not processed in time, especially in virtualized computing environments where delete I/Os must go through both the primary and secondary tiers.
Innovation Solution
A method is introduced that involves buffering delete I/O requests in a logical queue with dynamically adjustable slots, where a delete destage thread processes these requests independently of data write requests, and the number of slots dedicated to delete I/Os is adjusted based on parameters such as pending deletes and space usage to optimize processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If delete I/Os are processed sequentially through the tiered storage architecture (primary tier then secondary tier), then data consistency is maintained, but the time to free up storage spaces is excessively long
Solution Approach 1:
The patent segments the delete I/O processing into two independent parts: (1) metadata update in the primary tier which confirms the delete operation, and (2) actual data removal in the secondary tier which happens asynchronously. This segmentation allows the storage space to be freed in the primary tier immediately while the secondary tier continues cleanup in the background, resolving the contradiction between maintaining data consistency and quickly freeing storage spaces.
Solution Approach 2:
The patent performs preliminary action by updating the metadata and freeing storage spaces in the primary tier before the actual data deletion in the secondary tier is complete. The delete I/O is acknowledged and storage spaces are made available for reuse immediately, while the secondary tier continues to perform the actual data removal asynchronously, thus preliminarily achieving the space freeing goal without waiting for complete data deletion.
2Reliability
If the storage stack waits for delete I/O processing to complete before allowing subsequent write I/O, then data integrity is ensured, but the processing speed of subsequent write I/O is significantly delayed
Solution Approach 1:
The patent segments the I/O processing workflow into independent delete operations and write operations. Delete I/Os update metadata and free spaces without blocking subsequent write I/Os, which can immediately utilize the freed spaces. This segmentation eliminates the blocking relationship while maintaining data integrity through proper metadata management, thus resolving the contradiction between data integrity and write I/O speed.
Solution Approach 2:
The patent enables continuity of useful action by allowing subsequent write I/O operations to proceed without waiting for delete I/O processing to complete. The storage stack continuously accepts and processes write I/Os while delete operations continue independently in the background, ensuring that productive actions are not interrupted by cleanup operations, thereby maintaining both data integrity and high productivity.
3Device complexity
If a fixed number of slots are allocated for buffering delete I/Os in the logical queue, then queue management is simple, but the processing efficiency of delete I/Os cannot be optimized under varying workload conditions
Solution Approach 1:
The patent implements dynamic slot allocation for buffering delete I/Os in the logical queue. The number of slots is not fixed but adjusts based on workload conditions such as the number of pending delete operations and current system load. This dynamic adjustment allows the system to allocate more slots during high delete workloads to improve processing efficiency, while using fewer slots during low workloads to maintain simplicity, thus resolving the contradiction between queue management complexity and processing efficiency.
Data Source
AI summary
Example methods and systems to process input/output (I/O) requests in a distributed storage system in a virtualized computing environment are disclosed. One example method includes executing a first thread to destage one or more data writes, wherein the one or more data writes correspond to a first bucket; executing a second thread to destage the one or more data deletes, wherein the one or more data deletes correspond to a second bucket; in response to executing the first thread, buffering write I/Os associated with the one or more data writes in a logical queue; in response to executing the second thread, buffering delete I/Os associated with the one or more data deletes in the logical queue; and adjusting a number of slots in the logical queue dedicated to buffer the delete I/Os based on a relationship between the first bucket and the second bucket.