Disaggregated Storage Reclamation With QPC-Guided Node Pairing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In disaggregated storage systems, compaction processes lead to redundant data and unpredictable Quality of Service (QoS) impacts due to dynamic, real-time data movement, which is inefficient and affects cluster resources without effective endurance management.
Innovation Solution
A method and system for dynamically reclaiming storage space by identifying source and destination nodes based on delta ranges of data and endurance levels, using Quality-of-service Penalty Coefficient (QPC) to optimize data transfer paths, ensuring minimal resource consumption and maintaining QoS.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compaction is performed to reclaim storage space, then storage space utilization is improved, but Quality of Service (QoS) becomes unpredictable and cluster resources are consumed
Solution Approach 1:
The system performs preliminary identification of compactable storage segments before actual compaction occurs. By pre-analyzing storage segment characteristics, validity ratios, and endurance levels, the system prepares a compacted segment list in advance, allowing compaction to be scheduled during low-demand periods and preventing QoS degradation during critical operations.
Solution Approach 2:
The system dynamically adjusts compaction strategies based on real-time storage system state. It continuously monitors storage segment validity ratios, endurance levels, and QoS metrics, then adapts compaction parameters and segment selection accordingly. This dynamic adjustment ensures compaction operations respond to changing conditions while maintaining QoS requirements.
2Quantity of substance
If compaction moves data between storage segments, then storage space is reclaimed, but cluster resources such as CPUs and network are consumed
Solution Approach 1:
The system applies different compaction strategies to different storage segments based on their local characteristics. By analyzing validity ratios and endurance levels of individual segments, it identifies which segments are most suitable for compaction and directs operations accordingly. This localized approach minimizes unnecessary data movement across the entire cluster, reducing overall resource consumption while achieving effective space reclamation.
Solution Approach 2:
The system enables storage segments to self-identify their compaction suitability based on their own validity ratios and endurance levels. Segments with high invalid data ratios and low endurance levels are automatically prioritized for compaction, eliminating the need for centralized decision-making and reducing CPU overhead. This self-service mechanism optimizes resource usage by leveraging the segments' intrinsic properties.
3Quantity of substance
If compaction is performed as a background operation, then storage space is reclaimed, but QoS functionality may be affected in unknown ways
Solution Approach 1:
The system implements continuous feedback loops that monitor QoS metrics during and after compaction operations. By measuring actual QoS performance and comparing it against target thresholds, the system can detect negative impacts and adjust compaction parameters in real-time. This feedback mechanism ensures that compaction operations remain within acceptable QoS boundaries and enables iterative optimization of the compaction process.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present disclosure relates to a method and a system for dynamically reclaiming storage space in disaggregated storage system. The method includes retrieving a plurality of data levels and a plurality of endurance levels of a plurality of storage nodes, determining first delta range based on one or more first parameters associated with the plurality of data levels and workload, determining second delta range based on one or more second parameters associated with the plurality of endurance levels and workload, identifying one or more source nodes and one or more destination nodes from plurality of storage nodes based on first delta range and second delta range respectively, identifying set of storage node pairs, among one or more source nodes and one or more destination nodes, based on Quality-of-service Penalty Coefficient (QPC), performing reclamation of at least one storage segment among the set of storage node pairs.