Dynamic Soft-Reserve Space Management for Mixed SSD HDD Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data warehouse systems with mixed SSD and HDD storage, the migration of hot data to fastest locations often leaves little space available for spool allocation, leading to performance issues due to fixed soft-reserve settings that can be either too large or too small, and are not adaptable to varying customer workloads or system reconfigurations.
Innovation Solution
A dynamic method to adjust the soft-reserve space as a percentage of total storage by maintaining a spool constant metric (SCM) that updates based on real-time spool use measurements, allowing for responsive and customizable allocation of storage resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If hot data is migrated to fastest storage locations (SSDs), then data access performance is improved, but space availability for spool allocation deteriorates
Solution Approach 1:
The storage space is segmented into different zones: a soft-reserve zone specifically reserved for spool allocation and a general data storage zone for hot and cold data. This segmentation ensures that spool operations have dedicated space independent of data migration activities, resolving the conflict between data access performance and spool space availability.
Solution Approach 2:
The soft-reserve zone acts as an intermediary mechanism between the data storage system and spool allocation needs. It provides a buffer zone that mediates the conflict by ensuring spool space is always available regardless of how much space data migration occupies in the general storage area.
2Ease of operation
If fixed soft-reserve space is allocated, then spool allocation is simplified, but adaptability to varying workloads deteriorates
Solution Approach 1:
The soft-reserve space configuration transitions from a fixed static value to a dynamic adjustable parameter. The system can now adapt the soft-reserve percentage based on actual workload characteristics, allowing it to maintain simplicity in operation while gaining flexibility to handle varying workload demands effectively.
Solution Approach 2:
The soft-reserve space is defined by a configurable parameter (percentage of total storage) that can be changed based on workload conditions. This parameter change approach allows the system to maintain ease of operation through simple parameter adjustment rather than complex reconfiguration, while achieving adaptability to different workload scenarios.
3Reliability
If soft-reserve space is set too large, then spool performance is ensured, but storage efficiency deteriorates
Solution Approach 1:
Instead of allocating excessive soft-reserve space that wastes storage capacity, the system applies partial action by calculating the minimum sufficient soft-reserve percentage needed for reliable spool operations. This partial allocation approach ensures spool performance reliability while avoiding the waste associated with overly aggressive reserve space allocation.
4Quantity of substance
If soft-reserve space is set too small, then storage efficiency is improved, but spool performance deteriorates
Solution Approach 1:
The system incorporates feedback mechanisms to monitor spool operation characteristics and adjust the soft-reserve percentage accordingly. By continuously measuring actual spool usage patterns and performance, the system can dynamically tune the reserve space to ensure adequate performance while optimizing storage efficiency, avoiding both over-allocation and under-allocation.
Data Source
AI summary
A system and method for managing soft reserve space within a database system including fast solid state storage devices (SSDs) and traditional hard drive storage device (HDDs). The system and method monitors actual spool use within the database system and dynamically modifies the soft-reserve space as a percentage of total storage space in response to changes in spool use.