Storage Control System for Resource Redistribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In-line data compression in storage systems configured for 'in-place-updates' faces challenges due to variable compression sizes, leading to potential gaps in storage space allocation, which existing log-structured array (LSA) storage layouts attempt to address through out-of-place updates but at the cost of increased write amplification and storage media wear.
Innovation Solution
A storage control system that manages processing groups across server nodes based on resource utilization thresholds, redistributing processing groups from overloaded nodes to underutilized ones to optimize resource allocation and reduce bottlenecks, while maintaining efficient data placement and defragmentation within log-structured arrays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If in-line data compression is implemented in storage systems configured for in-place-updates, then storage utilization is enhanced, but storage space allocation gaps are created due to variable compression sizes
Solution Approach 1:
The storage system divides data into fixed-size blocks before compression, and uses a log-structured array to organize compressed data into fixed-size log segments. This segmentation allows the system to handle variable compression sizes by allocating entire log segments rather than individual data blocks, eliminating allocation gaps while maintaining high storage utilization.
Solution Approach 2:
The system changes the allocation unit from variable-size compressed data blocks to fixed-size log segments. By transforming the allocation parameter from data-centric to segment-centric, the system resolves the conflict between variable compression sizes and fixed storage allocation, enabling efficient in-place updates without creating allocation gaps.
2Adaptability or versatility
If log-structured array storage layout is used to accommodate variable compression sizes, then storage flexibility is improved, but write amplification increases and storage media wear increases
Solution Approach 1:
The system pre-allocates fixed-size log segments for storing compressed data before actual data writes occur. By preparing the storage structure in advance with predetermined segment boundaries, the system enables direct writes to specific segments without requiring dynamic reallocation or creating gaps, thereby reducing write amplification while maintaining flexibility for variable compression sizes.
Solution Approach 2:
The system uses metadata copies and pointers to reference data in log segments rather than moving actual data blocks frequently. When updates are needed, the system creates new log entries with updated data and updates metadata pointers, avoiding the need to physically relocate large amounts of data and reducing write amplification on the storage media.
3Ease of operation
If out-of-place updates are performed to rewrite updated compressed data into new log entries, then data update capability is maintained, but storage media wear increases
Solution Approach 1:
The system applies different update strategies to different parts of the storage structure. Metadata updates are performed frequently and locally by updating pointers and references, while actual data writes to log segments are performed less frequently and more efficiently by reusing existing segments. This local differentiation reduces overall write operations on storage media while maintaining full data update capability.
Solution Approach 2:
The system discards outdated log entries and recovers their space for future use through a log segment management mechanism. When log segments become full, they are marked as invalid and eventually reclaimed for new data writes. This approach allows the system to perform out-of-place updates efficiently by recycling storage space without requiring continuous growth of the storage structure, thereby reducing media wear.
Data Source
AI summary
At least one processor is configured to obtain measurement information comprising an indication of an amount of utilization of a hardware resource of a first server node by a plurality of processing groups and to determine that the amount of utilization of the hardware resource is above a threshold amount of utilization. The at least one processor is further configured to select a given processing group for redistribution based at least in part on the determination that the amount of utilization of the hardware resource is above the threshold amount and on an amount of utilization of the hardware resource by the given processing group. The at least one processor is further configured to determine that a second server node comprises enough available capacity of the hardware resource and to redistribute the given processing group to the second server node based at least in part on the determination.