Cold Hot Data Identification Threshold Calculation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current hierarchical storage systems lack a uniform criterion for identifying cold and hot data across storage nodes, limiting the practicality and availability of hierarchical data storage and migration.
Innovation Solution
A method for calculating a uniform identification threshold for distinguishing cold and hot data by aggregating IO statistical information from multiple storage nodes, using IO count and sequentiality values to determine the threshold, and implementing data migration across nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If distributed cold and hot data identification function modules are deployed on storage nodes, then hierarchical storage can be implemented on individual nodes, but uniform identification criterion across the entire storage system cannot be achieved
Solution Approach 1:
A storage management server is introduced as an intermediary component that collects IO statistical information from multiple storage nodes, performs centralized threshold calculation, and distributes unified identification criteria back to all nodes. This mediator enables system-wide uniformity while preserving distributed hierarchical storage capabilities.
Solution Approach 2:
The threshold calculation function is merged from individual storage nodes to a centralized storage management server. By combining IO statistical information from all nodes and performing unified threshold calculation, the system achieves consistent cold and hot data identification criteria across the entire storage system.
2Productivity
If storage system capacity and performance are increased to meet requirements, then performance and capacity needs are satisfied, but system cost increases
Solution Approach 1:
Different storage tiers with distinct performance characteristics are assigned to different data types. Hot data is stored on high-performance storage mediums while cold data is stored on lower-performance, lower-cost mediums. This local quality differentiation optimizes performance for frequently accessed data while reducing costs for less accessed data.
Solution Approach 2:
The storage system is segmented into multiple hierarchical tiers based on data access patterns. By dividing storage resources into hot data storage and cold data storage tiers, the system achieves cost-effective performance optimization without requiring all storage capacity to be high-performance.
3Adaptability or versatility
If data migration across storage nodes is enabled, then system-wide hierarchical storage is achieved, but system complexity increases
Solution Approach 1:
The storage management server acts as a mediator that handles the complexity of cross-node data migration. It calculates unified identification thresholds, determines which data blocks require migration, and coordinates the migration process, thereby simplifying the overall system architecture while enabling system-wide hierarchical storage.
Solution Approach 2:
The system implements feedback mechanisms where storage nodes report IO statistical information to the management server, which then adjusts identification thresholds and triggers appropriate data migration actions. This feedback loop enables adaptive data migration across nodes while maintaining system-wide consistency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method, an apparatus, and a system for calculating a identification threshold to distinguish cold data and hot data. The method includes: sending an IO statistical information query command for a data block to each storage node; receiving IO statistical information sent by each storage node; performing an addition operation on IO count field values in a same count range in the received IO statistical information of all the storage nodes, to generate an IO count statistical value corresponding to each count range; and obtaining, according to a total capacity of hot data storage mediums, a size of data blocks, and the IO count statistical values on all the storage nodes and when the hot data storage mediums are filled with the data blocks corresponding to large IO count statistical values, a minimum IO count value in a count range corresponding to a minimum IO count statistical value in the large IO count statistical value, and using the minimum IO count value as a identification threshold to distinguish cold data and hot data. In the present invention, all storage nodes in a storage system, independently but cooperatively, implement hierarchical data storage in the entire storage system.