Decoupled I/O Information Collection Process for Storage Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The complexity of storage nodes in distributed mass data storage systems is increased due to integrated functions of collecting and analyzing I/O information, which necessitates improved architecture and additional collection codes, leading to resource occupancy issues.
Innovation Solution
A method and apparatus that collect I/O information independently of the storage engine, adjust processes based on resource usage thresholds, and send the information to a target node for analysis, potentially compressing data and using breakpoint files or timers to manage resource occupancy and reduce complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If collection and analysis functions are integrated into the storage engine architecture, then I/O information can be collected and analyzed, but the storage node complexity increases and resource occupancy increases
Solution Approach 1:
The patent divides the storage node into separate functional modules: the storage engine handles data storage operations independently, while an independent process collects I/O information. This segmentation removes the collection function from the storage engine architecture, preventing complexity increase while maintaining full collection capability.
Solution Approach 2:
The I/O information collection function is extracted from the storage engine and placed in an independent process. This extraction eliminates the need to modify the storage engine architecture and adds collection codes to all I/O processes, thereby reducing storage node complexity while preserving information collection.
2Loss of information
If collection codes are added to all I/O processes in the system, then I/O information can be collected, but the storage node complexity increases
Solution Approach 1:
The patent creates a universal independent process that handles I/O information collection for all storage nodes. This single process type can collect information from multiple storage engines across different nodes, providing multi-functionality without increasing individual node complexity. The independent process serves multiple purposes: collecting I/O information, monitoring resource usage, and managing data transmission.
3Loss of information
If the independent process continuously collects I/O information, then complete I/O information is obtained, but resource occupancy increases
Solution Approach 1:
The independent process implements periodic collection of I/O information rather than continuous collection. The process collects information at defined intervals and uses timers to manage collection cycles, reducing resource occupancy while ensuring complete I/O information is captured at each period. This periodic approach balances information completeness with resource efficiency.
Solution Approach 2:
The patent makes the collection process dynamic by implementing resource usage monitoring and adaptive adjustment. When resource usage exceeds thresholds, the system dynamically adjusts the collection frequency or suspends temporary collection, allowing the process to adapt to changing system conditions and maintain information collection while controlling resource occupancy.
Data Source
AI summary
Embodiments of the present disclosure provide a method and an apparatus for processing I/O information of data, a method and an apparatus for analyzing I/O information of data, and related devices. The method includes: collecting I/O information of data in a storage engine by a process independent of the storage engine in a storage node; performing adjustment on the process in response to monitoring that a variation in usage of resource in the storage node exceeds a threshold, the adjustment being used to decrease a resource occupancy of the process; and sending the I/O information collected by the process to a target node, to enable the target node to analyze the I/O information. Embodiments of the present disclosure may reduce the complexity of the storage node, and avoid adding a load to the storage engine.