Monitoring Data Volume Reduction in Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face challenges in efficiently managing large amounts of system monitoring data, which occupy storage space and consume computing resources, with existing solutions being inflexible and reducing storage performance.
Innovation Solution
A method and system that adjusts or disables monitoring sampling through a monitoring unit and recording processor, using a log module to render and store logs, and an adjustment module to apply mechanisms like random, threshold, and priority adjustments to reduce the amount of stored monitoring data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all system monitoring data is retained for long-term storage, then complete historical monitoring information is preserved, but storage space is occupied and computing resources are consumed
Solution Approach 1:
The patent extracts only the essential and critical monitoring data for long-term retention, separating it from the large volume of routine monitoring data. The log module identifies and stores only significant events (errors, warnings, critical operations) while discarding redundant information, thereby preserving important information without occupying excessive storage space.
Solution Approach 2:
The system dynamically adjusts monitoring parameters such as sampling intervals, retention periods, and detail levels based on system state and importance. Critical parameters are monitored continuously with high detail, while non-critical parameters use reduced sampling rates, optimizing the balance between information retention and storage consumption.
2Quantity of substance
If monitoring data is compressed to reduce storage space, then storage capacity is optimized, but computing resources are consumed during compression
Solution Approach 1:
The system performs preliminary filtering and selection of monitoring data before storage, identifying only critical information that requires retention. This pre-processing avoids the need for heavy compression algorithms later, as the data volume to be stored is already minimized through selective logging of only essential monitoring events.
3Measurement precision
If monitoring sampling is performed continuously at high frequency, then complete system state information is captured, but large amounts of data are generated
Solution Approach 1:
The monitoring sampling frequency and detail level are dynamically adjusted based on system conditions and data importance. During normal operation, sampling occurs at lower frequencies to reduce data volume. When critical events or anomalies are detected, the system automatically increases sampling rates and detail levels to capture complete system state information, thereby maintaining measurement precision while minimizing overall data generation.
4Loss of information
If all monitoring operations are processed and stored in detail, then complete operational history is maintained, but storage performance is reduced
Solution Approach 1:
The system extracts and stores only the essential operational history information that has genuine retention value, separating critical events from routine operations. The log module filters monitoring data to retain only significant events (errors, warnings, critical operations) while summarizing or discarding routine operational data, maintaining operational history completeness for important events without degrading storage performance through excessive data volume.
Data Source
AI summary
The present disclosure provides a method for information storage and a system thereof, which adapts to a data storage system. A monitoring unit is configured to detecting and monitoring operations of a storage node in the data storage system to generate corresponding one and more monitoring data. A recording processor is configured to receiving the one or the plurality of monitoring data, and rendering one or a plurality of logs according to the difference of content of the one or the plurality of monitoring data. The adjustment mechanism is performed according to the stored logs, thereby the amount of large data generated during monitoring is effectively reduced.