Storage System Performance Jitter Identification via Thread Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Debugging IO performance-jitter in high-end storage systems is challenging due to the multiplicity and diversity of flows and complex dependencies between them, making it difficult to identify and address performance degradation effectively.
Innovation Solution
Implementing flow-level system monitoring with correlation and clustering analysis to automate the performance-jitter problem-solving process, which involves generating threads to operate the storage system, updating performance counters, creating performance snapshots, identifying intervals of degradation, and determining correlations between tasks and performance issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If flow-level monitoring and correlation analysis are implemented to identify performance degradation, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent segments the complex storage system into multiple independent threads, each associated with specific tasks. By monitoring and analyzing individual thread performance counters separately, the system can identify performance degradation sources without being overwhelmed by the complexity of the entire system. This segmentation allows precise measurement of specific thread behaviors while managing overall system complexity.
Solution Approach 2:
The patent introduces performance counters as intermediary elements that mediate between thread operations and performance analysis. These counters serve as intermediaries that capture thread state changes and make them analyzable, enabling precise performance measurement without directly complicating the core thread execution logic.
2Loss of information
If performance snapshots are generated by sampling thread states and counters, then information about performance degradation is captured, but loss of time occurs due to sampling overhead
Solution Approach 1:
The patent performs preliminary actions by maintaining performance counters that continuously track thread states in the background. This preliminary data collection allows snapshots to be generated quickly by simply reading pre-collected counter values, rather than gathering data during the snapshot process itself, thus minimizing time loss.
Solution Approach 2:
The patent creates copies of performance data through snapshots that capture thread states and counter values at specific moments. These snapshots are lightweight copies that can be generated and analyzed without significantly impacting the original system performance, enabling comprehensive performance monitoring with minimal time overhead.
3Productivity
If correlation analysis is performed between tasks and performance degradation intervals, then productivity in problem solving is improved, but difficulty of detecting and measuring increases
Solution Approach 1:
The patent implements feedback mechanisms by continuously monitoring performance counters and comparing them against expected ranges. When performance degradation is detected, the system provides feedback about which thread counters deviate from normal behavior, making it easier to identify correlated tasks without requiring complex manual analysis.
Solution Approach 2:
The patent enables self-service through automated correlation analysis that uses clustering algorithms to automatically identify relationships between thread activities and performance degradation. This automation reduces the difficulty of detecting correlations by having the system perform the complex analysis work itself, improving problem-solving productivity without requiring extensive manual intervention.
Data Source
AI summary
A computer implemented method including generating one or more threads to operate a storage system, the storage system including one or more system resources, and each of the one or more threads associated with one of a plurality of tasks of the storage system. The method includes updating one or more performance counters of the storage system in response to a change in operating state of one of the one or more threads. The method includes generating a performance snapshot by sampling a state of the tasks associated with the threads and the performance counters, identifying one or more intervals of performance degradation of the storage system from the performance snapshot, and determining a correlation between one or more of the plurality of tasks and the one or more identified intervals of performance degradation.