Proactive Storage Failure Prediction via Streaming Telemetry
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems often encounter execution errors that require immediate resolution to restore operational state, leading to costly dial-home events and customer experience impact.
Innovation Solution
Implementing streaming machine telemetry (SMT) event counters in critical code paths and using a time series deep learning model to predict errors before they occur, enabling proactive service requests and reducing downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dial-home events are implemented to detect and notify failures, then failure detection capability is improved, but system downtime and customer experience deteriorate
Solution Approach 1:
The system performs preliminary actions by analyzing telemetry data patterns and predicting potential failures before they occur. The failure prediction module continuously monitors system metrics and identifies trends that precede failures, enabling proactive intervention before the actual failure happens, thus avoiding downtime associated with reactive dial-home events.
Solution Approach 2:
The system implements feedback mechanisms where telemetry data from system operations is continuously collected, analyzed, and fed back into the prediction model. This feedback loop allows the system to learn from historical data and improve its prediction accuracy over time, enabling more reliable early warning of potential failures without triggering unnecessary dial-home events.
2Ease of repair
If reactive support response is implemented after failures occur, then immediate corrective action is possible, but productivity and system availability deteriorate
Solution Approach 1:
The system performs preliminary diagnostic actions by analyzing telemetry patterns that indicate impending failures. Before the actual failure occurs, the system can prepare diagnostic information and alert support teams in advance, allowing them to prepare appropriate corrective actions and spare parts, thus reducing the actual repair time when the failure occurs.
Solution Approach 2:
The system performs self-diagnosis by automatically analyzing its own telemetry data to identify potential failure modes. This self-service capability reduces the burden on support teams by pre-characterizing the issue, allowing them to focus on executing known corrective procedures rather than diagnosing the problem from scratch.
Data Source
AI summary
Streaming machine telemetry (SMT) event counters are placed in critical code paths of software executing on a storage system. Each monitoring interval the values of the SMT counters are reported. When a critical error occurs on the storage system, a time series set of SMT counters from a set of previous monitoring intervals is labeled with the error type and used as a training example for a learning process. The learning process is trained to learn to learn recursions between time series sets of SMT counter values and labeled error types. Once trained, a checkpoint of the learning model is deployed as an inference model and used to predict the likely occurrence of errors before the errors occur. Predicted errors are logged into a proactive service request queue, and feedback related to the predicted errors are used as feedback to continue training the learning process.