Disk Failure Prediction via Context-Validated Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for predicting disk failures in storage systems are inaccurate due to noise in training data and inability to distinguish between hardware and system-level failures, leading to potential data loss and system instability.
Innovation Solution
A method that filters operation and failure information using system context to validate failure data, allowing for the creation of a machine learning model that accurately predicts disk failures by distinguishing between valid and invalid failure information, thereby improving prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If failure information is used to train machine learning models for disk failure prediction, then prediction capability is improved, but prediction accuracy deteriorates due to noise and system-level failure confusion
Solution Approach 1:
The patent extracts and separates valid disk-level failure information from invalid system-level failure information through a validation mechanism. By taking out only the relevant failure data (where the storage system itself is operational), the system trains machine learning models on clean data, resolving the accuracy issue while maintaining automated prediction capability.
Solution Approach 2:
The patent introduces a validation mechanism as an intermediary between failure information collection and model training. This intermediary component filters and validates failure information by checking system context, acting as a mediator that prevents noisy or invalid data from reaching the machine learning models, thus improving prediction accuracy.
2Quantity of substance
If all failure information is used for training, then training data quantity is improved, but data quality deteriorates due to inclusion of system-level failures
Solution Approach 1:
The patent applies local quality by differentiating the treatment of different failure information based on their specific characteristics. Instead of uniformly processing all failure data, the system evaluates each piece of failure information against system context to determine its validity, ensuring high-quality data is selected for training while maintaining sufficient data quantity through targeted collection.
Solution Approach 2:
The patent converts the potentially harmful presence of system-level failure information into a beneficial validation opportunity. By encountering system-level failures in the training process, the system learns to distinguish and filter them out, transforming what was initially a data quality problem into a feature that improves the model's ability to identify valid failure patterns.
Data Source
AI summary
A method of facilitating prediction of a disk failure The method includes obtaining operation information and failure information associated with a plurality of disks, wherein the plurality of disks comprising a first set of disks indicated by the failure information as having a failure and a second set of disks indicated by the failure information as having no failure; determining validity of the failure information associated with a first set of disks based on system context information related to the storage system comprising the first set of disks; and in response to determining that the failure information associated with at least one of the first set of disks is valid, generating a machine learning model based on the operation information and the failure information associated with the first set of disks.