Storage System Security Monitoring with Unsupervised Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional storage systems face inefficiencies in data management and security threat monitoring, particularly in handling large volumes of data across distributed networks, where existing solutions lack robustness in data redundancy and failover mechanisms.
Innovation Solution
The implementation of a storage system with dual Peripheral Component Interconnect (PCI) flash storage, non-volatile random access memory (NVRAM), and erasure coding schemes to ensure data redundancy and high availability, along with a security threat monitoring system utilizing unsupervised and supervised machine learning models for real-time threat detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional storage systems are used for data management, then device complexity is reduced, but data storage reliability and security are insufficient
Solution Approach 1:
The storage system is segmented into multiple independent storage devices, each with its own controller. Data is divided into fragments and distributed across these devices, with each device operating semi-independently. This segmentation improves reliability through distribution while managing complexity by creating modular, self-contained units.
Solution Approach 2:
The system implements erasure coding that encodes data with redundant information beforehand, creating a cushion against data loss. When failures occur, the pre-encoded redundant data enables recovery without requiring complex real-time reconstruction, thus improving reliability while keeping the recovery mechanism relatively simple.
2Reliability
If erasure coding schemes are implemented for data redundancy, then data storage reliability is improved, but processing time increases
Solution Approach 1:
Erasure coding is performed as a preliminary action during data writing operations. The encoding process completes before data is committed to storage, so that redundancy is already in place before any potential failure occurs. This eliminates the need for time-consuming real-time encoding during read or recovery operations.
Solution Approach 2:
The erasure coding process is segmented and distributed across multiple storage devices and controllers. Each device performs a portion of the encoding work independently, parallelizing the process to reduce overall processing time while maintaining the reliability benefits of comprehensive erasure coding.
3Speed
If dual PCI flash storage and NVRAM are used, then data recovery speed is improved, but device complexity increases
Solution Approach 1:
Different storage media with different performance characteristics are assigned to different functions: dual PCI flash storage provides high-speed data recovery for active operations, while NVRAM provides persistent storage for critical metadata and control information. This local optimization of storage quality improves recovery speed for time-critical operations without requiring the entire system to be over-engineered.
Solution Approach 2:
Multiple storage technologies (PCI flash storage and NVRAM) are merged into a unified storage hierarchy that works together seamlessly. The system combines the high speed of flash storage with the non-volatility of NVRAM, achieving fast recovery speeds while maintaining data persistence, and presenting a unified interface that masks the underlying complexity.
4Reliability
If machine learning models are deployed for threat detection, then security monitoring capability is improved, but computational resource consumption increases
Solution Approach 1:
The system deploys machine learning models selectively for threat detection rather than applying comprehensive ML analysis to all data operations. Supervised learning models are used for known threat patterns where high accuracy is needed, while unsupervised models handle anomaly detection only when triggered by specific conditions, reducing overall computational resource consumption while maintaining effective security monitoring.
Data Source
AI summary
An exemplary security threat monitoring system receives performance metric data representative of a performance metric for a storage system, applies the performance metric data as an input to an unsupervised machine learning model, and identifies, based on an output of the unsupervised machine learning model, an anomaly in the performance metric data.