Autonomous Data Protection via ML Anomaly Offloading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication techniques are insufficient for comprehensive data protection, as they are outweighed by operational overhead and do not adequately safeguard against data corruption, compromise, or loss, especially with growing data volumes.
Innovation Solution
Implementing autonomous data protection using machine learning engines for anomaly detection, which take continuous snapshots of data storage elements, model change rate trends, and offload identified anomalies to remote storage based on memory capacity, bandwidth, and service level priorities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If current data replication techniques are used, then data protection is provided, but operational overhead increases and data protection becomes insufficient for growing data volumes
Solution Approach 1:
The system performs autonomous data protection operations by automatically detecting anomalies through machine learning models, dynamically allocating bandwidth, and offloading data to remote storage without human intervention. The anomaly detection learning engine continuously monitors storage devices and autonomously determines when and where to replicate data, eliminating the need for manual configuration and management of data protection operations.
Solution Approach 2:
The system dynamically adjusts bandwidth allocation parameters based on real-time conditions. It monitors storage device performance states, service level priorities, and predicted anomaly rates to continuously optimize the bandwidth capacity allocated for offloading anomalies. This dynamic parameter adjustment allows the system to adapt to varying workloads and performance requirements without fixed operational overhead.
2Reliability
If continuous snapshots are taken for anomaly detection, then data protection improves, but storage resource utilization increases
Solution Approach 1:
The system extracts only the essential anomaly data from continuous snapshots and offloads it to remote storage, rather than storing complete snapshot copies. By identifying and isolating only the anomalous portions of data through machine learning analysis, the system maintains comprehensive data protection while significantly reducing the storage resources required at the primary storage device.
Solution Approach 2:
The system segments storage resources by separating local snapshot storage from remote anomaly offloading. Continuous snapshots are taken locally for immediate anomaly detection, but the actual anomaly data is segmented and transferred to remote storage based on memory capacity and bandwidth availability. This segmentation allows the system to maintain protection coverage while distributing storage demands across multiple locations.
3Reliability
If anomalies are offloaded to remote storage, then data integrity is ensured, but bandwidth consumption increases
Solution Approach 1:
The system performs partial offloading by transmitting only the identified anomaly data to remote storage rather than continuously streaming all snapshot data. By applying partial action only when anomalies are detected, the system ensures data integrity for critical portions while minimizing overall bandwidth consumption during normal operation periods.
Solution Approach 2:
The system uses periodic anomaly detection and offloading based on detected change rates and predicted anomaly timing. Instead of continuous offloading, the system periodically assesses storage device performance states and service level priorities to determine when anomaly offloading should occur, thereby reducing bandwidth consumption while maintaining data integrity through timely anomaly transmission.
Data Source
AI summary
Embodiments of the present disclosure relate to autonomous data protection. An input/output (I/O) stream can be received for a storage device. One or more anomalies corresponding to the I/O stream can be identified. At least one of the one or more anomalies can be offloaded anomalies to a remote storage based on a capacity of memory allocated to store at least one snapshot of the storage device that include at least one of the one or more anomalies.