Variational Autoencoder for Cloud Storage Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud provider networks face security breaches due to misconfigured data access policies and malicious insider attacks, compromising the security of sensitive data stored in object storage services.
Innovation Solution
A variational autoencoder model is employed to identify anomalous events in object storage services by modeling legitimate access patterns and using a pre-trained embedding layer to detect unauthorized or abusive behavior, calculating normalcy scores for incoming requests to determine their anomalous nature.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional access control mechanisms are used to protect data, then basic security is provided, but security breaches still occur due to stolen credentials or policy misconfigurations
Solution Approach 1:
The system performs preliminary action by training the autoencoder model on legitimate access patterns before deployment. The model learns and stores the normal behavior characteristics in advance, enabling it to proactively detect anomalies rather than reactively respond to breaches. This preliminary training phase establishes the baseline for what constitutes normal access behavior.
Solution Approach 2:
The system implements feedback by continuously monitoring access patterns and using the autoencoder to generate reconstruction errors. When the reconstruction error exceeds a threshold, the system triggers alerts or automated responses. This closed-loop feedback mechanism allows the system to adapt and respond to new threat patterns over time.
2Measurement precision
If comprehensive monitoring of all access events is implemented, then anomaly detection capability is improved, but system complexity and computational overhead increase
Solution Approach 1:
The system replaces complex rule-based monitoring mechanisms with a machine learning-based autoencoder model. Instead of implementing numerous manual security rules and thresholds, the system uses an unsupervised learning model that automatically learns patterns from data. This substitution reduces operational complexity while maintaining or improving detection accuracy.
Solution Approach 2:
The autoencoder model performs self-service by automatically learning legitimate access patterns from training data without requiring manual configuration of detection rules. The model self-adjusts its parameters and thresholds based on the data it processes, reducing the need for ongoing manual tuning and configuration by security personnel.
3Speed
If real-time analysis of access patterns is performed, then detection speed is improved, but computational resources and processing time are consumed
Solution Approach 1:
The system performs preliminary action by pre-training the autoencoder model offline on large datasets of legitimate access patterns. This heavy computational work is done in advance rather than in real-time. Once trained, the model can quickly infer anomalies from new data with minimal computational overhead, enabling fast real-time detection.
Solution Approach 2:
The system applies partial action by focusing computational resources only on evaluating new access patterns against the learned model, rather than re-processing all historical data in real-time. The model evaluates only the necessary features and dimensions required for anomaly detection, reducing unnecessary computational expenditure while maintaining detection speed.
Data Source
AI summary
Techniques for enabling the identification of anomalous events associated with an object storage service of a cloud provider network using a variational autoencoder model including a pre-trained embedding for selected features of events are described. A variational autoencoder, for example, encodes data into a latent space and reconstructs approximations of the data from an encoding in the latent space. In this context, for example, anomalous events of interest might represent unauthorized or abusive behavior associated with storage resources provided by an object storage service (or in association with other types of computing resources provided by other services of a cloud provider network). Legitimate (or benign) access patterns to an object storage service can be modeled by utilizing observed data plane events stored by an account activity monitoring service. Once trained, the model can be used to identify anomalous events.