Object Store Access-Log Aggregation for Data Lake Security
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The large scale and flexibility of data lakes make it difficult to monitor and detect security risks such as misconfigurations and bad practices, leading to potential security incidents and breaches, as existing monitoring tools like query engines are decoupled from the data and struggle with the vast amount of access log records.
Innovation Solution
A processing device aggregates access log data from a data lake based on various characteristics, generating aggregated log data to identify security incidents by monitoring anomalies and vulnerabilities over time, using methods like time-based clustering to analyze metrics such as database operations, bytes transferred, and accessed partitions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If monitoring tools like query engines are used to track security activity in data lakes, then security monitoring capability is provided, but the tools become decoupled from the data and struggle with the vast amount of access log records
Solution Approach 1:
The patent introduces a security monitoring system that directly monitors object store access logs instead of relying on query engines. This intermediary monitoring approach bridges the gap between data storage and security analysis, providing reliable security monitoring while avoiding the decoupling problem of query engine-based tools.
Solution Approach 2:
The monitoring system segments access log analysis by focusing on specific security-relevant patterns and metrics rather than attempting to process all access logs comprehensively. This segmentation enables effective security monitoring by dividing the vast log data into manageable security-focused analysis units.
2Reliability
If existing monitoring tools process vast access log records to detect security risks, then security incident detection is attempted, but the scale and volume of logs make monitoring difficult and costly
Solution Approach 1:
The patent extracts and focuses on security-relevant information from vast access logs by implementing targeted monitoring for specific security risks such as unauthorized access patterns, data exfiltration attempts, and misconfiguration detection. This extraction approach processes only the critical security-related portions of logs rather than analyzing every access record.
Solution Approach 2:
The monitoring system implements partial action by focusing on key security metrics and patterns rather than attempting complete analysis of all access log data. This selective monitoring approach provides sufficient security incident detection capability while avoiding the computational burden of processing the entire log volume.
3Reliability
If comprehensive security monitoring is implemented across the entire data lake, then security coverage is improved, but the cost and computational resources required increase significantly
Solution Approach 1:
The patent implements local quality by applying different monitoring intensities to different data lake regions based on security sensitivity. High-security areas receive comprehensive monitoring while less sensitive areas use lighter monitoring approaches, optimizing resource utilization while maintaining adequate security coverage across the entire data lake.
Data Source
AI summary
Embodiments of the present disclosure provide a method for detecting security incidents in an object store by aggregating log files generated by a monitoring program of the object store and monitoring the aggregated log data. A processing device may periodically execute database operations to access data stored in the object store. In response to each database operation, an access log set may be generated and stored in an access log storage. The processing device may periodically aggregate access log data from a plurality of access log sets currently stored in the access log storage to generate aggregated log data. The processing device may then monitor the aggregated log data over time to identify one or more security incidents of the object store.