Real-Time Database Anomaly Detection via Heterogeneous Stream Correlation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems are inadequate in detecting insider threats and data exfiltration due to their inability to monitor all users and systems in real-time, often revealing issues only after the fact, and are costly and resource-intensive, failing to account for user, database, application, and network activity simultaneously.
Innovation Solution
A method and system for real-time anomaly detection that correlates heterogeneous data streams from agents, audit programs, and sensors to identify patterns of normalcy and detect deviations, using complex event processing and machine learning to alert on unauthorized activities, such as data exfiltration, by integrating data streams and applying models of normalcy and rules to detect anomalies across multiple data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If real-time monitoring of all users and systems is implemented, then detection capability is improved, but resource consumption and cost increase significantly
Solution Approach 1:
The system segments monitoring resources by deploying lightweight agents on specific databases, applications, and workstations rather than monitoring everything centrally. Each agent independently collects and processes local data, dividing the monitoring task into manageable segments that consume fewer resources overall while maintaining comprehensive detection capability.
Solution Approach 2:
The patent introduces an intermediary analysis layer that receives data from multiple agents and applies correlation rules to detect anomalies. This intermediary layer filters and processes data before presenting it for analysis, reducing the computational burden on the central system while maintaining high detection capability through intelligent data correlation.
2Loss of energy
If monitoring data is stored and analyzed off-line, then resource consumption is reduced, but detection speed deteriorates causing late discovery of issues
Solution Approach 1:
The system performs preliminary actions by continuously collecting and pre-processing monitoring data in real-time through distributed agents before analysis is needed. Data is normalized, correlated, and prepared in advance with anomaly detection rules already applied, so when analysis occurs, it can quickly identify issues without the lag of offline batch processing.
Solution Approach 2:
The monitoring system maintains continuous operation with agents constantly collecting data and the analysis engine continuously evaluating anomalies. This continuous useful action ensures real-time detection capability while managing resources efficiently through ongoing incremental processing rather than intermittent bulk analysis.
3Measurement precision
If dedicated analysts conduct real-time monitoring, then detection accuracy is improved, but operational cost increases
Solution Approach 1:
The system implements self-service capabilities through automated anomaly detection algorithms that independently analyze monitoring data and identify security threats without human intervention. The correlation rules and machine learning models enable the system to autonomously detect patterns, flag anomalies, and alert security personnel, maintaining high detection accuracy while eliminating the need for dedicated human analysts to perform routine monitoring.
Solution Approach 2:
The patent replaces the mechanical system of human analysts manually reviewing monitoring data with automated computational systems. Software-based anomaly detection algorithms, correlation engines, and machine learning models substitute for human cognitive processing, providing consistent, scalable detection accuracy without the ongoing operational costs of human labor.
4Reliability
If multiple heterogeneous data streams are correlated, then anomaly detection capability is improved, but system complexity increases
Solution Approach 1:
The system achieves universality by creating a standardized data correlation framework that handles multiple heterogeneous data streams from different sources (databases, applications, workstations, network devices). The correlation engine uses universal rules and patterns that can be applied across diverse data types, enabling the system to detect anomalies across the entire IT infrastructure without requiring separate complex systems for each data source.
Data Source
AI summary
A system and method for real-time detection of anomalies in database or application usage is disclosed. Embodiments provide a mechanism to detect anomalies in database or application usage, such as data exfiltration attempts, first by identifying correlations (e.g., patterns of normalcy) in events across different heterogeneous data streams (such as those associated with ordinary, authorized and benign database usage, workstation usage, user behavior or application usage) and second by identifying deviations/anomalies from these patterns of normalcy across data streams in real-time as data is being accessed. An alert is issued upon detection of an anomaly, wherein a type of alert is determined based on a characteristic of the detected anomaly.