Near Real-Time Risk Score Generation via Batch and Stream Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional 'Big Data' anomaly detection models perform batch training and inference, limiting their ability to provide near real-time risk score computations essential for entity authentication and alert notifications, especially in scenarios like cyber-attacks and abnormal behavior detection.
Innovation Solution
A system and method that generate risk scores in near real-time by combining intermediate risk scores from real-time application events with batch risk scores, using models trained from extracted features, and applying time-dependent decay to volumetric and point-in-time anomalies, with actions initiated if the risk score exceeds a predetermined threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If batch training and inference are used for anomaly detection, then model accuracy and reliability are improved, but near real-time risk score computation capability deteriorates
Solution Approach 1:
The patent segments the risk score computation into two independent parts: batch risk scores (computed periodically with high accuracy) and near real-time risk scores (computed continuously for immediate anomalies). This segmentation allows each component to optimize for its specific requirements without compromising the other, resolving the contradiction between accuracy and real-time performance.
Solution Approach 2:
The patent merges batch risk scores and near real-time risk scores to produce a comprehensive final risk score. This combination allows the system to leverage both the high accuracy of batch processing and the timeliness of real-time processing, achieving both reliability and near real-time capability simultaneously.
2Use of energy by moving object
If only batch processing is used for anomaly detection, then computational resources are optimized, but responsiveness to immediate security threats deteriorates
Solution Approach 1:
The system segments processing into batch mode (for comprehensive analysis with optimized resources) and near real-time mode (for immediate threat response). This allows computational resources to be allocated efficiently based on priority, with batch processing handling bulk analysis and real-time processing handling critical immediate threats.
Solution Approach 2:
The batch process performs preliminary anomaly detection and risk scoring in advance, preparing risk profiles and anomaly patterns before actual threats occur. This preliminary action enables the near real-time system to respond faster by relying on pre-computed models and baseline data, reducing the computational burden during immediate threat response.
3Speed
If real-time anomaly detection is implemented, then security responsiveness is improved, but system complexity increases
Solution Approach 1:
The patent divides the anomaly detection system into separate batch processing components and near real-time processing components. This segmentation reduces overall system complexity by allowing each component to be designed and optimized independently, with clear interfaces between them for score combination and anomaly handling.
Solution Approach 2:
The patent introduces an intermediary layer that combines batch risk scores and near real-time risk scores to produce final risk assessments. This intermediary mechanism simplifies the system architecture by providing a standardized integration point, reducing the complexity of coordinating between batch and real-time processing systems.
Data Source
AI summary
Embodiments of the present disclosure provide a system for generating risk scores in near real-time. The system includes a processor and a memory coupled with and readable by the processor and storing therein a set of instructions. When executed by the processor, the processor is caused to generate risk scores in near real-time by receiving near real-time application events associated with an application in near real-time and identifying anomalies from the near real-time application events. The processor is further caused to generate risk scores in near real-time by generating an intermediate near real-time risk score for the identified anomalies and combining the intermediate near real-time risk score with a batch risk score generated from a batch process executed prior to receiving the near real-time application events to generate a near real-time risk score.