Silent Rule Evaluation for Security Detection Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing rule-based detection systems in security management generate a high number of false positives and have limited coverage, affecting user experience with increasing alerts, and there is a need to evaluate the quality of these detections in a silent manner without disrupting user experience.
Innovation Solution
A method for evaluating the quality of rule-based detections using silent rules that operate without generating alerts, collecting telemetry events from sensors, aggregating them, and analyzing them to calculate quality metrics such as precision, recall, and performance, with verified rules being released as alerting types once quality targets are met.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If rule-based detections are used to detect malicious attacks, then detection coverage is improved, but false positive rate increases and user experience deteriorates
Solution Approach 1:
The patent applies preliminary action by implementing a silent evaluation phase before deploying rules to production. During this phase, rules are tested in a silent mode (without generating alerts) to collect telemetry data and evaluate their quality metrics. This preliminary testing allows the system to identify and eliminate rules with high false positive rates before they are released to users, thereby maintaining detection coverage while reducing false positives.
2Reliability
If increasing number of rules are added to improve detection coverage, then more attacks are detected, but user experience is heavily affected by false positive alerts
Solution Approach 1:
The patent implements preliminary action by establishing a quality gate process before rules are deployed. Rules undergo silent evaluation with quality assessment using precision, recall, and performance metrics. Only rules that meet predetermined quality thresholds are released to production. This preliminary filtering ensures that rules added to improve detection coverage have been vetted to minimize negative impact on user experience.
Solution Approach 2:
The patent applies feedback by continuously monitoring rule performance through quality metrics (precision, recall, performance) during silent evaluation. This feedback mechanism allows the system to identify rules that perform poorly and prevent their deployment, while highlighting rules that meet quality standards for release. The feedback loop ensures continuous improvement of rule quality and maintains ease of operation.
3Measurement precision
If rules are evaluated in production environment, then quality assessment is accurate, but user experience is affected by alerts during evaluation
Solution Approach 1:
The patent applies segmentation by separating rule evaluation from rule execution. Rules are evaluated in a silent mode where they are tested against telemetry events but do not generate alerts or take action. This segmentation allows accurate quality assessment in the production environment without the harmful side effect of alerting users. The silent evaluation phase is distinct from the production enforcement phase, enabling both accurate measurement and good user experience.
Data Source
AI summary
The present invention relates to a method for evaluating quality of signature-based detections in an infrastructure provided with a plurality of sensors, comprising defining predefined rules for the rule-based detections, wherein the rules are of a silent type such that operate without generating alerts to the user of the infrastructure, collecting telemetry events at each of the sensors, storing the telemetry events of each of the sensors to respective local sensor databases operatively connected to the sensors, aggregate, at predetermined aggregating time intervals, the telemetry events from the local sensor databases to a central database, analyzing the telemetry events at the central database, by evaluating the telemetry events with respect to the rules and calculating the quality measurements of the rules, according to a plurality of predefined quality metrics in a predefined metrics time interval, wherein the quality metrics comprise precision metric, by counting the instances of false positives of the telemetry events with respect to the predefined rules, recall metric, by counting the instances of false negatives of the telemetry events with respect to the predefined rules and performance metric, by counting the instances of rules hits over predefined evaluation time interval and the ratio between the partial and full of the rules matching, wherein the method for evaluating quality of rule-based detections further comprises releasing verified rules for the rule-based detections as predefined rules having the quality measurements within a predetermined quality target range, and wherein the verified rules are of alerting type such that operate generating alerts to the user of the infrastructure.