Report Prioritization via KL Divergence and L1 Norm
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods lack a standard, quantitative analysis technique to filter and prioritize reports effectively, especially in handling large volumes of dynamically changing data, and fail to provide repeatable results or objective evidence of report importance.
Innovation Solution
The use of Kullback-Leibler divergence and L1 norm ratio to determine shape and volume changes between current and baseline report distributions, allowing for the prioritization and filtering of reports based on dynamic thresholds, which are functions of the total bin number and average sample count.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a predefined report set with qualitative correlation rules is used, then reports can be correlated into neighborhoods, but the technique cannot provide objective evidence of report importance and cannot prioritize reports
Solution Approach 1:
The patent transforms the qualitative correlation approach into a quantitative analysis by changing the parameter from subjective correlation rules to objective statistical measurements. Specifically, it uses KL-divergence to measure the statistical difference between current report data and baseline data, providing a numerical parameter (divergence value) that objectively quantifies report importance. This parameter change enables automatic prioritization based on measurable statistical evidence rather than qualitative judgments.
2Productivity
If traditional report analysis methods are used, then all reports can be generated, but analysts cannot handle the large volume of reports due to lack of prioritization
Solution Approach 1:
The patent extracts the essential characteristic that distinguishes important reports from unimportant ones by calculating the KL-divergence between current and baseline data. This extraction process isolates the statistical significance of each report, allowing the system to separate important reports (high divergence) from unimportant ones (low divergence). By extracting this key metric, the system can prioritize reports without losing information about their relative importance.
3Reliability
If quantitative analysis techniques are applied, then repeatable results can be achieved, but the techniques must handle dynamic changes in report data effectively
Solution Approach 1:
The patent implements dynamics by establishing a baseline representation of normal report data characteristics and continuously comparing current report data against this baseline. The KL-divergence calculation dynamically adapts to changing conditions by measuring the statistical difference between the current state and the baseline state. This dynamic approach allows the system to reliably identify important reports even as the underlying data distribution changes over time, maintaining both repeatability and adaptability.
Data Source
AI summary
The raw data for a plurality of numerical reports (distributions or histograms) concerning malware infection in a computer network are stored in a data source. The data source is queried to produce any number of reports. Each report's content comes from a distribution of data within a time interval, and a baseline distribution is formed for comparison by the corresponding historical data. The shape change for the distributions is determined by using Kullback-Leibler divergence. The change of volume (i.e., total sample count) for the distributions is determined using the L1 norm ratio. A cutoff threshold is determined for the K-L divergence and the volume ratio threshold is determined for the count change. A measure value for each report is determined by multiplying the shape change by the volume change (modified by raising it to a particular power). The reports are ranked based upon their measure values. A report is determined to be important if its shape change is greater than the cutoff threshold, if it's volume change is greater than the count ratio threshold, or if the measure value is greater than a measure threshold. The invention can be applied to all kinds of reports suitable for a distribution or histogram, and also provides one approach to detect anomalous behaviors.