Categorical Feature Anomaly Detection via Probability Distribution Distance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions face challenges in detecting deviations from baseline behavior patterns for categorical features in network security, as standard statistical methods are ineffective for numerical representation and analysis of categorical data.
Innovation Solution
A method involving the determination of discrete probability distributions for categorical variables based on network activity data, using a distance function to compare these distributions and detect anomalies by determining if the output scalar value exceeds a threshold, thereby identifying normal or anomalous behavior.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If standard statistical methods are applied to categorical features, then numerical analysis can be performed, but the analysis does not provide meaningful information about the data
Solution Approach 1:
The patent transforms categorical data by changing its parameter representation from raw category labels to probability distributions derived from baseline behavior patterns. This transformation enables the application of statistical distance functions while preserving the meaningful semantic relationships inherent in categorical data, thus resolving the contradiction between method applicability and result meaningfulness
2Productivity
If categorical data is represented using numerical values, then statistical calculations can be performed, but the calculations do not effectively represent the categorical nature of the data
Solution Approach 1:
The patent introduces probability distributions as an intermediary representation between raw categorical data and statistical analysis. This intermediary layer enables computational operations while maintaining fidelity to the categorical nature of the data, as the probability distributions are derived from actual baseline behavior patterns rather than arbitrary numerical assignments
Solution Approach 2:
The patent changes the parameter representation from direct numerical encoding of categories to probability distributions that capture the likelihood of different category occurrences based on baseline behavior. This parameter transformation enables meaningful statistical comparisons while preserving the categorical semantics
3Ease of operation
If existing anomaly detection methods are used for categorical features, then detection can be performed, but the methods cannot effectively detect deviations from baseline behavior patterns
Solution Approach 1:
The patent transforms the detection parameter from direct categorical comparison to distance-based comparison of probability distributions. By representing both baseline behavior and observed behavior as probability distributions and computing the distance between them, the system achieves both operational simplicity and precise detection of deviations from baseline patterns
Data Source
AI summary
A system and method for detecting deviations from baseline behavior patterns for categorical features. A method includes determining a first discrete probability distribution for a categorical variable based on a first set of network activity data; determining a second discrete probability distribution for a unique observation based on a second set of network activity data; comparing the second discrete probability distribution to the first discrete probability distribution by applying a distance function to the first and second discrete probability distributions, wherein an output of the distance function is a scalar value representing a difference between the first and second discrete probability distributions; determining whether the scalar value is above a threshold; detecting an anomaly with respect to the categorical variable when the scalar value is above the threshold; and determining that a behavior with respect to the categorical variable is normal when the scalar value is not above the threshold.