Weighted Histogram-Based Outlier Scoring for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection methods, such as Histogram-Based Outlier Scoring (HBOS), face challenges in handling dependent features and providing clear explanations for anomaly scores, especially in complex scenarios, and lack intelligent machine learning systems for effective loss prevention in retail domains, relying heavily on labor-intensive manual checks.
Innovation Solution
The Weighted Histogram-Based Outlier Scoring (W-HBOS) model addresses these issues through automated feature selection and orthogonal feature transformation using Principal Component Analysis (PCA), generating reason codes to explain anomalies and identifying top-scoring entities, which can be integrated into a loss prevention system for detecting and explaining anomalies in merchant data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional Histogram-Based Outlier Scoring (HBOS) is used for anomaly detection, then the detection process is simple and fast, but it cannot effectively handle dependent features and provides unclear explanations for anomaly scores
Solution Approach 1:
The patent segments the anomaly detection process into distinct components: feature selection module, orthogonal transformation module (PCA), and weighted histogram scoring module. This segmentation allows each component to specialize - feature selection identifies relevant attributes, PCA handles dependencies through orthogonal transformation, and weighted scoring provides both detection and explanation, thereby resolving the contradiction between speed and accuracy.
Solution Approach 2:
The patent introduces Principal Component Analysis (PCA) as an intermediary between the raw features and the histogram-based outlier scoring. PCA transforms the original dependent features into orthogonal principal components, serving as a mediator that eliminates feature dependencies while preserving information. This intermediary layer enables the simple HBOS method to work effectively on transformed features, maintaining speed while improving accuracy.
2Loss of information
If manual anomaly detection is performed, then clear explanations can be provided for each anomaly, but the process becomes labor-intensive and inefficient for large datasets
Solution Approach 1:
The system implements self-service by automatically generating explanations for anomalies through the feature selection and PCA transformation modules. Instead of requiring manual analysis, the system autonomously identifies which features contribute most to each anomaly score and provides interpretive explanations automatically. This eliminates the need for labor-intensive manual review while maintaining high-quality explanations, resolving the contradiction between explanation quality and detection time.
3Measurement precision
If feature transformation using PCA is applied, then feature dependencies are reduced and detection accuracy improves, but the system complexity increases
Solution Approach 1:
The patent segments the complex PCA transformation process into manageable modular components that can be independently configured and executed. The feature selection module, orthogonal transformation module, and weighted histogram module are distinct and can be applied selectively based on data characteristics. This modular segmentation reduces the perceived and actual system complexity while maintaining the accuracy benefits of PCA transformation.
4Loss of information
If weighted histogram-based outlier scoring is used instead of traditional HBOS, then anomaly explanations become more informative, but the computational overhead increases
Solution Approach 1:
The patent applies partial action by implementing weighted histogram scoring only for the selected principal components rather than all possible features. The feature selection module filters out irrelevant features before transformation, and the weighted scoring is applied selectively to the most informative principal components. This partial application reduces computational overhead while still providing the explanation quality benefits of weighted scoring, resolving the contradiction between explanation quality and computational resources.
Data Source
AI summary
Different automatic tasks are facilitated via outlier detection in datasets using a Weighted Histogram-based Outlier Scoring (W-HBOS). An initial set of features is extracted from a processed dataset. The initial set of features is further filtered by applying robust statistics for size reduction. A second round of automatic feature selection is implemented based on maximum-entropy estimation so that a selected set of features that can give maximum possible information from different dimensions towards detecting anomalies are selected. The selected set of features are transformed to generate principal components that are provided to the W-HBOS-based model for outlier detection. A subset of outliers in one of the directions can be selected and reason codes are identified using back transformation for the execution of a desired automatic task.