Hyper-rectangular Data Space Partitioning for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data exploration and visualization methods fail to effectively identify insufficient data or anomalous combinations of feature values in datasets, particularly in high-dimensional spaces with mixed feature types, leading to challenges in interpreting and gaining insights from data.
Innovation Solution
A system and method that partition the data space into human-interpretable regions by calculating observation density using distance or anomaly-based metrics, creating a map of hyper-rectangular shapes that represent varying levels of density, including empty spaces, which can be displayed on a GUI for user selection and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data space is partitioned into fine-grained regions to improve detection precision of anomalous areas, then measurement precision improves, but device complexity increases due to the large number of partitions required in high-dimensional space
Solution Approach 1:
The high-dimensional data space is segmented into multiple hyper-rectangular partitions based on feature ranges. Each partition represents a specific combination of feature value ranges, allowing precise identification of anomalous regions while managing complexity through systematic division of the feature space.
Solution Approach 2:
The patent transforms the visualization and analysis of high-dimensional data into a lower-dimensional representation by projecting density information onto a 2D map. This dimensionality reduction allows complex multi-dimensional partitions to be visualized and interpreted without requiring direct visualization of all dimensions simultaneously.
2Measurement precision
If observation density is calculated using complex distance or anomaly-based metrics to improve detection accuracy, then measurement precision improves, but use of energy increases due to computationally intensive calculations
Solution Approach 1:
Density values for all partitions are pre-calculated and stored before the actual anomaly detection process. This preliminary computation allows the system to quickly query and compare density values during runtime without performing repeated complex distance calculations, significantly reducing real-time computational energy consumption.
Solution Approach 2:
The patent creates a simplified copy of the density information in the form of a 2D map representation. Instead of performing complex multi-dimensional distance calculations during query operations, the system uses pre-computed density values stored in the map structure, which can be efficiently accessed and compared.
3Ease of operation
If the system visualizes all partition details to improve ease of operation, then ease of operation improves, but loss of information increases due to the complexity overwhelming human interpretation
Solution Approach 1:
The patent extracts only the essential information needed for human interpretation from the complex partition data. The 2D map visualization displays key density information and anomalous regions while omitting unnecessary computational details, allowing users to understand data distribution without being overwhelmed by the full complexity of the partition structure.
Solution Approach 2:
Different regions of the visualization are presented with different levels of detail appropriate to their significance. Areas with anomalous density values receive special highlighting and detailed information, while normal regions are displayed with less detail, allowing users to focus on the most important findings without information overload.
Data Source
AI summary
Systems and methods for automatically identifying in a dataset insufficient data for learning, or records with anomalous combinations of feature values, by partition of numeric and/or categorical data space into human-interpretable regions are disclosed. The method comprises: receiving a dataset of numeric and/or categorical features with a plurality of observations.Calculating observation density for each observation according to a distance or anomaly based metric, and receiving a density measurement. Partitioning the dataset along the numeric and/or categorical features according to the density measurement of each observation by a perpendicular cut along the feature spaces, receiving a map of a plurality of hyper-rectangular shapes representing various levels of density including empty spaces. Displaying the received map, being human-interpretable regions on a Graphic user interface, GUI, wherein the plurality of hyper-rectangular shapes are selectable and present information about the selected hyper-rectangular shape level of density when selected by a user.