Heterogeneous Data Distance Metrics for Accurate Anomaly Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distance metrics, such as Euclidean distance, are inadequate for evaluating data points with heterogeneous variables having different value ranges, leading to undesirable results in anomaly detection and system evaluation, as they are dominated by variables with the largest or smallest value ranges.
Innovation Solution
A method is developed to determine a distance metric by training a data-based system model, calculating quality levels and distance values for each class of variables, and applying a weighted L2-Norm to normalize and balance the influence of variables, allowing for effective evaluation of data points with diverse formats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional distance metrics (e.g., Euclidean distance) are used for heterogeneous data points, then the calculation is simple, but the evaluation accuracy deteriorates because the distance dimension is dominated by variables with the largest value ranges
Solution Approach 1:
The patent transforms the distance metric from a simple Euclidean distance calculation to a quality-based distance metric by changing the parameters used for evaluation. Instead of directly using raw value differences, the system uses quality levels derived from trained system models to weight and normalize distance calculations across heterogeneous variables, thereby improving evaluation accuracy without excessive complexity
Solution Approach 2:
The patent introduces a new dimension of evaluation by adding quality levels as an intermediate layer between raw data values and distance calculations. This quality dimension allows the system to evaluate heterogeneous variables on a common scale, preventing any single variable's value range from dominating the overall distance metric
2Reliability
If distance metrics are determined separately for each class of variables, then the influence of each variable is balanced, but the computational complexity increases
Solution Approach 1:
The patent applies preliminary action by training system models and determining quality levels in advance during an offline phase. These pre-computed quality metrics are then reused during online anomaly detection, allowing separate evaluation for each variable class without incurring the full computational cost during real-time operation
Solution Approach 2:
The patent segments the evaluation process into distinct components: training data preparation, system model training for each variable class, quality level determination, and final distance metric calculation. This segmentation allows each component to be optimized independently and facilitates parallel processing during the training phase
Data Source
AI summary
A method determines a distance metric for determining a distance to a data point having heterogeneous classes of variables. The method includes providing training records each assigning a label to a data point, the training records partitioned into training data points of a training amount and validation data points of a validation amount, and training a data-based system model with the training amount, such that the data-based system model associates data points with a model output, respectively. The method further includes for each validation data point of the validation amount, determining a quality level of the data-based system model and a distance value to a nearest training data point for each of the heterogeneous classes of variables. The distance value to the nearest training data point is determined separately with respect to a respective class of variables.