Partial AUC Scoring for Risk Engine Authentication Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing adaptive authentication systems face challenges in effectively evaluating the classification performance of risk engine models, particularly due to the narrow operating range and skewness of performance data, which makes it difficult to determine if a new model is significantly better than the current one and prevents direct comparison across different environments.
Innovation Solution
The use of partial area under the curve (pAUC) of the receiver operating characteristic (ROC) curve, optionally standardized with the McClish Transformation, provides a performance score for authentication methods, allowing for comparison and assessment of their classification performance, especially focusing on the region of false positives, and includes a confidence level based on natural test statistics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional performance evaluation methods are used for risk engine models, then the evaluation process is simple, but the measurement precision is insufficient due to narrow operating range and skewness of performance data
Solution Approach 1:
The patent extracts and focuses on the region of interest (false positive region) from the complete ROC curve by calculating partial AUC instead of total AUC. This extraction allows precise measurement of performance in the critical low false positive rate region where conventional methods fail due to data skewness and narrow operating range.
Solution Approach 2:
The patent transforms the performance evaluation from a single metric approach to a two-dimensional ROC curve analysis, then further to a standardized score dimension. By plotting true positive rate against false positive rate and calculating area under the curve, it creates a new measurement dimension that overcomes the limitations of narrow operating ranges in conventional single-metric evaluations.
2Adaptability or versatility
If risk engine models are evaluated using traditional metrics, then the evaluation is straightforward, but the ability to compare performance across different environments is lost due to data skewness
Solution Approach 1:
The patent applies McClish Transformation to change the parameter scale of AUC values, transforming them into standardized scores with known statistical properties. This parameter transformation enables valid comparisons across different environments and datasets by normalizing the performance metrics while preserving the ability to detect significant differences through confidence intervals.
Solution Approach 2:
The patent incorporates confidence interval calculation as feedback to determine whether observed performance differences are statistically significant. This feedback mechanism allows the evaluation system to adaptively assess whether model improvements are real or due to data variability, enabling reliable cross-environment comparisons.
3Measurement precision
If the complete AUC is used to evaluate authentication methods, then the overall performance is captured, but the specific performance in the false positive region is diluted and not sufficiently highlighted
Solution Approach 1:
The patent segments the ROC curve into different regions of interest, specifically focusing on the false positive region (low false positive rate area) by calculating partial AUC. This segmentation allows precise measurement of performance in the critical security region while the methodology can be extended to evaluate other regions, thus maintaining comprehensive performance understanding.
Data Source
AI summary
Methods and apparatus are provided for evaluating the classification performance of different risk engine models. A classification performance of an authentication method is evaluated by obtaining performance data for an authentication method; generating a receiver operating characteristic (ROC) curve for the obtained performance data; determining a partial area under the curve (pAUC) for a region of interest of the ROC curve; and providing a performance score for the authentication method based on the pAUC. The region of interest comprises, for example, a region of false positives. The pAUC is optionally standardized using a McClish Transformation. The performance score for the authentication method can be compared to a second performance score for a second authentication method. A confidence level can optionally be provided for the comparison based on a natural test statistic.