ML Classifiers Auto-Escalate Adverse Events
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems that use fuzzy logic to evaluate potential dangers often face challenges with false negatives and false positives, leading to costly mistakes or overwhelming human reviewers with unnecessary flags, as they struggle to accurately discriminate between clearly adverse and innocuous events.
Innovation Solution
A machine learning system that employs a multi-model framework with two classifiers to auto-approve or auto-escalate events by vectorizing and enriching data, using a first model for adverse escalation and a second model for innocuous approval, with human review only when models are uncertain, and updates training data based on human feedback.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If an automated system uses conservative heuristics to identify adverse events, then false positives increase and human reviewer workload increases, but false negatives decrease
Solution Approach 1:
The patent divides the evaluation process into two separate machine learning models: a first model that evaluates whether an event is adverse (escalation candidate) and a second model that evaluates whether an event is innocuous (approval candidate). This segmentation allows each model to specialize in one aspect of the decision-making process, improving their individual accuracy while reducing the need for conservative filtering that would otherwise require extensive human review.
Solution Approach 2:
The system implements feedback loops where human reviewers evaluate events that were flagged by the first model but not approved by the second model. The results of human review are used to retrain both machine learning models, continuously improving their accuracy over time. This feedback mechanism allows the system to become more precise in its classifications, reducing false positives while maintaining low false negative rates.
2Productivity
If an automated system uses liberal heuristics to approve events, then false positives decrease and human reviewer workload decreases, but false negatives increase
Solution Approach 1:
By separating the evaluation into two distinct models with different objectives, the system can optimize each model's decision criteria. The first model can be tuned to be more sensitive (liberal) in identifying potential adverse events, while the second model can be tuned to be more selective in approving innocuous events. This segmentation resolves the trade-off between sensitivity and specificity that plagues single-model systems.
Solution Approach 2:
The second machine learning model acts as an intermediary between the first model's adverse event detection and the final decision. Events flagged by the first model must also be evaluated by the second model, which determines whether they are truly innocuous. This intermediary layer provides an additional safety check that reduces false negatives while still allowing liberal approval of clearly benign events.
3Device complexity
If a single machine learning model is used to evaluate events, then system complexity is reduced, but discrimination accuracy between clearly adverse and clearly innocuous events decreases
Solution Approach 1:
The patent explicitly divides the classification task into two separate models, each trained to perform a specific function. The first model is trained to identify adverse events with high sensitivity, while the second model is trained to identify innocuous events with high precision. This segmentation of the classification task allows each model to specialize in one aspect, achieving higher overall accuracy than a single general-purpose model could provide.
Solution Approach 2:
Each machine learning model is optimized for its specific local task: the first model focuses on detecting adverse events with appropriate sensitivity thresholds, while the second model focuses on approving innocuous events with appropriate specificity thresholds. This local optimization of each model's quality metrics enables the system to achieve high overall classification accuracy that would be difficult for a single model to attain.
Data Source
AI summary
A system and method for using machine learning classifiers to auto-approve or auto-escalate unknown events is disclosed. The system has queues for receiving a stream of data concerning the unknown events; for storing data concerning escalated events believed to be adverse; and for storing data concerning approved events believed to be innocuous. When software instructions are executed, a device will retrieve data concerning an event from the first queue; vectorize and enrich the data; classify the vectorized, enriched data by a first machine learning model to decide whether to auto-escalate the data concerning the event to the second queue for review by a first human reviewer; and classify the vectorized, enriched data by a second machine learning model to decide whether to auto-approve the data concerning the event to the third queue.