Machine-Learning Time-Stamped Event Lists With Human Review
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning algorithms used in generating textual lists of time-stamped events for public safety incidents are prone to bias due to feedback loops that incorporate biased labels and scores, leading to incorrect identifications and potential wrongful convictions.
Innovation Solution
A computing device that compares machine learning confidence scores to a threshold, redacts low-confidence events, and allows human input to validate or correct these events, integrating the input into a machine learning feedback loop to reduce bias.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If machine learning algorithms are used to generate textual lists of time-stamped events automatically, then productivity is improved, but reliability deteriorates due to machine learning bias
Solution Approach 1:
The patent introduces an intermediary review process where human operators validate machine-generated event lists. The system presents machine learning outputs to human reviewers who can confirm or correct event identifications, serving as a mediator between automated processing and final reliable output. This resolves the contradiction by maintaining high productivity through automation while ensuring reliability through human verification of low-confidence predictions.
Solution Approach 2:
The patent implements a feedback loop where human corrections and validations of machine-generated events are fed back into the machine learning training process. This continuous feedback mechanism allows the system to learn from its errors and improve over time, resolving the reliability issue while maintaining automated productivity. The feedback loop transforms initial low-reliability automated outputs into progressively more reliable results.
2Measurement precision
If machine learning confidence scores are used to filter events, then measurement precision is improved, but loss of information increases due to redaction of low-confidence events
Solution Approach 1:
Human operators serve as intermediaries for redacted low-confidence events, reviewing and potentially restoring events that were filtered out by confidence score thresholds. This prevents permanent information loss while maintaining the precision benefits of threshold filtering, as human reviewers can recover valid events that the machine learning model incorrectly flagged as low-confidence.
Solution Approach 2:
The system performs preliminary filtering using confidence scores before final event list compilation, but maintains the option to review and restore filtered events. This preliminary action approach allows efficient filtering while preserving the ability to recover potentially valid events, preventing information loss while maintaining measurement precision benefits.
3Reliability
If human validation is implemented for low-confidence events, then reliability is improved, but device complexity increases
Solution Approach 1:
The patent applies human validation selectively only to low-confidence events rather than all events. This local quality approach means that the complex validation process is applied only where needed (low-confidence predictions), while high-confidence events pass through automatically. This resolves the contradiction by improving reliability for critical uncertain cases while minimizing the complexity overhead of the validation system.
Solution Approach 2:
The system changes the parameter of validation intensity based on confidence score thresholds. High-confidence events undergo minimal processing, while low-confidence events trigger enhanced validation protocols. This parameter-based approach allows the system to maintain high reliability for uncertain events while keeping overall system complexity manageable through selective application of validation procedures.
Data Source
AI summary
A device, system, and method for reducing machine learning bias in machine learning generated textual lists of time-stamped events is provided. The device generates, via a machine learning algorithm, using sensor data related to the incident, a textual list of time-stamped events for an incident, one or more of the events associated with respective machine learning confidence scores. For a given time-stamped event having a respective machine learning confidence score less than a threshold confidence score, the device redacts the given event in the list and replaces it in the list with a field for receiving input, the list of rendered at a display screen with the given event in the list replaced with the field. The device receives, via an input device, input at the field and after receiving the input, un-redacts the given event, and renders, at the display screen, the received input received and the given event.