Block Rate Matching for Fraud Detection Model Updates

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Deploying updated machine learning models for fraudulent transaction detection disrupts merchant expectations regarding the number or percentage of transactions blocked, leading to potential losses or increased security concerns due to changes in fraud detection efficacy.

Innovation Solution

The implementation of a block rate matching system that adjusts merchant thresholds and generates adjusted fraud detection thresholds to maintain consistent block rates when deploying new or updated machine learning models, ensuring that the expected block rate remains unchanged while improving fraud detection accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If updated machine learning models are deployed to improve fraud detection accuracy, then fraud detection efficacy is improved, but the block rate changes disrupting merchant expectations

Engineering Contradiction:

Improvefraud detection efficacyVSAvoidmerchant expectations continuity

Core Design Contradiction:

ReliabilityVSEase of operation

Solution Approach 1:

The system adjusts the fraud detection threshold parameter when deploying updated machine learning models. By dynamically changing the threshold parameter, the system maintains a consistent block rate despite improvements in model accuracy, thereby resolving the contradiction between improved fraud detection efficacy and maintaining merchant expectations continuity

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The system implements a feedback mechanism that monitors block rates and compares them against expected ranges. When updated models cause block rate deviations, the feedback loop triggers threshold adjustments to bring the block rate back within expected ranges, thus maintaining merchant expectations while allowing model improvements

Inventive Principle:

Principle #23Feedback

2Ease of operation

If the fraud detection threshold is adjusted to maintain block rate consistency, then merchant expectations are maintained, but fraud detection accuracy may be compromised

Engineering Contradiction:

Improvemerchant expectations continuityVSAvoidfraud detection accuracy

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The system dynamically adjusts thresholds based on the specific characteristics of updated models and current fraud patterns. Rather than using static thresholds, the system adapts thresholds in real-time to maintain both merchant expectations continuity and optimal fraud detection accuracy, resolving the contradiction between these two requirements

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system performs preliminary testing and validation of updated machine learning models before full deployment. By pre-adjusting thresholds and validating performance metrics on test datasets, the system ensures that both merchant expectations and fraud detection accuracy are maintained before rolling out updates to production

Inventive Principle:

Principle #10Preliminary action

3Adaptability or versatility

If machine learning models are re-trained with new data to improve detection capabilities, then fraud detection capabilities are enhanced, but the distribution and scoring of transactions change

Engineering Contradiction:

Improvefraud detection capabilitiesVSAvoidtransaction scoring distribution

Core Design Contradiction:

Adaptability or versatilityVSStability of the object's composition

Solution Approach 1:

The system introduces threshold adjustment as an intermediary mechanism between model re-training and production deployment. This intermediary step transforms the output distribution of re-trained models to match expected scoring distributions, allowing enhanced fraud detection capabilities while maintaining stability in transaction scoring

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system performs preliminary distribution matching and validation before deploying re-trained models. By pre-adjusting the scoring distribution of updated models to match historical patterns, the system ensures that both enhanced detection capabilities and stable scoring distributions are achieved

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20240095741A1Systems and methods for block rate matching when updating a machine learning model based fraud detection system

Publication Date: 2024.03.21 STRIPE LLC

AI summary

A method and apparatus for machine learning model based fraud detection of transactions are described. The method may include determining, by a fraud detection system, a first block rate of a first fraud detection machine learning model (MLM) that scores transactions of a server computer system, wherein the fraud detection system blocks a transaction when a score generated for the transaction by the first fraud detection MLM does not satisfy a first threshold value. The method may also include accessing, by the fraud detection system, a second fraud detection MLM to be used by the fraud detection system for blocking future transactions. Furthermore, the method may include determining, by the fraud detection system, a second threshold value using a set of prior transactions input into the second fraud detection MLM that results in a second block rate of the second fraud detection MLM within a predetermined margin of the first threshold. The method may also include deploying, by the server computer system, the second fraud detection MLM using the second threshold value for performing fraud detection of at least one new transaction processed by the server computer system.