Learning Model Evaluation System for Fraud Detection Accuracy

Solution Overview

Problem

The accuracy of fraud detection models in services decreases over time due to changing user actions, leading to incorrect classification of fraudulent and valid actions, necessitating a method to accurately evaluate and update the model's accuracy.

Innovation Solution

A learning model evaluation system that acquires authenticated user information, uses this data to evaluate the accuracy of a fraud detection model by comparing outputs to actual actions, and updates the model based on the latest trends to maintain high accuracy.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a learning model is trained with historical data to detect fraud, then the model can identify fraudulent actions, but the accuracy decreases over time as user actions change

Engineering Contradiction:

Improvefraud detection accuracyVSAvoidadaptability to changing user behaviors

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The system performs preliminary evaluation of the learning model's accuracy using authenticated user data before actual fraud detection occurs. By evaluating the model in advance with known authenticated actions, the system can detect accuracy degradation trends and trigger retraining before false positives or false negatives increase during actual fraud detection operations.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system establishes a feedback loop where the evaluation unit continuously monitors the learning model's performance using authenticated user action data. This feedback mechanism provides real-time information about model accuracy degradation, enabling the training unit to update the model with new data patterns and maintain high detection accuracy despite changing user behaviors.

Inventive Principle:

Principle #23Feedback

2Stability of the object's composition

If the learning model is not updated to learn latest trends, then the model structure remains stable, but the accuracy of detecting current fraud actions decreases

Engineering Contradiction:

Improvemodel structure stabilityVSAvoidfraud detection accuracy

Core Design Contradiction:

Stability of the object's compositionVSMeasurement precision

Solution Approach 1:

The system performs preliminary evaluation of the learning model's accuracy using authenticated user data before actual fraud detection occurs. By evaluating the model in advance with known authenticated actions, the system can detect accuracy degradation trends and trigger retraining before false positives or false negatives increase during actual fraud detection operations.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system establishes a feedback loop where the evaluation unit continuously monitors the learning model's performance using authenticated user action data. This feedback mechanism provides real-time information about model accuracy degradation, enabling the training unit to update the model with new data patterns and maintain high detection accuracy despite changing user behaviors.

Inventive Principle:

Principle #23Feedback

3Measurement precision

If authenticated user actions are used for evaluation, then the evaluation accuracy is improved, but the system complexity increases

Engineering Contradiction:

Improveevaluation accuracyVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The authenticated information acquisition unit serves multiple functions: it acquires data for model training, provides evaluation data for accuracy assessment, and supplies data for continuous model improvement. This multi-functionality reduces the need for separate dedicated components, thereby limiting the increase in system complexity while maintaining high evaluation accuracy.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The evaluation unit acts as an intermediary that uses authenticated user action data to assess model performance. This intermediary component bridges the gap between raw authenticated data and model evaluation, processing and interpreting the data to provide accurate measurements without requiring direct complex interactions between all system components.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20240202743A1Learning model evaluation system, learning model evaluation method, and program

Publication Date: 2024.06.20 RAKUTEN GROUP INC

AI summary

At least one processor of a learning model evaluation system acquires authenticated information relating to an action of an authenticated user who has executed a predetermined authentication from a user terminal from which a predetermined service is usable. At least one processor acquires, based on the authenticated information, an output from a learning model for detecting fraud in the service. At least one processor evaluates an accuracy of the learning model based on the output corresponding to the authenticated information.