LLM-Based Fraud Detection for Online Refund Events
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for determining legitimate refund events in online concierge systems are inconsistent and prone to false positives and negatives due to poor feature inputs, often related to refund sources and reasons that are not reliable or OOS-related, making it difficult to distinguish between genuine out-of-stock issues and fraudulent behavior.
Innovation Solution
Training a fraud detection model using the output of a large language model to analyze transaction data and determine whether a refund event is legitimate or fraudulent by providing a prompt with relevant information such as chat transcripts, customer service interactions, and item data to classify the reason for the refund event.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional refund detection methods are used with pre-coded standardized refund reasons, then the system is easy to operate, but the measurement precision of fraud detection deteriorates due to poor feature inputs
Solution Approach 1:
A language model is introduced as an intermediary component between the refund event data and the fraud detection model. The language model processes unstructured refund reason text and generates structured feature representations that capture semantic meaning, thereby improving the quality of inputs to the fraud detection model without requiring changes to the overall system architecture or user interaction patterns
Solution Approach 2:
The system transforms the parameter representation of refund reasons from fixed pre-coded categories to dynamic semantic features generated by the language model. This parameter transformation enables the fraud detection model to access richer, more nuanced information about refund events, improving detection precision while maintaining the same operational interface
2Device complexity
If pre-coded standardized refund reasons are used, then the device complexity is reduced, but the reliability of fraud detection deteriorates due to inconsistent results
Solution Approach 1:
The language model serves as an intermediary that bridges the simple pre-coded refund reasons and the fraud detection model. It enriches the limited pre-coded reasons with semantic analysis of the full refund reason text, providing the fraud detection model with more reliable features while keeping the overall system architecture simple and the device complexity low
Solution Approach 2:
The system replaces the mechanical classification approach (direct use of pre-coded reasons) with an intelligent processing approach using the language model. This substitution enables the system to extract meaningful features from unstructured text without requiring complex manual classification rules or increasing device complexity
3Measurement precision
If detailed analysis of refund events is performed to improve detection accuracy, then the measurement precision improves, but the loss of time increases due to additional processing
Solution Approach 1:
The language model automatically performs detailed semantic analysis of refund events, replacing what would otherwise require complex manual analysis rules. This automated linguistic processing achieves high measurement precision in fraud detection while minimizing time loss, as the language model processes text efficiently without requiring human intervention or complex rule evaluation
Data Source
AI summary
Embodiments relate to an automatic detection of fraudulent behavior for a transaction at an online system. The online system requests a large language model (LLM) to determine, based on a prompt input into the LLM, information about a refund event for a first order placed by a user of the online system. The online system accesses a computer model trained to detect a fraudulent behavior associated with an order placed with the online system. The online system applies the computer model to determine a score associated with the refund event, based on the information about the refund event received from the LLM. The online system determines, based on the score, whether the refund event was due to a fraudulent behavior of the user. The online system performs at least one action associated with the online system, based on the determination whether the refund event was due to the fraudulent behavior.