Data Record Anomaly Reconciliation Using Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems often reject entire data records due to anomalous line items, even if the overall record satisfies pricing rules, leading to manual and labor-intensive review processes that cause delays.
Innovation Solution
A method using machine learning models to assign line items to clusters, calculate neighborhood scores, and adjust parameters based on predefined rules to reconcile anomalies, thereby producing a reconciled data record.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional rule-based validation is applied to each line item, then pricing rule compliance is ensured, but processing time increases and approval efficiency decreases
Solution Approach 1:
The patent segments the validation process into two levels: line item level (clustering and neighborhood scoring) and data record level (overall anomaly scoring). This hierarchical segmentation allows efficient processing at the line item level through clustering, while maintaining comprehensive validation at the record level, thus resolving the contradiction between reliability and productivity.
Solution Approach 2:
The patent performs preliminary clustering of line items and calculation of neighborhood scores before final anomaly detection. By pre-processing and organizing line items into clusters with established neighborhood relationships, the system reduces the computational burden during actual validation, improving approval efficiency while maintaining pricing rule compliance.
2Measurement precision
If manual review processes are used for anomalous line items, then accurate anomaly detection is achieved, but labor intensity increases and processing delays occur
Solution Approach 1:
The patent implements self-service anomaly detection through automated machine learning models that independently identify anomalies in line items and data records. The system performs clustering, calculates neighborhood scores, and detects anomalies without requiring manual intervention, thereby maintaining high detection accuracy while eliminating processing delays and labor intensity associated with manual reviews.
3Reliability
If entire data records are rejected due to single anomalous line items, then pricing rule violations are prevented, but valid quotations are lost and productivity decreases
Solution Approach 1:
The patent applies local quality by treating each line item and data record with differentiated validation intensity. Line items are clustered and evaluated based on their neighborhood characteristics, allowing anomalies to be identified and corrected locally without automatically rejecting the entire data record. This selective, localized approach maintains pricing rule enforcement while preserving valid quotations, thus improving the approval rate.
4Measurement precision
If comprehensive line item validation is performed, then anomaly detection accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent merges similar line items into clusters based on their characteristics, combining them into groups with shared properties. By performing validation at the cluster level rather than individually for each line item, the system reduces computational complexity while maintaining comprehensive anomaly detection accuracy through the neighborhood scoring mechanism that considers relationships within clusters.
Data Source
AI summary
Techniques are provided for data record anomaly reconciliation using machine learning models. One method comprises obtaining a data record comprising multiple line items; assigning the line items to a given cluster of similar line items to determine a line item neighborhood score for each line item based on a comparison of a given line item to other available line items in the assigned cluster; applying features of the data record to a machine learning model to determine a data record score for the data record based on a combination of the line item neighborhood scores for the data record; identifying anomalies in the data record based on the data record score and/or the line item neighborhood scores for the data record; and adjusting parameters of the line items to address the anomalies identified in the data record to produce a reconciled data record, based on the line item neighborhood scores and/or predefined adjustment rules.