Duplicate Data Entry Detection Using ML Feature Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently detecting and mitigating duplicate data entries in databases, leading to inefficiencies in resource management and potential security threats due to unintentional or malicious duplication.
Innovation Solution
A computer-based method that utilizes a processor to analyze data entries with electronic activity values, timestamps, entity identifiers, and location identifiers, generating feature vectors to classify duplicates using a trained model, and provides a graphical user interface for users to dispute incorrect classifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional duplicate detection methods are used to scan all data entries in large databases, then detection completeness is improved, but detection speed and system efficiency deteriorate
Solution Approach 1:
The patent segments the duplicate detection process into multiple stages: first identifying candidate duplicates using entity identifiers and location identifiers, then applying machine learning classification only to these candidates. This segmentation allows comprehensive detection while maintaining speed by avoiding full database scanning.
Solution Approach 2:
The system performs preliminary filtering by matching entity identifiers and location identifiers before applying the more computationally intensive machine learning model. This preliminary action reduces the dataset size significantly, enabling fast detection without sacrificing completeness.
2Measurement precision
If machine learning models are trained on large historical datasets to improve detection accuracy, then classification precision is improved, but training time and computational resources worsen
Solution Approach 1:
The patent extracts only the necessary features (entity identifiers, location identifiers, electronic activity values, and timestamps) from the data entries to create feature vectors for training. This extraction reduces the dimensionality and size of training data while preserving the essential information needed for accurate duplicate detection.
3Measurement precision
If comprehensive data analysis is performed on all data entry attributes, then duplicate detection accuracy is improved, but computational complexity and resource usage worsen
Solution Approach 1:
The patent applies different levels of analysis to different data attributes: exact matching for entity identifiers and location identifiers, proximity-based comparison for timestamps, and machine learning classification for综合判断. This local quality approach optimizes computational resources by applying appropriate analysis methods to each attribute type.
Data Source
AI summary
Systems and methods of the present disclosure enable a processor to automatically detect duplicate data entries by receiving data entries associated with a user, where each data entry includes a value, a time, an entity identifier, and a location. Pairs of similar data entries are determined by matching the entity identifier and the location pairs data entries. Candidate duplicate data entries are determined based on a proximity in time between data entries of the similar data entries. For each candidate duplicate data entry, a feature vector is generated including the entity identifier, location, value and time, and each feature vector is submitted to a duplicate classification model to automatically determine duplicate data entries from the candidate duplicate data entries, the duplicate classification model being trained according to a historical dispute entries.