Data Cleansing System Using ML for De-duplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data management systems face inefficiencies due to data duplication and inconsistencies in databases, particularly when handling free text formats, leading to redundant and incorrect data entries, which complicates pricing analysis and memory constraints.
Innovation Solution
A method and system utilizing machine learning models to cleanse and de-duplicate data by filtering unnecessary records, generating similarity vectors, and applying matching rules to identify and merge duplicate records, thereby creating a de-duplicated cleansed database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If free text format is used for data entry, then ease of data entry is improved, but data quality and consistency deteriorate due to redundant and incorrect entries
Solution Approach 1:
The system performs preliminary data cleansing and de-duplication processing before data is fully integrated into the database. Cleansing rules are applied in advance to standardize formats, and de-duplication algorithms identify and merge duplicate records before they cause problems in pricing analysis or vendor rationalization.
Solution Approach 2:
The patent replaces manual data verification and cleaning processes with automated machine learning models and algorithms. The system uses automated similarity comparison, machine learning-based de-duplication, and rule-based cleansing to substitute human effort, improving both efficiency and consistency while maintaining ease of entry.
2Quantity of substance
If duplicate records are stored in the database, then data volume is increased, but system efficiency and memory usage deteriorate
Solution Approach 1:
The system extracts and removes duplicate records from the database through automated de-duplication processes. By identifying redundant data and extracting only unique records, the system reduces data volume while maintaining the integrity of essential information, thereby improving system efficiency and reducing memory constraints.
Solution Approach 2:
The patent implements a process where duplicate records are discarded and only essential unique data is retained in the database. The system recovers and preserves important information while eliminating redundant copies, optimizing storage efficiency and improving productivity in pricing analysis and vendor rationalization.
3Reliability
If comprehensive data cleansing and de-duplication is performed, then data quality is improved, but processing time and computational resources increase
Solution Approach 1:
The system applies partial cleansing and de-duplication actions by focusing on the most critical data quality issues first. Machine learning models are trained on representative samples and then applied to larger datasets, achieving substantial data quality improvement without processing every single record with full computational intensity.
Solution Approach 2:
The patent implements self-service mechanisms where the system automatically learns from data patterns and improves its cleansing and de-duplication capabilities over time. The machine learning models adapt and refine their algorithms based on feedback, reducing the need for manual intervention and decreasing processing time while maintaining high data quality standards.
Data Source
AI summary
Method and system for cleansing and de-duplicating data in database are provided. The method includes filtering garbage records from a plurality of records based on data fields, and applying cleansing rules to create a cleansed database. A similarity vector is generated, where each vector corresponds to pairwise comparison of distinct data entries in cleansed database. Matching rules are applied to label each vector as one of matched, unmatched and unclassified. The method analyzes the vectors labeled as matched and unmatched to train a machine learning model to identify duplicates in the cleansed database. Unclassified vectors in the cleansed database are labeled as matched or unmatched by applying machine learning model on unclassified vectors. Thereafter, the method processes all the vectors labeled as matched to create clusters of records that are duplicates of each other. Further, records in each cluster are merged to obtain de-duplicated cleansed database using predefined consolidated rules.