Cloud Normalization Service Clustering for Supplier Record Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Online, real-time, multi-tenant transaction processing systems face inefficiencies due to duplicate supplier records, leading to excessive storage consumption and CPU cycles, as multiple buyer representatives create similar records with slight variations, cluttering databases and complicating data management.
Innovation Solution
A computer-implemented method using a normalization service in a cloud computing facility to cluster similar supplier records based on exact matches and most common values, employing data cleaning functions to remove invalid values and filter outliers, thereby consolidating records and reducing duplicates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple buyer representatives create supplier records independently, then each representative can quickly configure transactions, but duplicate supplier records accumulate in the database
Solution Approach 1:
The system performs preliminary actions by creating temporary clusters of potentially duplicate records during data entry, and executes deduplication operations before records are permanently stored in the database, preventing duplicate accumulation while maintaining fast configuration speeds
2Reliability
If all supplier records are stored in the database without filtering, then data completeness is maintained, but storage space and CPU cycles are wasted
Solution Approach 1:
The patent introduces temporary clusters as intermediary structures that hold potentially duplicate records during analysis, allowing the system to identify and consolidate duplicates before permanent storage, thus maintaining data completeness while reducing storage and processing waste
3Quantity of substance
If similar supplier records are clustered and consolidated, then storage efficiency improves, but the complexity of data processing increases
Solution Approach 1:
The patent segments the data processing task into distinct phases: initial clustering based on key fields, temporary cluster formation, similarity comparison, and consolidation. This segmentation makes the complex deduplication process more manageable and efficient
4Ease of operation
If duplicate records are not consolidated, then data retrieval operations are simple, but search accuracy and report generation become inefficient
Solution Approach 1:
The patent creates temporary cluster representations that copy and group similar records together during processing, allowing the system to perform efficient similarity comparisons and consolidations without permanently altering the original data structure, thus maintaining simplicity while improving search accuracy
Data Source
AI summary
A computer-implemented method, comprising: using a normalization service of a first application virtual compute instance hosted in and executing in a cloud computing facility, accessing a plurality of digitally stored data records of a transaction processing system, each of the data records comprising a plurality of data values for a corresponding plurality of attributes; using the normalization service, executing a data cleaning operation on the data records by programmatically calling two or more data cleaning functions, each of the data cleaning functions corresponding to a particular attribute among the plurality of attributes; using the normalization service, creating and storing a plurality of clusters in memory of the first application virtual compute instance by clustering the data records based on exact matches of the data values of a first attribute and creating and storing references to the data records in the clusters; using the normalization service, joining two or more clusters among the plurality of the clusters in the memory to produce two or more joined clusters based on matches of most common values in the data records referenced in the plurality of clusters; updating the data records by adding cluster identifiers of the joined clusters.