Asynchronous Distributed Data Cleansing With Similarity Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of reconciling and processing diverse data representations of real-world entities across large and evolving datasets is compounded by the variety of formats in which entities are represented in electronic form, leading to inefficiencies in data processing and reconciliation.
Innovation Solution
A system that utilizes a similarity search data store, machine learning algorithms, and an asynchronous distributed architecture to process input records, mapping and merging them into normalized master data records by using a normalization engine with components for similarity search, machine learning, and result processing, while asynchronously handling large volumes of data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data processing methods are used to handle diverse data representations, then processing can be performed, but processing efficiency deteriorates and accuracy decreases
Solution Approach 1:
The patent introduces a similarity search data store as an intermediary component between input records and master data records. This intermediary enables efficient matching by pre-indexing master records and allowing rapid similarity queries, thereby improving both processing efficiency and matching accuracy without requiring exhaustive comparisons of all data representations.
Solution Approach 2:
The patent transforms data representations by normalizing diverse formats into standardized master data records. The system changes parameters such as data format, structure, and representation to create a unified schema, which enables efficient processing while maintaining high accuracy in reconciling different data representations of the same entity.
2Speed
If synchronous processing is used for data cleansing, then processing can be performed sequentially, but processing speed decreases and scalability is limited
Solution Approach 1:
The patent divides the data cleansing system into independent modular components including similarity search data store, machine learning processing units, and result processing modules. This segmentation enables parallel processing of different data records across multiple threads or nodes, significantly increasing processing speed while keeping each individual component relatively simple.
Solution Approach 2:
The patent implements an asynchronous dynamic processing architecture where the system can adaptively adjust processing throughput based on data volume and complexity. The machine learning algorithm dynamically processes records at varying rates, allowing the system to scale processing speed according to workload without requiring complex synchronous coordination mechanisms.
3Measurement precision
If machine learning algorithms are applied to all input records, then matching accuracy improves, but processing time increases
Solution Approach 1:
The patent performs preliminary actions by pre-processing master data records and storing them in the similarity search data store with optimized indexing. This preliminary preparation enables rapid retrieval and filtering of candidate matches, so that when machine learning algorithms process records, they only need to evaluate a reduced set of likely matches rather than all possible records, thereby maintaining high accuracy while reducing processing time.
Solution Approach 2:
The patent applies partial action by using the machine learning algorithm selectively on the top N most similar master records returned by the similarity search, rather than applying it to all input records or all possible matches. This partial application of the computationally intensive machine learning algorithm maintains high matching accuracy for the most relevant matches while significantly reducing overall processing time by skipping less likely candidates.
Data Source
AI summary
In one embodiment, the present disclosure pertains to data cleansing. In one embodiment, data cleansing is performed across a distributed master data store asynchronously in a scalable architecture, thereby allowing vast amounts of input records to be processed more efficiently.