Cloud-First API Cluster for Batch and Real-Time Entity Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current information and data processing systems lack an efficient method for determining data quality in real-time, especially when dealing with large datasets such as those containing 100 million entries, leading to issues like over/under utilization of resources, inefficient data handling, and imprecise data resolution.
Innovation Solution
A computer-implemented method utilizing a cloud-first multifunction API cluster with microservices capable of both batch and real-time processing. This system employs a microservice module that leverages containerization technology to scale APIs, cleanse data, determine blocking engine status, generate candidate entity pairs, and utilize machine learning to score matches, ultimately merging records based on user selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data processing systems are used to handle large datasets, then data can be processed, but processing speed is slow and resource utilization is inefficient
Solution Approach 1:
The system divides the data processing task into multiple independent microservices (blocking engine, feature engine, machine learning engine, etc.), each handling specific functions. This segmentation allows parallel processing and improves throughput while maintaining manageable complexity through modular architecture
Solution Approach 2:
The patent introduces a cloud-based distributed architecture that adds a new dimensional layer to processing - moving from single-node to multi-node distributed computing. This enables horizontal scaling and leverages cloud resources to achieve high processing speeds without proportionally increasing on-premise system complexity
2Quantity of substance
If traditional systems process large datasets, then data can be handled, but memory requirements are excessive
Solution Approach 1:
The processing pipeline segments data handling into stages where different microservices process subsets of data independently. The blocking engine filters records, the feature engine processes features, and the machine learning engine scores matches - each stage working with reduced data volumes compared to processing the entire dataset at once
Solution Approach 2:
The system dynamically adjusts processing parameters based on data characteristics and workload. The machine learning engine receives pre-processed features and blocks, adjusting its scoring parameters to efficiently handle large datasets without requiring proportional increases in memory, leveraging algorithmic optimizations instead
3Reliability
If real-time processing is implemented, then data quality can be determined immediately, but processing load on blocking engine increases
Solution Approach 1:
The blocking engine performs preliminary filtering and blocking of records before they reach the more computationally intensive feature engine and machine learning engine. By pre-processing and eliminating obviously non-matching records early in the pipeline, the system achieves real-time data quality determination while reducing the processing load on subsequent stages
Solution Approach 2:
The system dynamically routes records through the processing pipeline based on their characteristics and the current state of each microservice. The blocking engine can adjust its filtering thresholds and the machine learning engine can dynamically tune its scoring parameters to maintain real-time processing capability while optimizing resource utilization under varying load conditions
Data Source
AI summary
The disclosure is related to computer-based information processing systems configured to resolve entity records in a database. Specifically, the systems and methods utilize natural language models in a cloud-first multifunction API cluster to provide microservices capable of both batch and real-time processing of a large number of entity records.