Neural Entity Matching for Semi-Structured Table Reconciliation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems struggle to match entities across multiple tables due to inconsistent data normalization, making it difficult to identify and reconcile different formats of information, and prior machine-learned models require domain knowledge that is often unavailable.
Innovation Solution
A deep neural network is used to learn relationships and patterns from historical data, without requiring domain knowledge, to match entities by utilizing separate neural networks for structured and unstructured fields, and employing a decomposable attention and aggregation component for probabilistic scoring.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional machine-learned models are used to match entities, then matching can be performed, but domain knowledge is required which is often unavailable
Solution Approach 1:
The deep neural network performs self-learning by automatically extracting features and relationships from historical data without requiring external domain knowledge. The model trains itself on the data patterns, eliminating the need for manual feature engineering or domain expertise to configure matching rules.
Solution Approach 2:
The system transitions from static, manually-defined matching parameters to dynamic, learned parameters. The neural network adjusts its internal parameters through training on historical data, adapting to different data formats and relationships automatically rather than requiring pre-programmed domain-specific parameters.
2Measurement precision
If data is stored in normalized formats, then matching across tables becomes easier, but flexibility in handling different data formats is reduced
Solution Approach 1:
The deep neural network dynamically adjusts its feature extraction and comparison parameters based on the actual data formats it encounters. Rather than relying on fixed normalization rules, the model learns optimal parameters from historical data, enabling it to handle both normalized and unnormalized formats with high matching accuracy.
Solution Approach 2:
The neural network serves as a universal matching engine that can handle multiple data formats simultaneously. Through its learned representations, it performs the function of both data normalization and pattern recognition, making it adaptable to various formats while maintaining precise matching capability.
3Reliability
If manual feature engineering is used, then domain-specific matching can be achieved, but the system becomes less generic and harder to apply across different contexts
Solution Approach 1:
The system performs self-learning by automatically discovering relevant features and relationships from historical data in each specific context. This self-service approach allows the model to adapt its feature engineering to the specific domain at hand, achieving reliable matching without requiring pre-defined domain-specific features.
Solution Approach 2:
The neural network performs preliminary learning from historical data before actual matching operations. This preliminary training action enables the model to internalize domain-specific patterns and relationships, allowing it to achieve reliable matching in specific contexts while maintaining the ability to generalize to new contexts through continued learning.
Data Source
AI summary
In an example embodiment, a deep neural network may be utilized to determine matches between candidate pairs of entities, as well as confidence scores that reflect how certain the deep neural network is about the corresponding match. The deep neural network is also able to find these matches without requiring domain knowledge that would be required if features for a machine-learned model were handcrafted, which is a drawback of prior art machine-learned models used to match entities in multiple tables. Thus, the deep neural network improves on the functioning of prior art machine learned models designed to perform the same tasks. Specifically, the deep neural network learns the relationships of tabular fields and the patterns that define a match from historical data alone, making this approach generic and applicable independent of the context.