Machine-Learning Entity Matching via Linear Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to accurately match entities across multiple tables with differing data formats, leading to inefficiencies and labor-intensive manual tasks, particularly in enterprise finance and logistics, where matching invoices and shipments is challenging due to non-normalized data.
Innovation Solution
A deep neural network is employed to match entities in semi-structured data, utilizing mixed integer programming and machine learning techniques to efficiently identify matching subsets while avoiding the need for domain-specific features, by converting non-linear objective functions to linear forms for scalable optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional entity matching methods are used across tables with differing data formats, then the system can handle various data types, but the matching accuracy deteriorates and manual effort increases
Solution Approach 1:
The patent transforms the entity matching problem from a complex non-linear optimization problem into a linear optimization problem by changing the parameter representation. Specifically, it uses linear programming with binary variables to represent matching decisions, allowing the system to maintain high accuracy across different data formats while enabling efficient computational solving through standard linear programming solvers.
Solution Approach 2:
The patent introduces an intermediary linear programming formulation that bridges the gap between diverse data formats and the matching objective. By representing matching relationships through linear constraints and objective functions, the system can uniformly process different data types without sacrificing matching precision, as the linear framework serves as a universal mediator for heterogeneous data.
2Measurement precision
If exhaustive search methods are used to find optimal matching subsets, then matching accuracy is maximized, but computational time and resources increase exponentially
Solution Approach 1:
The patent replaces the mechanical exhaustive search approach with a mathematical linear programming system. Instead of systematically enumerating all possible matching subsets (mechanical brute force), the invention formulates the problem as a linear optimization task that can be solved efficiently by standard solvers, reducing computational time from exponential to polynomial complexity while maintaining optimal matching accuracy.
Solution Approach 2:
The patent changes the problem parameters from requiring exhaustive enumeration of all subsets to using linear programming variables and constraints. This parameter transformation allows the system to find optimal matching subsets through efficient linear optimization algorithms rather than time-consuming exhaustive search, achieving the same accuracy goal with dramatically reduced computational resources.
3Reliability
If domain-specific features are engineered for entity matching, then matching performance improves for specific domains, but system complexity and development effort increase
Solution Approach 1:
The patent creates a universal linear programming framework that can handle entity matching across multiple domains without requiring domain-specific feature engineering. The linear optimization model serves as a multi-functional platform that adapts to different domains through flexible constraint and objective function definitions, eliminating the need for separate feature engineering processes for each domain while maintaining high matching performance.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In an example embodiment, a solution for matching entities in a query table with one or more entities in a target table, in the presence of a constraint on the value sum of the matching targets (hereinafter called the "value constraint"), using machine learning techniques, is provided. Specifically, a non-linear objective function is converted to a linear objective function and a machine learning model is trained using the linear objective function, allowing for the use of solver functions from libraries in order to speed matching over existing methods.