Neural Encoders for Exact Matching of Structured Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data matching techniques in database systems are resource inefficient and fail to provide accurate, exact matches, often resulting in ranked lists of potential matches that require additional resources and time to identify the correct match.
Innovation Solution
The implementation of deep learning methods using neural networks to encode structured data sets into multi-dimensional vectors, allowing for exact matching between query and target data sets by processing the intrinsic characteristics of the data structure, rather than relying on rule-based approaches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional rule-based approaches are used for data matching, then the system can provide a ranked list of candidate matches, but additional resources and time are needed to identify the exact match from the ranked list
Solution Approach 1:
The patent replaces traditional rule-based mechanical matching systems with a neural network-based deep learning system. The neural network encoders automatically learn and apply complex matching rules and patterns from training data, substituting manual rule configuration and ranked list evaluation with automated vector-based exact matching. This resolves the contradiction by providing both high accuracy and efficiency simultaneously.
Solution Approach 2:
The patent transforms data from traditional tabular formats into multi-dimensional vector representations. By changing the parameter representation from discrete field values to continuous vector spaces, the system enables direct exact matching through vector comparison operations, eliminating the need for ranked list evaluation and significantly reducing matching time while maintaining high accuracy.
2Productivity
If traditional rule-based approaches are used for data matching, then the system can operate with simple rules, but the system is resource inefficient
Solution Approach 1:
The patent implements preliminary action by pre-training neural network encoders on large datasets before deployment. The encoders learn optimal matching patterns and representations in advance, so that during actual data matching operations, the system can efficiently compute vector embeddings and perform rapid comparisons without requiring complex rule evaluations, thereby improving productivity while managing computational resources effectively.
Solution Approach 2:
The patent uses copying by creating vector representations (embeddings) of data records that capture their essential characteristics. These vector copies can be rapidly compared using efficient distance metrics, avoiding the need to repeatedly evaluate complex rules against original data structures. This copying mechanism significantly improves matching efficiency while reducing computational overhead during operation.
3Reliability
If traditional rule-based approaches are used for data matching, then the system can provide matched results, but the system fails to provide an accurate, or clear result
Solution Approach 1:
The patent implements feedback mechanisms through the neural network training process, where the system learns from labeled training data containing correct match examples. The loss function provides continuous feedback during training, adjusting encoder parameters to minimize mismatches and maximize correct identifications. This feedback loop ensures high reliability and precision in the final matching results, resolving the contradiction between reliability and precision.
Data Source
AI summary
Implementations of the present disclosure include methods, systems, and computer-readable storage mediums for receiving first and second data sets, both the first and second data sets including structured data in a plurality of columns, for each of the first data set and the second data set, inputting each column into an encoder specific to a column type of a respective column, the encoder providing encoded data for the first data set, and the second data set, respectively, providing a first multi-dimensional vector based on encoded data of the first data set, providing a second multi-dimensional vector based on encoded data of the second data set, and outputting the first multi-dimensional vector and the second multi-dimensional vector to a loss-function, the loss-function processing the first multi-dimensional vector and the second multi-dimensional vector to provide an output, the output representing matched data points between the first and second data sets.