Neural Encoding of Structured Data for Exact Record Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data matching techniques in database systems are resource inefficient and fail to provide accurate, exact matches between data sets, often resulting in ranked lists that require additional processing to identify the correct matches, and are not adaptable to new scenarios.
Innovation Solution
Implement deep learning techniques using neural networks to encode structured data sets based on column type, generating multi-dimensional vectors for each data set, and using a loss-function to determine exact matches between data sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional rules-based approaches are used for data matching, then the system is simple to implement, but the matching accuracy is low and resource efficiency is poor
Solution Approach 1:
The patent replaces traditional rules-based mechanical matching systems with a neural network-based deep learning system. The neural network automatically learns matching patterns from data without requiring explicit rule definitions, thereby improving matching accuracy while the automated learning process reduces the need for manual rule maintenance and system complexity management.
Solution Approach 2:
The patent transforms structured data into multi-dimensional vector representations, changing the parameter space from discrete data values to continuous vector embeddings. This transformation enables the neural network to capture complex relationships and similarities between data points, significantly improving matching accuracy beyond what traditional rule-based systems can achieve.
2Loss of time
If traditional ranked list approaches are used, then additional processing is required to identify exact matches, but the system can handle large data sets
Solution Approach 1:
The patent replaces the two-stage process of traditional ranked list generation followed by manual exact match identification with a single-stage neural network system that directly outputs exact matches. The loss function is specifically designed to identify exact matches without requiring subsequent filtering or ranking processing, thereby reducing processing time and improving efficiency.
Solution Approach 2:
The patent performs preliminary encoding of all data into multi-dimensional vectors before the matching process. This pre-processing transforms the data into a format that enables direct comparison and exact match identification, eliminating the need for subsequent processing steps required by traditional approaches that generate ranked lists first.
3Adaptability or versatility
If traditional data matching systems are used, then they can process structured data, but they are not adaptable to new scenarios
Solution Approach 1:
The patent implements a dynamic system where the neural network can be retrained and adapted to new scenarios by providing new training data. The system's parameters (weights and biases) automatically adjust through the loss function optimization process, enabling adaptability to new matching scenarios without requiring system redesign, while the efficient vector-based processing maintains resource consumption at acceptable levels.
Data Source
AI summary
Implementations of the present disclosure include methods, systems, and computer-readable storage mediums for receiving first and second data sets, both the first and second data sets including structured data in a plurality of columns, for each of the first data set and the second data set, inputting each column into an encoder specific to a column type of a respective column, the encoder providing encoded data for the first data set, and the second data set, respectively, providing a first multi-dimensional vector based on encoded data of the first data set, providing a second multi-dimensional vector based on encoded data of the second data set, and outputting the first multi-dimensional vector and the second multi-dimensional vector to a loss-function, the loss-function processing the first multi-dimensional vector and the second multi-dimensional vector to provide an output, the output representing matched data points between the first and second data sets.