Data Mapper Tool for Automated Column Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual data mapping between external and internal systems is inefficient and error-prone, especially for large data sets, requiring significant time and resources to establish accurate mappings for data import.
Innovation Solution
A data mapper tool that automatically identifies similar columns in the internal database by comparing entries from external data, generating vectors based on term frequencies, and determining similarity measures to suggest mapping candidates, thereby streamlining the data import process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual data mapping is performed for large data sets, then mapping accuracy can be maintained through human review, but the process becomes highly inefficient and time-consuming
Solution Approach 1:
The system performs self-service by automatically generating data mappings without human intervention. The processor compares external data entries with internal database entries, calculates similarity measures, and automatically identifies mapping candidates, allowing the system to serve itself rather than requiring manual human mapping for each data import operation
Solution Approach 2:
The system transforms the mapping problem into a parameter-based similarity calculation problem. By converting data entries into comparable parameters and calculating similarity measures between external and internal data, the system automatically determines mapping relationships without manual intervention, resolving the contradiction between automation and accuracy
2Reliability
If manual data mapping is performed for large data sets, then mappings can be carefully reviewed, but the process requires days or weeks of time investment
Solution Approach 1:
The system replaces the mechanical manual process of data mapping with an automated computational system. The processor automatically compares data entries, calculates similarity measures, and generates mappings, substituting human mechanical work with automated information processing that completes in minutes rather than days or weeks
Solution Approach 2:
The system introduces similarity measures as an intermediary mechanism between external and internal data. By calculating and comparing similarity measures automatically, the system mediates the mapping process, enabling rapid automated decision-making about data correspondence without requiring manual review of each mapping
3Adaptability or versatility
If manual data mapping is performed, then mappings can be customized for specific needs, but the process becomes error-prone for large data sets
Solution Approach 1:
The system performs self-service by automatically generating data mappings without human intervention. The processor compares external data entries with internal database entries, calculates similarity measures, and automatically identifies mapping candidates, allowing the system to serve itself rather than requiring manual mapping for each data import operation
Solution Approach 2:
The system uses feedback from similarity measure calculations to automatically adjust and determine mapping relationships. By continuously comparing external data with internal database entries and using the similarity feedback to identify the best mapping candidates, the system maintains high accuracy while automating the process
Data Source
AI summary
An apparatus includes a processor. The processor extracts a column from an external source for import into a database configured to store a set of columns including a first and second column. The processor splits the entries of the import column into a set of terms. The processor generates a first, second, and third vector based on the frequency of each term of the set of terms in the first, second, and import columns, respectively. The processor determines a first similarity measure between the first and third vectors and a second similarity measure between the second and third vectors. The first similarity measure is greater than the second. In response, the processor provides an indication to a user that the first column is a mapping candidate for the import column, such that entries of the import column may be stored in the database as additional entries in the first column.