Auto-mapping Data Attributes via Metadata Similarity Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The current data transformation process requires significant manual effort for mapping data attributes between source and target systems, which is inefficient due to the large number of tables and attributes involved, typically exceeding 10,000 mappings in medium projects.
Innovation Solution
A processor-implemented method and system that auto-map source and target data attributes based on metadata similarity using techniques like fuzzy string matching, Hybrid Tokenization, and pattern matching, computing weighted scores to determine associations and reduce manual effort.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual mapping is performed by data analysts, then mapping accuracy can be maintained through domain understanding, but the time consumption and effort increase significantly due to the large number of attributes (10,000+ mappings)
Solution Approach 1:
The system performs self-service by automatically computing similarity scores between source and target attributes using metadata comparison algorithms. The auto-mapping engine independently identifies and proposes mappings without requiring manual analyst intervention for each attribute pair, thereby reducing time consumption while maintaining accuracy through algorithmic similarity assessment.
Solution Approach 2:
The patent replaces the mechanical manual process of attribute mapping with an automated computational system. The manual cognitive process of analysts comparing attributes is substituted by machine-based metadata similarity computation, using algorithms to calculate similarity scores and generate mapping proposals automatically.
2Adaptability or versatility
If the number of tables and attributes is increased to handle complex data transformation projects, then the system's capability to handle diverse data is improved, but the complexity of the mapping activity increases significantly
Solution Approach 1:
The patent segments the complex mapping task into manageable components by comparing attributes individually based on their metadata characteristics. Each attribute is evaluated separately using similarity scoring, allowing the system to handle large numbers of attributes systematically without being overwhelmed by overall complexity.
Solution Approach 2:
The system changes the approach from manual holistic assessment to parameter-based automated comparison. By transforming the mapping problem into a parameter-driven similarity computation task (comparing metadata parameters like data types, lengths, and constraints), the system can handle complex transformations with many attributes through standardized parameter comparison rather than complex manual analysis.
3Productivity
If automated mapping is implemented to reduce manual effort, then productivity is improved, but the requirement for sophisticated similarity computation algorithms increases system complexity
Solution Approach 1:
The system creates simplified copies of attribute metadata for comparison purposes. Instead of implementing complex semantic understanding algorithms, the patent uses metadata copying and string similarity computation (such as token-based comparison) to generate similarity scores, achieving automated mapping through relatively simple computational techniques.
Solution Approach 2:
The patent introduces an intermediary similarity score as a mediator between source and target attributes. Rather than directly establishing complex mapping relationships, the system uses computed similarity scores as an intermediate representation that simplifies the mapping decision process, making the automated system more manageable while maintaining productivity.
Data Source
AI summary
In computing, an unresolved problem of high manual effort required for mapping data attributes between the source and the target systems involved in process of the data transformation. This disclosure relates to method of auto-mapping of a plurality of source data attributes and a plurality of target data attributes associated with one or more applications based on similarity of a metadata information is provided. The processor implemented method include processing, a corresponding source table to a target table based on similarity associated with a table name; determining, a candidate source data attributes for at least one target data attribute based on similarity associated with a data attribute name; computing, a weighted score of each of the candidate source data attributes for corresponding at least one target data attribute; and mapping, the weighted score to determine an association of a final data attribute.