Metadata Matching Rules for Characterizing Match Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In metadata discovery, matching data elements across diverse sources with different naming standards and formats is challenging, as exact matches are often complicated by transformations, synonyms, and categorizations, requiring a method to characterize match quality effectively.
Innovation Solution
A system and method that preprocess data elements to generate source processing information, including rules for characterizing matches, using techniques like stemming, synonym lookup, and TF-IDF weighting to normalize and transform terms, and assign grades to match quality, enabling efficient matching across multiple sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If metadata discovery is performed across diverse sources with different naming standards and formats, then the ability to find matches between data elements is improved, but the complexity of characterizing match quality increases
Solution Approach 1:
The match quality characterization is divided into multiple independent grading criteria (exact match, stem match, synonym match, category match, partial match) that can be evaluated separately and combined. This segmentation allows the system to handle diverse naming standards through multiple specialized matching approaches rather than one complex unified approach.
Solution Approach 2:
A standardized metadata schema acts as an intermediary framework that translates diverse naming standards and formats from different sources into a common structure. This intermediary layer enables match characterization by providing a unified basis for comparison while preserving the ability to handle source-specific variations.
2Quantity of substance
If a large number of metadata sources are processed, then the completeness of the master metadata collection is improved, but the processing time and computational resources increase
Solution Approach 1:
Metadata sources are pre-processed and standardized before being added to the master collection. This preliminary action includes transforming diverse naming standards into a common format, extracting key attributes, and organizing data structures in advance. This preparation reduces the computational burden during subsequent matching operations and enables faster processing when new sources are added.
Solution Approach 2:
The system transforms metadata from various sources by changing key parameters such as naming conventions, data formats, and structural organization into a standardized form. This parameter transformation enables efficient processing and comparison while maintaining the ability to handle a large quantity of diverse metadata sources.
3Measurement precision
If exact matching is used between data elements, then the precision of matches is improved, but the number of matches found decreases due to transformations and synonyms
Solution Approach 1:
The system merges multiple matching approaches (exact match, stem match, synonym match, category match) into a unified grading system. Each approach contributes to the overall match quality assessment, allowing the system to maintain high precision through exact matching while also capturing additional matches through synonym and category-based approaches.
Solution Approach 2:
The matching system dynamically adjusts the strictness of matching criteria based on the context and data characteristics. For fields where exact precision is critical, stricter matching is applied, while for fields where semantic equivalence is more important, more flexible matching approaches are used. This dynamic approach balances precision with the quantity of matches found.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Processing metadata includes storing, in a data storage system, a specification for each of multiple sources, each specification including information identifying one or more data elements of the corresponding source; and processing, in a data processing system coupled to the data storage system, data elements from the sources, including generating a set of rules for each source based on a corresponding one of the stored specifications, and matching data elements of different sources and determining a quality metric characterizing a given match between a first data element of a first source and a second data element of a second source according to the set of rules generated for the first source and the set of rules generated for the second source.