Semantic Type Detection for Unlabeled Data Schema Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual mapping of unlabeled input data onto a target schema is time-consuming and prone to errors, especially with complex data schemas and large volumes of data, where human intervention is intensive and error-prone.
Innovation Solution
The implementation of a system that automatically maps unlabeled input data to a target schema through semantic type detection using deep learning methods, clustering fields and data elements, and updating probabilities based on global statistics to determine accurate pairings between data columns and field clusters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual mapping methods are used to map input data to target schema, then mapping accuracy can be maintained through human judgment, but the time consumption and labor intensity increase significantly
Solution Approach 1:
The patent replaces the manual mechanical process of schema mapping with an automated computational system that uses deep learning models and semantic analysis algorithms to perform mapping tasks, thereby eliminating human labor while maintaining or improving mapping accuracy through consistent algorithmic application
Solution Approach 2:
The system transforms the mapping problem from a manual qualitative assessment into a quantitative computational process by converting schema elements and data elements into vector representations and comparing them using mathematical similarity metrics, enabling automated decision-making based on computed parameters
2Reliability
If manual mapping is performed for complex data schemas, then detailed human analysis can ensure correct mappings, but the effort and resources required continue to increase
Solution Approach 1:
The patent substitutes human expert analysis with an automated system comprising deep learning models, embedding generators, and matching algorithms that systematically process complex schema relationships, thereby maintaining reliability through consistent computational evaluation while reducing the need for human expertise in complex scenarios
Solution Approach 2:
The mapping system is divided into distinct modular components including embedding generation, similarity computation, threshold-based filtering, and iterative refinement stages, allowing each component to specialize in specific aspects of the mapping process and enabling independent optimization and debugging
3Productivity
If automated mapping methods are implemented, then processing speed and efficiency improve, but mapping precision may deteriorate due to lack of human judgment
Solution Approach 1:
The system achieves both automation and precision by transforming semantic relationships into quantifiable vector embeddings and using mathematical similarity metrics with configurable thresholds, allowing the automated system to make precise comparisons based on semantic meaning rather than simple keyword matching
Solution Approach 2:
The patent incorporates iterative refinement mechanisms where initial mapping results are evaluated against confidence thresholds, and low-confidence mappings are subjected to additional processing or human review, with feedback loops that continuously improve mapping quality through learned patterns from successful mappings
4Measurement precision
If deep learning methods are used for semantic type detection, then automatic mapping accuracy improves, but computational complexity and processing time increase
Solution Approach 1:
The system performs preliminary processing by pre-computing vector embeddings for schema elements and data elements before the actual matching process, storing these representations for efficient retrieval and comparison, thereby separating the computationally intensive embedding generation from the iterative matching operations
Solution Approach 2:
The computational process is segmented into distinct phases: embedding generation, similarity computation, threshold filtering, and iterative refinement, with each phase handling specific computational tasks that can be optimized independently and parallelized where applicable
Data Source
AI summary
Automatically mapping unlabeled input data onto a target schema via semantic type detection is described. The input data includes data elements that are structured as 2D table rows and columns forming cells. Each data element is included in a cell. The target schema includes a set of fields. Schema mapping includes mapping each column to one or more fields. More particularly, the fields are clustered into field clusters, where each field cluster includes one or more of the fields. Each column is automatically mapped to one of the field clusters of the set of field clusters. The mapping between schema fields and data columns is automatically performed based on appropriate pairings of the detected semantic types, where the semantic types are encoded in vector representations of the fields, the field clusters, and the data elements.