Statistical Classifier for Source Column Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multi-tenant database systems, accurately mapping columns from a source file to a target database model is challenging due to inconsistencies and inaccuracies in contact information, especially when dealing with structured data and character strings, which hinders the transformation and importation of data into the database model.
Innovation Solution
A statistical classifier using advanced features like Bayes classifier, last word, digit-blurring, words, and full string features is employed to recognize entities in character strings and map columns accurately, incorporating probabilistic scoring to determine the likelihood of each string belonging to defined entities, thereby facilitating the transformation of source files into the target database model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If conventional database approaches are used to maintain contact information, then individuals can access and manage their own data, but inconsistencies and inaccuracies in contact information arise
Solution Approach 1:
The patent introduces an intermediary system (the database system administrator or automated validation system) that mediates between individual data submission and final data storage. This intermediary validates, standardizes, and verifies contact information before it is stored, thereby maintaining individual data management convenience while ensuring data accuracy and consistency across the organization.
2Ease of manufacture
If manual methods are used to identify entities in character strings from source files, then data can be imported, but accurate mapping of columns to database model entities is difficult
Solution Approach 1:
The patent replaces manual mechanical methods of entity identification with automated computational methods. The system uses statistical analysis, pattern recognition, and machine learning algorithms to automatically identify entities in character strings and map source file columns to database model entities, thereby maintaining ease of data import while significantly improving mapping accuracy and eliminating human error.
Solution Approach 2:
The patent transforms the entity identification problem from a qualitative manual judgment task into a quantitative automated process by analyzing multiple parameters simultaneously (string length, character patterns, positional information, contextual relationships). By changing from single-parameter manual assessment to multi-parameter automated analysis, the system achieves both operational simplicity and high precision in column mapping.
3Productivity
If simple entity recognition methods are used for character strings, then processing is faster, but accuracy in recognizing entities and mapping columns deteriorates
Solution Approach 1:
The patent applies preliminary filtering and preprocessing steps that quickly eliminate obviously incorrect entity matches before applying more complex and time-consuming analysis methods. By performing preliminary actions such as basic pattern matching and rule-based filtering first, the system maintains high processing speed while reserving advanced computational resources for cases that require higher precision, thus achieving both speed and accuracy.
Data Source
AI summary
A system and method for mapping columns from a source file to a target file. The header for each source column is evaluated heuristically to see if the header matches a predefined entity. The contents of a group of cells in the source column are evaluated probabilistically to determine a probability that the cell contents correspond to at least one of the predefined entities. A score is assigned to the likelihood that the column corresponds to one or more predefined entities. If the score meets a threshold, then the correspondence between the source column and one or more predefined entities is mapped. If the score fails to meets the threshold, then the correspondence between the source column and one or more undefined entities is mapped. Finally, each source column is transformed into a target column in accord with the map.