Transactional Data Name Normalization With Staged Matching Logic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for categorizing and normalizing transactional data face challenges such as low accuracy, high resource consumption, and non-transferability of Machine Learning models, leading to unorganized record maintenance and inefficient storage.
Innovation Solution
An end-to-end orchestration process involving a data management system with a mapping generator, data classifier, and name normalizer to create a global table, classify transactional data into categories, and normalize names using multiple determinations and search engine interactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If direct lookup on database is used to match raw names, then normalization process is simple, but accuracy of normalization is low
Solution Approach 1:
The normalization process is divided into multiple stages: first preprocessing the received name to remove content from a first list, then performing direct lookup, and if that fails, submitting to search engine and performing second preprocessing. This segmentation allows each stage to handle specific cases, improving overall accuracy while maintaining simplicity where possible.
Solution Approach 2:
The system performs first preprocessing of the received name before attempting direct lookup, removing certain content in advance. This preliminary action prepares the data for more accurate matching in subsequent steps, improving normalization accuracy without significantly increasing complexity.
2Adaptability or versatility
If third party search engine is used for direct lookup, then normalization coverage is improved, but resource consumption increases and query limits are reached
Solution Approach 1:
The system segments the normalization approach by attempting direct lookup on the database first (which is faster and consumes fewer resources), and only submitting to the search engine when direct lookup fails. This reduces overall resource consumption while maintaining good normalization coverage.
Solution Approach 2:
The system performs partial preprocessing (removing content from first list) before direct lookup, and only performs full preprocessing with search engine submission when necessary. This partial action approach reduces resource consumption while achieving sufficient normalization for most cases.
3Measurement precision
If iterative queries are submitted to search engine for invalid raw names, then normalization accuracy improves, but query submission limit is exceeded and processing is delayed
Solution Approach 1:
The system performs first preprocessing to remove content from a first list before submitting to the search engine. This preliminary action reduces the likelihood of invalid queries and improves the quality of submissions, reducing the need for iterative queries and staying within submission limits.
Solution Approach 2:
The system extracts and removes specific content from names before search engine submission, taking out elements that would cause invalid queries. This extraction improves query validity and reduces the need for iterative submissions, maintaining both accuracy and productivity.
4Measurement precision
If multiple preprocessing steps are performed to improve matching accuracy, then normalization accuracy improves, but processing time increases
Solution Approach 1:
The preprocessing is segmented into two distinct stages: first preprocessing (removing content from first list) performed before direct lookup, and second preprocessing (removing content from second list) performed after search engine submission. This segmentation allows the system to perform minimal preprocessing when possible, reducing processing time while maintaining accuracy when needed.
Solution Approach 2:
The system performs partial preprocessing (first preprocessing only) for most cases that match directly, and performs additional second preprocessing only when search engine results need to be processed. This partial action approach balances matching accuracy with processing time efficiency.
Data Source
AI summary
Methods, systems, and computer-readable storage media for classification and normalization of transactional data. Description for each name included in the transactional data is extracted. Words and a frequency of words in the description with respect to each of features pre-defined for the transactional data are identified. Based on weights of the words and features and feature encoding rules, the transactional data is classified and names from the transactional data are identified for normalization. Further, one or more of various types (type 1, 2, 3, and 4) of determinations are performed to determine whether the names identified to be normalized match with any of raw names present in a global table of a database. In response to positive results of any of the determinations, normalized names are extracted from the global table for the names. The global table includes a mapping of the raw names and associated normalized names.