Text Mining Multiword Term Normalization for Data Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Text mining in the airline industry faces challenges in consistently identifying part names within free-form text data records due to variations in spelling, abbreviations, acronyms, and inconsistent terminology, which complicates data analytics and can lead to inaccurate or incomplete data.
Innovation Solution
A method combining domain knowledge, linguistic knowledge, and machine learning to extract multiword terms of interest from electronic documents, using a machine learning algorithm that iteratively identifies and normalizes terms by searching for seed nouns, building a training set, and refining terms based on linguistic features to produce a collection of normalized terms for efficient data analytics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional text mining methods are used to identify part names in free-form text, then the process is simple, but the identification accuracy deteriorates due to spelling variations, abbreviations, and inconsistent terminology
Solution Approach 1:
The patent segments the text mining process into multiple specialized modules: a spelling variation handler that manages different spellings of the same term, an abbreviation expansion module that converts abbreviations to full forms, and a normalization component that standardizes terminology. This segmentation allows each module to specialize in handling specific types of text variations, thereby improving identification accuracy while keeping the overall system manageable through modular architecture
Solution Approach 2:
The patent introduces intermediary components that act as mediators between the raw free-form text and the final part name identification. These intermediaries include spelling variation handlers that serve as buffers between different spelling forms, abbreviation expansion modules that mediate between abbreviated and full forms, and normalization layers that mediate between inconsistent terminology and standardized terms. These intermediaries transform the input data into a consistent format that improves downstream processing accuracy
2Ease of operation
If free-form text is used in data records to document events and activities, then the ease of documentation is improved, but the consistency of information identification deteriorates
Solution Approach 1:
The patent applies parameter changes by transforming the textual representation of part names through multiple processing stages. The system changes spelling parameters by handling variations, changes abbreviation parameters by expanding to full forms, and changes terminology parameters by normalizing to standard forms. These parameter transformations maintain the ease of free-form documentation while stabilizing the information composition for consistent identification and analysis
3Productivity
If data records are populated with free-form text including abbreviations and misspellings, then the speed of data entry is improved, but the reliability of data analytics deteriorates
Solution Approach 1:
The patent applies preliminary action by performing text normalization, spelling standardization, and abbreviation expansion on data records at the time of entry or storage, rather than waiting for analytics processing. This preliminary processing of spelling variations, abbreviation expansion, and terminology normalization ensures that the data is already in a consistent, reliable format when analytics are performed, thereby maintaining both fast data entry and high analytics reliability
Data Source
AI summary
A method is provided for analyzing and interpreting a dataset composed of electronic documents including free-form text. The method includes text mining the documents for terms of interest, including receiving a set of seed nouns as input to an iterative process an iteration of which includes searching for multiword terms having seed nouns as their head words, at least some of which define a training set of a machine learning algorithm used to identify additional multiword terms at least some of which have nouns outside the set of seed nouns as their head words. The iteration also includes adding the nouns outside the set of seed nouns to the set and thereby identifying a new set of seed nouns for a next iteration. The method includes unifying terms of interest to produce normalized terms of interest for application to generate features of the documents for data analytics performed thereon.