Text Normalization via Statistical and Human-in-the-Loop Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems fail to effectively normalize text from various sources, leading to inconsistencies in language modeling, as they do not account for domain-specific vocabularies and evolving internet language, which hampers downstream analysis and machine learning performance.
Innovation Solution
A system and method combining statistical models, data models, and human-in-the-loop (HITL) normalization, where terms are extracted, searched, and prioritized based on context and probability, with human intervention to add new terms to a digitized data model, ensuring dynamic and consistent text representation across sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If text is gathered from various sources to create a language model, then the vocabulary coverage is improved, but inconsistencies in terminology and syntax across domains worsen
Solution Approach 1:
The patent introduces a normalization layer as an intermediary between text gathering and language model construction. This normalization process standardizes terminology and syntax from various sources before they are incorporated into the language model, thereby maintaining vocabulary diversity while ensuring consistency. The normalization module acts as a mediator that transforms heterogeneous text data into a unified format.
2Stability of the object's composition
If manual normalization is performed to ensure terminology consistency, then terminology consistency is improved, but the time and effort required worsen
Solution Approach 1:
The system implements self-service through automated normalization algorithms that operate without continuous human intervention. The language model construction process automatically handles terminology standardization by gathering text from multiple sources and applying consistent normalization rules, eliminating the need for manual review of each term while maintaining high consistency standards.
Solution Approach 2:
The patent applies preliminary normalization actions during the text gathering phase rather than performing manual normalization later. By pre-processing text data to standardize terminology before model construction, the system eliminates the need for subsequent manual intervention, thereby maintaining consistency while reducing time investment.
3Measurement precision
If domain-specific vocabularies are incorporated to improve accuracy, then the domain accuracy is improved, but the complexity of managing multiple vocabularies worsens
Solution Approach 1:
The patent creates a universal normalization framework that handles multiple domain-specific vocabularies through a single unified process. The language model construction system is designed to accommodate various domains (insurance, telecommunications, etc.) using the same normalization methodology, thereby maintaining domain accuracy while avoiding the complexity of separate management systems for each vocabulary.
Data Source
AI summary
According to principles described herein, unsupervised statistical models, semi-supervised data models, and HITL methods are combined to create a text normalization system that is both robust and trainable with a minimum of human intervention. This system can be applied to data from multiple sources to standardize text for insertion into knowledge bases, machine learning model training and evaluation corpora, and analysis tools and databases