Hybrid Text Normalization Pipeline for Ambiguous Pattern Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text normalization methods rely heavily on hand-crafted rules, which are time-consuming and resource-intensive, and suffer from ambiguous scenarios due to overlapping conditions, necessitating more efficient techniques.
Innovation Solution
A hybrid two-stage pipeline approach using a statistical model, such as BERT, for sequence labeling followed by a rule-based model for precise conversion, including segment merging, position switching, and metric/mark replacement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If hand-crafted rules are used for text normalization, then coverage of various scenarios can be achieved, but the number of rules increases to thousands and processing becomes time and resource consuming
Solution Approach 1:
The patent segments the text normalization process into multiple stages: a first stage using a statistical model for initial processing, and a second stage using rule-based models for refined processing. This segmentation allows the system to handle diverse scenarios efficiently without requiring all rules to process every input, thereby improving productivity while maintaining adaptability.
Solution Approach 2:
The patent introduces an intermediary statistical model that processes text before it reaches the rule-based models. This intermediary layer filters and prepares the input, reducing the complexity and number of rules needed in subsequent stages, thus resolving the contradiction between comprehensive scenario coverage and processing efficiency.
2Adaptability or versatility
If hand-crafted rules are used for text normalization, then various text patterns can be handled, but ambiguous scenarios with overlapping conditions require reasonable ranking of rules
Solution Approach 1:
The patent replaces the manual mechanical process of crafting and ranking rules with an automated statistical model in the first stage. This model learns optimal processing strategies from data, eliminating the need for manual rule ranking while maintaining the ability to handle diverse text patterns. The statistical model automatically resolves ambiguous scenarios without requiring complex rule prioritization logic.
3Manufacturing precision
If hand-crafted rules are used for text normalization, then specific conversion operations can be performed, but the process is time and resource consuming
Solution Approach 1:
The patent divides the normalization process into two stages: a fast statistical model for initial processing that handles time-sensitive operations, and a more precise rule-based model for final refinement. This segmentation ensures that most processing is completed quickly while maintaining conversion precision through the second stage, thereby reducing overall processing time without sacrificing accuracy.
Solution Approach 2:
The patent applies partial action by having the statistical model handle the majority of processing needs in the first stage, with only necessary cases being passed to the rule-based models. This approach achieves sufficient precision for most cases quickly, reserving detailed rule-based processing only when needed, thus reducing overall processing time while maintaining conversion precision.
Data Source
AI summary
Methods and devices to efficiently normalize text by processing inputted text based on a text normalization model that includes processing the input text in a first stage including a statistical model as a first output, processing the first output in a second stage including a rule based model as a normalized text, and outputting the normalized text.