Two-Stage Text Normalization for Rule Ambiguity Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text normalization methods rely on hand-crafted rules, which are resource-intensive and prone to ambiguity due to overlapping conditions, necessitating more efficient techniques.
Innovation Solution
A hybrid two-stage pipeline approach using a statistical model, such as BERT, for sequence labeling followed by a rule-based model for precise conversion, including segment merging, position switching, and metric/mark replacement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If hand-crafted rules are used for text normalization, then coverage of natural language scenarios can be achieved, but time and resource consumption increases significantly
Solution Approach 1:
The patent segments the text normalization process into two distinct stages: a statistical model stage that handles general patterns and a rule-based model stage that handles specific cases. This segmentation allows each stage to specialize, reducing the overall complexity and resource requirements while maintaining comprehensive coverage.
Solution Approach 2:
The patent introduces an intermediary statistical model layer between the input text and the final rule-based processing. This intermediary pre-processes the text, identifies patterns, and prepares data structures that make subsequent rule-based processing more efficient and less resource-intensive.
2Ease of manufacture
If hand-crafted rules are used for text normalization, then specific text patterns can be processed, but ambiguity arises due to overlapping conditions
Solution Approach 1:
The patent segments the rule set into two distinct groups: rules applied to the original text and rules applied to the statistical model output. This segmentation eliminates overlapping conditions by ensuring that each rule set operates on different data representations with clearly defined scopes.
Solution Approach 2:
The statistical model acts as an intermediary that transforms ambiguous input text into structured representations with clear semantic meanings. This intermediary layer disambiguates overlapping conditions by providing explicit labels and structures that make subsequent rule application unambiguous.
3Ease of operation
If hand-crafted rules are used for text normalization, then text conversion can be performed, but rule ranking complexity increases
Solution Approach 1:
The patent segments the rule ranking problem into two independent ranking tasks: ranking rules for the original text stage and ranking rules for the statistical model output stage. This segmentation reduces complexity by limiting the scope of each ranking task and eliminating the need to rank all rules against each other.
Solution Approach 2:
The statistical model output serves as an intermediary representation that simplifies the rule ranking problem. By transforming text into structured formats with explicit semantic labels, the intermediary reduces the complexity of determining rule priority and applicability.
Data Source
AI summary
Methods and devices to efficiently normalize text by processing inputted text based on a text normalization model that includes processing the input text in a first stage including a statistical model as a first output, processing the first output in a second stage including a rule based model as a normalized text, and outputting the normalized text.