Regular Expression Learning via Iterative Transformation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The manual effort required to design high-quality regular expressions for complex information extraction tasks is significant, with existing methods providing little assistance in reducing this effort.
Innovation Solution
A method for regular expression learning that starts with an initial regular expression, applies transformations such as character class restrictions, quantifier restrictions, and negative lookaheads, and selects candidate regular expressions based on improved F-measure, iteratively refining the expression until no further improvement is achieved.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual design of regular expressions is used, then extraction quality can be high, but human effort and time consumption are significant
Solution Approach 1:
The system performs preliminary actions by automatically generating candidate regular expressions and applying transformations (character class restrictions, quantifier restrictions, negative lookaheads) before human review. This preliminary automated work reduces the time and effort humans need to spend on manual design while maintaining high extraction quality through iterative refinement based on labeled positive and negative matches.
2Loss of time
If automated regular expression generation is used, then human effort is reduced, but extraction quality may deteriorate
Solution Approach 1:
The system implements feedback mechanisms by using labeled positive matches and negative matches to evaluate candidate regular expressions using F-measure. This feedback loop allows the automated system to iteratively refine and improve regular expressions, ensuring high extraction quality while reducing human effort. The feedback from labeled data guides the transformation process to produce increasingly accurate patterns.
3Measurement precision
If complex transformations are applied to refine regular expressions, then extraction precision improves, but system complexity increases
Solution Approach 1:
The system segments the complex task of regular expression refinement into distinct transformation components: character class restrictions, quantifier restrictions, and negative lookaheads. Each transformation type addresses specific aspects of pattern refinement independently. This segmentation makes the complex system more manageable and systematic, allowing precise control over how regular expressions are refined to improve extraction precision.
Data Source
AI summary
A method, device, and computer program product are provided for regular expression learning is provided. An initial regular expression may be received from a user. The initial regular expression is executed over a database. Positive matches and negative matches are labeled. The initial regular expression and the labeled positive and negative matches are input in a transformation process. The transformation process may iteratively execute character class restrictions, quantifier restrictions, negative lookaheads on the initial regular expression to transform the initial regular expression into the pool of candidate regular expressions. The transformation process may execute, one at a time, the character class restrictions, quantifier restrictions, the negative lookaheads. A candidate regular expression is selected from the pool of candidate regular expressions, where the selected candidate regular expression has a best F-Measure out of the pool of candidate regular expressions.