Regular Expression Generation Using Positive-Negative Pattern Examples
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data formatting and preprocessing methods for large-scale data sets in big data systems are inefficient and impractical, leading to inaccurate results due to poor signal-to-noise ratios and the inclusion of duplicate and erroneous data.
Innovation Solution
A regular expression generator system that uses a symbolic wildcard-matching language to generate regular expressions based on longest common subsequences of character sequences, incorporating positive and negative examples, and employs a depth-first traversal of a fully-connected graph to optimize the LCS algorithm for efficient generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual data formatting and preprocessing processes are used, then data can be cleaned and curated into a common format, but the process becomes inefficient and impractical for large-scale data sets
Solution Approach 1:
The system automatically generates regular expressions by analyzing input data patterns itself, without requiring manual intervention. The regex generator self-adapts to the data format by computing longest common subsequences and frequency distributions, enabling autonomous data preprocessing at scale
Solution Approach 2:
The patent replaces manual mechanical data formatting processes with an automated computational system that uses algorithms (LCS computation, frequency analysis) to generate regular expressions, substituting human effort with machine-based pattern recognition and automation
2Reliability
If traditional regular expression generation methods are used, then pattern matching can be performed, but false positive and false negative matches occur in yet-to-be-seen data
Solution Approach 1:
The system performs preliminary analysis of training data to compute longest common subsequences and character frequency distributions before generating the final regular expression. This pre-computation of pattern statistics enables the regex to be tuned to actual data characteristics, reducing false positives and negatives in unseen data
Solution Approach 2:
The system uses frequency distribution analysis and LCS computation to feedback-adjust the regular expression pattern based on observed data characteristics. The generated regex incorporates frequency-based weighting and pattern refinement that adapts to the specific data domain, improving generalization to new data
3Manufacturing precision
If LCS algorithm is performed on three or more character sequences simultaneously, then comprehensive pattern matching is achieved, but runtime increases exponentially
Solution Approach 1:
The patent segments the LCS computation into pairwise comparisons between character sequences rather than computing a single N-way LCS. By dividing the problem into multiple two-sequence LCS computations, the system achieves comparable pattern identification with significantly reduced computational complexity
Solution Approach 2:
The system merges the results of multiple pairwise LCS computations to construct the final regular expression. By combining insights from individual sequence pairs through frequency analysis and pattern aggregation, the system achieves comprehensive pattern coverage without the exponential runtime of direct N-way LCS computation
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed herein are techniques related to automated generation of regular expressions. In some embodiments, a regular expression generator may receive input data comprising one or more character sequences. The regular expression generator may convert character sequences into a sets of regular expression codes and/or span data structures. The regular expression generator may identify a longest common subsequence shared by the sets of regular expression codes and/or spans, and may generate a regular expression based upon the longest common subsequence.