Regular Expression Generation Using LCS for Accurate Data Formatting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data formatting and preprocessing methods for large-scale data sets in big data systems are inefficient and impractical, leading to inaccurate results due to poor signal-to-noise ratios and the inclusion of duplicate and erroneous data.
Innovation Solution
A regular expression generator system that uses a longest common subsequence (LCS) algorithm to generate regular expressions based on user-selected character sequences, converting them into intermediate regular expression codes (IRECs) and employing a fully-connected graph for efficient generation, handling both positive and negative examples, and optimizing execution order to avoid exponential runtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual data formatting and preprocessing processes are used, then data can be cleaned and curated into a common format, but the process becomes inefficient and impractical for large-scale data sets
Solution Approach 1:
The system automatically generates regular expressions by analyzing input data patterns without requiring manual intervention. The regex generator self-serves the data preprocessing task by converting sample data into executable regex patterns that can independently format and validate large datasets, eliminating the need for continuous manual preprocessing operations.
Solution Approach 2:
The patent introduces regular expressions as an intermediary between raw data and the data processing system. These regex patterns serve as automated mediators that enforce data formatting rules, validate patterns, and transform data into consistent formats, replacing manual preprocessing operations with automated pattern-matching mechanisms.
2Reliability
If manual data cleaning processes are used, then data quality can be improved, but the process is inefficient for large-scale data sets
Solution Approach 1:
The patent replaces manual mechanical data cleaning operations with automated computational processes. Regular expressions provide algorithmic pattern matching and validation that automatically detects and corrects data quality issues such as inconsistent formats, invalid patterns, and formatting errors, enabling high-throughput processing while maintaining reliability.
3Measurement precision
If regular expressions are generated without using longest common subsequence algorithm, then generation process is simpler, but accuracy decreases leading to false positives and negatives
Solution Approach 1:
The system performs preliminary analysis of input data patterns by computing the longest common subsequence before generating the final regular expression. This preliminary step identifies the core repeating patterns and structures in the data, which are then used as the foundation for constructing accurate regex patterns that minimize false positives and negatives.
4Measurement precision
If LCS algorithm is applied to three or more character sequences, then pattern matching accuracy improves, but runtime increases exponentially
Solution Approach 1:
The patent segments the computation of longest common subsequence into pairwise comparisons between character sequences rather than computing the LCS of all sequences simultaneously. By breaking down the complex multi-sequence problem into multiple simpler two-sequence problems, the system maintains pattern matching accuracy while avoiding exponential runtime increases.
Data Source
AI summary
Disclosed herein are techniques related to automated generation of regular expressions. In some embodiments, a regular expression generator may receive input data comprising one or more character sequences. The regular expression generator may convert character sequences into a sets of regular expression codes and/or span data structures. The regular expression generator may identify a longest common subsequence shared by the sets of regular expression codes and/or spans, and may generate a regular expression based upon the longest common subsequence.