Regular Expression Generation Using LCS for Data Preprocessing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data formatting and preprocessing methods for large-scale data sets in big data analytics are inefficient and impractical, leading to inaccurate results due to poor signal-to-noise ratios and the need for manual processes.
Innovation Solution
A regular expression generator system that uses a symbolic wildcard-matching language and a longest common subsequence (LCS) algorithm to automatically generate regular expressions based on user input, allowing for efficient and automated data formatting and pattern matching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual data formatting and preprocessing processes are used, then data can be cleaned and curated, but the process becomes inefficient and impractical for large-scale data sets
Solution Approach 1:
The system automatically generates regular expressions by analyzing input data patterns without requiring manual intervention. The regex generator self-services by detecting patterns, generating expressions, and refining them through feedback loops, eliminating the need for manual data formatting while maintaining high efficiency for large-scale datasets
Solution Approach 2:
The patent replaces manual mechanical processes of data formatting with an automated computational system. The regex generator uses pattern recognition algorithms and machine learning to substitute human operators, achieving scalable automation that maintains productivity while reducing manual intervention
2Measurement precision
If more input data is used to generate regular expressions, then pattern matching accuracy improves, but the generation time increases
Solution Approach 1:
The system performs preliminary analysis of data patterns before final regex generation. By pre-processing input data to identify common patterns and structures, the system prepares the foundation for accurate regex generation without requiring exhaustive analysis of all data, thus reducing generation time while maintaining accuracy
Solution Approach 2:
The regex generator uses a feedback mechanism that refines patterns iteratively. Instead of requiring complete analysis upfront, the system generates initial patterns and refines them through partial re-analysis of relevant data subsets, achieving high accuracy without processing the entire dataset multiple times
3Ease of operation
If manual data formatting processes are used, then data can be manipulated into a common format, but the process is inefficient for large-scale data sets
Solution Approach 1:
The regular expression generator serves multiple functions: pattern detection, regex generation, data validation, and format standardization. This universal system handles diverse data formats and structures through a single automated platform, maintaining ease of operation while dramatically increasing processing throughput for large-scale datasets
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed herein are techniques related to automated generation of regular expressions. In some embodiments, a regular expression generator may receive input data comprising one or more character sequences. The regular expression generator may convert character sequences into a sets of regular expression codes and/or span data structures. The regular expression generator may identify a longest common subsequence shared by the sets of regular expression codes and/or spans, and may generate a regular expression based upon the longest common subsequence.