Regular Expression Generation via Longest Common Subsequence Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data preprocessing methods for big data analytics are inefficient, especially for large-scale datasets, as they often rely on manual and ad-hoc processes, leading to inaccurate results due to poor signal-to-noise ratios and the handling of diverse data formats, including structured, semi-structured, and unstructured data.
Innovation Solution
A system for generating regular expressions using a longest common subsequence (LCS) algorithm that converts input data into regular expression codes, allowing for the automatic creation of patterns to match and extract data, with features like user interfaces for selecting character subsets and handling both positive and negative examples to minimize false positives and negatives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If manual and ad-hoc processes are used for data preprocessing, then flexibility in handling diverse data formats is maintained, but efficiency and productivity deteriorate significantly for large-scale datasets
Solution Approach 1:
The system automatically generates regular expressions by analyzing sample data and identifying patterns, eliminating the need for manual regex creation. The algorithm self-adapts to different data formats by learning from provided examples, thereby maintaining versatility while dramatically improving productivity through automation
Solution Approach 2:
Manual text-based regex creation is replaced with an automated algorithmic system that uses machine learning techniques to generate regular expressions. This substitution transforms the mechanical process of manual pattern matching into an automated computational process, resolving the contradiction between manual flexibility and automated efficiency
2Measurement precision
If manual data cleaning processes are used, then accuracy can be maintained through human judgment, but time consumption and loss of time increase significantly
Solution Approach 1:
The system performs preliminary analysis on sample data to learn patterns and generate regular expressions before actual data processing begins. This pre-learning phase enables the system to quickly and accurately process large datasets without sacrificing precision, as the pattern recognition logic is already established
Solution Approach 2:
The system creates a model of data patterns by analyzing sample examples, then applies this copied knowledge to process entire datasets. This copying approach allows manual-level accuracy to be achieved through automated pattern recognition, eliminating time-consuming manual review while maintaining precision
3Productivity
If automated regular expression generation is implemented, then productivity and speed improve, but system complexity increases
Solution Approach 1:
The system introduces an intermediary learning phase that bridges raw data and final regular expressions. This intermediary process automatically analyzes patterns and generates optimized regex expressions, managing system complexity by breaking down the transformation into manageable stages while maintaining high productivity
Solution Approach 2:
The system dynamically adjusts parameters such as pattern sensitivity, matching strictness, and optimization criteria based on the characteristics of the input data. This parameter adaptation allows the system to handle diverse data formats efficiently without requiring complex manual configuration, resolving the contradiction between automation and complexity
4Measurement precision
If comprehensive pattern matching is performed to ensure accuracy, then measurement precision improves, but computational resources and energy consumption increase
Solution Approach 1:
The system performs partial pattern matching by focusing on the most discriminative features of data patterns rather than analyzing every possible variation. This selective approach maintains high matching accuracy by concentrating computational resources on the most informative patterns, thereby reducing overall energy consumption while preserving precision
Data Source
AI summary
Techniques for generated regular expressions are disclosed. In some embodiments, a regular expression generator may receive input data comprising one or more character sequences. The regular expression generator may convert character sequences into a sets of regular expression codes and/or span data structures. The regular expression generator may identify a longest common subsequence shared by the sets of regular expression codes and/or spans, and may generate a regular expression based upon the longest common subsequence. Alignment of span data structures may be performed when generating the regular expression.