Automated Regular Expression Generation for Data Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analytics techniques struggle with extracting relevant information from large, complex data sets in modern ICT systems, particularly due to the lack of uniform structure and the time-consuming, error-prone process of creating regular expressions for data extraction from semi-structured and unstructured data sources.
Innovation Solution
A computer-implemented method that identifies common substrings across data entry strings to create regular expressions for extracting data, combining token regular expressions to identify matching substrings and common substring analysis to indirectly identify variable substrings, thereby conveying semantic context and facilitating data extraction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual creation of regular expressions is used to extract data from semi-structured data sets, then extraction accuracy can be achieved, but the process is time-consuming and error-prone
Solution Approach 1:
The system performs self-service by automatically analyzing data sets and generating regular expressions without human intervention. The pattern recognition module scans data entries, identifies common substrings, and creates regular expressions autonomously, eliminating the manual creation process while maintaining extraction accuracy
Solution Approach 2:
The manual mechanical process of creating regular expressions is replaced with an automated computational system. The pattern recognition module uses algorithmic analysis to substitute human operators, rapidly processing data sets and generating regular expressions through systematic pattern matching rather than manual inspection
2Quantity of substance
If data volume increases from multiple sources with varying structures, then more comprehensive data analysis is enabled, but the lack of uniform structure makes analysis increasingly difficult
Solution Approach 1:
The regular expression generation system performs multiple functions: it analyzes diverse data formats, identifies patterns across different structures, and generates universal extraction rules that work across varied data sources. This multi-functionality allows the system to handle heterogeneous data volumes without proportionally increasing complexity
Solution Approach 2:
The system changes parameters by dynamically adjusting pattern recognition criteria based on the specific data structure being analyzed. Rather than requiring uniform structure, the system adapts its analysis parameters to each data source's characteristics, enabling comprehensive data volume processing while managing structural diversity through flexible parameter adjustment
3Productivity
If traditional processing techniques are used on large data sets, then system resource requirements are lower, but the analysis capabilities are insufficient for complex data sets
Solution Approach 1:
The system performs preliminary action by pre-processing data sets to identify patterns and generate regular expressions before full analysis. This preparatory step organizes the data structure in advance, enabling more efficient subsequent processing and allowing traditional techniques to work effectively on pre-organized data, thereby increasing analysis capability without proportionally increasing system complexity
Data Source
AI summary
A computer implemented method is provided for creating a regular expression for extracting data from a data set. The method comprises the steps of: identifying a first common substring, the first common substring being present in at least two data entry strings of a first data set, by comparing a first and second data entry string of the first data set; and creating a first regular expression for extracting data from a second data set, the first regular expression including a first part for matching the first common substring.