Regex Decomposition for Faster Large-Scale Log Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Evaluating regular expressions on large volumes of operational logs is time-consuming and resource-intensive, and existing optimization techniques lead to inefficiencies in substring conversion and matching.
Innovation Solution
A framework for regular expression decomposition and evaluation that identifies pattern and literal components, executes evaluation processes on a sample of data to determine performance characteristics, and selects an optimal process for further data analysis, reducing reliance on statistics or catalogs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If regular expression evaluation is performed on large volumes of operational logs using existing optimization techniques, then matching accuracy is maintained, but execution time and resource consumption increase significantly
Solution Approach 1:
The regular expression is decomposed into multiple evaluation processes by identifying pattern components (requiring regex engine) and literal components (amenable to string matching). This segmentation allows different parts of the regex to be evaluated using optimized techniques specific to their type, reducing overall execution time while maintaining matching accuracy.
Solution Approach 2:
The system performs preliminary evaluation of literal components using string matching algorithms before submitting pattern components to the regex engine. This preliminary action filters out non-matching cases early, reducing the workload on the more resource-intensive regex engine and decreasing total execution time.
2Measurement precision
If regular expression evaluation is performed on large volumes of operational logs using existing optimization techniques, then matching accuracy is maintained, but resource consumption increases significantly
Solution Approach 1:
By segmenting the regex evaluation into literal component evaluation (using efficient string matching) and pattern component evaluation (using regex engine), the system reduces overall resource consumption. The string matching operations consume fewer resources than full regex evaluation, allowing accurate processing of large log volumes with reduced energy and computational resource usage.
Solution Approach 2:
The system performs partial regex evaluation by handling literal components through string matching and only submitting pattern components to the regex engine. This partial action approach maintains matching accuracy for the pattern portions while avoiding the excessive resource consumption of applying the full regex engine to entire expressions.
3Productivity
If existing optimization techniques are used for regular expression evaluation, then some performance improvement is achieved, but inefficiencies remain in substring conversion and matching processes
Solution Approach 1:
The regex evaluation process is segmented into distinct evaluation processes based on component types. Literal components are evaluated separately using string matching, while pattern components are evaluated separately using the regex engine. This segmentation improves evaluation efficiency by applying optimized techniques to each component type while managing complexity through systematic organization of the evaluation processes.
Data Source
AI summary
Regular expression (“regex”) decomposition and evaluation is disclosed. In an aspect of the disclosure, a literal component and a pattern component in a regex are determined. A plurality of evaluation processes is determined based on the regex, including a first evaluation process configured to identify text that matches the literal component and the pattern component. The evaluation processes are executed with respect to a sample of data to determine performance characteristics including a performance characteristic respective to each evaluation process. An evaluation process of the evaluation processes is selected based on the determined performance characteristics. The selected evaluation process is executed with respect to further data to retrieve results that satisfy the selected evaluation process. In another aspect, the pattern component is decomposed into subcomponents. In another aspect, the evaluation processes include an evaluation process that specifies a process to provide the regex to a regex engine for execution thereof.