Cluster Computing Regex Generation via Scoring and Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating regular expressions in machine learning and information extraction, such as mutual bootstrapping, face challenges with handling diverse data formats, producing duplicate patterns, and requiring manual effort, leading to inefficiencies and inaccuracies, especially when dealing with operational logs.
Innovation Solution
A method that uses parallel processing and relation caching in a cluster computing system to automatically generate relevant regular expressions by iteratively adding seed patterns and words, pruning irrelevant results, and optimizing regex computational time, ensuring convergence and relevance without human intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If prior-art mutual bootstrapping method is used to generate extraction patterns, then automation is improved, but duplicate patterns and inaccuracies are produced
Solution Approach 1:
The patent implements a feedback mechanism where generated patterns are scored based on their extraction results, and low-scoring patterns are pruned. The system continuously evaluates pattern performance and uses this feedback to refine the pattern set, eliminating duplicates and inaccuracies while maintaining automation.
Solution Approach 2:
The patent introduces pattern scoring as a new parameter to evaluate pattern quality. By changing from simple pattern generation to scored pattern selection, the system can identify and remove duplicate or inaccurate patterns, improving precision while preserving automation benefits.
2Adaptability or versatility
If prior-art bootstrapping method processes large amounts of diverse data formats, then coverage is improved, but computational time increases significantly
Solution Approach 1:
The patent segments the pattern generation process into distinct phases: initial pattern generation, scoring evaluation, and pruning. This segmentation allows the system to efficiently handle diverse log formats by processing patterns through multiple stages rather than attempting to process all data simultaneously.
Solution Approach 2:
The patent performs preliminary pattern generation and scoring before final pattern selection. By pre-evaluating patterns on diverse data formats and scoring them in advance, the system reduces computational time during the actual extraction phase while maintaining broad format coverage.
3Ease of operation
If lazy quantifier is used in regex generalized from prefix, then pattern matching flexibility is improved, but backtracking increases computational overhead
Solution Approach 1:
The patent uses inexpensive, temporary placeholder patterns during the generation phase that are quickly evaluated and discarded. These disposable patterns allow flexible matching during exploration but are replaced with optimized final patterns that minimize backtracking, balancing flexibility with computational efficiency.
4Manufacturing precision
If manual task of writing regex codes is performed, then precision is improved, but productivity decreases
Solution Approach 1:
The patent implements a self-service system where the pattern generation process automatically evaluates and refines its own output through scoring and pruning. The system serves itself by identifying high-quality patterns without manual intervention, achieving both precision and productivity through automated self-optimization.
Data Source
AI summary
A process for generating regular expressions (regexes) as extraction patterns in a cluster computing system includes: receiving log events, a set of seed words, and a set of seed patterns; determining whether the set of seed words is full; if not, generating a new patterns by iteratively adding new patterns whose pattern scores S1 surpass a first preset score into the set of seed patterns; selecting a subset of seed patterns from the set of seed patterns; determining whether the subset of seed patterns is empty if not, generating a subset of seed words whose word scores S2 surpass a second preset score and iteratively adding the subset of seed words into the set of seed words; and repeating the above steps until the set of seed patterns is full and the set of seed words is empty; and finally pruning the set of seed patterns.