Automated Regular Expression Generation for Log Line Data Extraction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Manual generation of regular expressions for extracting data from large volumes of log lines is time-consuming and difficult for human operators, making it challenging to monitor and analyze web application performance effectively.

Innovation Solution

An automated system that generates regular expressions based on user input, allowing users to identify desired fields in log lines, with multiple instances of regular expression portions combined to achieve the desired extraction, reducing the need for manual generation and simplifying the process.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If manual generation of regular expressions is used, then extraction accuracy can be achieved, but time consumption and operational difficulty increase significantly

Engineering Contradiction:

Improveextraction accuracyVSAvoidtime consumption

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system enables self-service automated generation of regular expressions by allowing users to input sample log lines and desired extraction fields, then the system automatically generates and refines the regular expression patterns without requiring manual expertise in regex syntax

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system introduces an intermediary automated processing layer that translates user-friendly field specifications into complex regular expression patterns, acting as a mediator between simple user requirements and the sophisticated regex syntax needed for accurate extraction

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If manual generation of regular expressions is used, then customization to specific extraction needs is possible, but operational complexity and difficulty increase

Engineering Contradiction:

Improvecustomization capabilityVSAvoidoperational difficulty

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The system inverts the traditional approach by not requiring users to write regular expressions directly. Instead, users specify what they want to extract in simple terms, and the system automatically generates the appropriate regular expressions, flipping the complexity from user-side to system-side

Inventive Principle:

Principle #13The other way round (Inversion)

3Productivity

If automated generation is implemented, then time efficiency and ease of use improve, but system complexity increases

Engineering Contradiction:

Improvetime efficiencyVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The automated generation process is segmented into distinct modular components: log line parsing module, field identification module, pattern generation module, and validation module. Each module handles a specific aspect of the generation process, making the overall complex system manageable and maintainable

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS10775751B2Automatic generation of regular expression based on log line data

Publication Date: 2020.09.15 CISCO TECHNOLOGY INC

AI summary

In one aspect, a regular expression is automatically generated based on user input for fields that are desired to be extracted from log lines. The input may be received by user through an interface provided by a machine such as a controller. The input may identify one or more fields within a log line that should be extracted. Multiple instances of potential regular expression portions may be generated based on the user input, and different portions are combined together to determine if they achieve the desired extraction. Once a complete regular expression is generated based on user input, a user may provide additional input to identify examples or counterexamples of log line fields that satisfy or don't satisfy the user's intended extraction.