Real-Time Extraction Rule Refinement for Unstructured Data

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Large datasets, particularly machine-generated data, often present challenges in analysis due to their unstructured nature, making it difficult to extract relevant field values efficiently, especially in real-time, which can lead to improper or ineffective extraction rules and missed significant values.

Innovation Solution

A system and method for real-time display of event records and extracted values using a graphical user interface that allows users to automatically generate or manually edit extraction rules, including regular expressions, enabling real-time updates and statistics on unique extracted values, facilitating efficient data analysis and rule refinement.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If automated extraction rules are used on large unstructured datasets, then extraction speed and productivity improve, but accuracy and reliability deteriorate due to improper or ineffective rules

Engineering Contradiction:

Improveextraction speedVSAvoidextraction accuracy

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system implements feedback mechanisms by monitoring extraction results and using them to automatically refine and improve extraction rules. The system tracks extracted values, identifies patterns in rejected values, and adjusts rules accordingly to maintain high accuracy while processing large datasets in real-time

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system dynamically adjusts extraction rule parameters based on data characteristics. By changing parameters such as regex patterns, field delimiters, and value formats according to the specific dataset being processed, the system maintains both high extraction speed and accuracy across diverse unstructured data types

Inventive Principle:

Principle #35Parameter changes

2Reliability

If manual rule creation is used to ensure accuracy, then extraction reliability improves, but time consumption and productivity worsen

Engineering Contradiction:

Improveextraction accuracyVSAvoidrule creation time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by pre-processing data samples to identify common patterns, data formats, and field structures before actual extraction begins. This preliminary analysis enables the system to generate accurate extraction rules automatically, eliminating the need for time-consuming manual rule creation while maintaining high accuracy

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements self-service by automatically generating, refining, and optimizing extraction rules without human intervention. The system analyzes data characteristics, creates appropriate rules, validates them against sample data, and iteratively improves them, thereby achieving both high accuracy and fast rule deployment

Inventive Principle:

Principle #25Self-service

3Productivity

If real-time processing is implemented, then productivity and response time improve, but system complexity and computational resources worsen

Engineering Contradiction:

Improvereal-time processing capabilityVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system segments the data processing workflow into distinct modular components: data ingestion modules, extraction rule engines, validation modules, and result processing modules. Each segment handles specific tasks independently, enabling real-time processing while keeping individual component complexity manageable and allowing parallel processing

Inventive Principle:

Principle #1Segmentation

4Measurement precision

If comprehensive data analysis is performed on large datasets, then measurement precision and extraction accuracy improve, but processing time and productivity worsen

Engineering Contradiction:

Improvefield value extraction accuracyVSAvoiddata processing speed

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The system applies partial analysis by focusing extraction efforts on the most relevant and frequently occurring fields and patterns in the data. Rather than analyzing every possible data element in depth, the system identifies and extracts key information using optimized rules, achieving high precision for critical fields while maintaining overall processing speed

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11822372B1Automated extraction rule modification based on rejected field values

Publication Date: 2023.11.21 CISCO TECHNOLOGY INC

AI summary

Embodiments are directed towards real time display of event records and extracted values based on at least one extraction rule, such as a regular expression. A user interface may be employed to enable a user to have an extraction rule automatically generate and/or to manually enter an extraction rule. The user may be enabled to manually edit a previously provided extraction rule, which may result in real time display of updated extracted values. The extraction rule may be utilized to extract values from each of a plurality of records, including event records of unstructured machine data. Statistics may be determined for each unique extracted value, and may be displayed to the user in real time. The user interface may also enable the user to select at least one unique extracted value to display those event records that include an extracted value that matches the selected value.