Log Entry Merging via Regular Expression Mapping

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing network monitoring systems face challenges in automatically collecting high-level event information from log entries that are generated under problematic conditions, such as incomplete sequences and interspersed log entries from multiple devices.

Innovation Solution

A system comprising a parser, a grouping tracker module, and a mapping module that separates and merges log entries into tokens, analyzes them to determine merged events using configurable merge properties, and maps them to a data structure using regular expressions to identify and group related events, even if they are interspersed or incomplete.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If multiple devices send log entries to central collection points, then event information can be collected from various sources, but log entries arrive interspersed and out of sequence making automatic collection difficult

Engineering Contradiction:

Improveability to collect events from multiple devicesVSAvoidcomplexity of processing interspersed log entries

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system segments the log processing task into distinct modules: a parser module that separates arriving log entries into tokens, a grouping tracker module that analyzes tokens to determine merged event membership, and a mapping module that maps log entries to merged event fields. This segmentation allows each module to handle specific aspects of the interspersed log entry problem independently, reducing overall processing complexity while maintaining multi-device collection capability

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary data structure called a merged event that serves as a buffer between the incoming interspersed log entries and the final event collection. The grouping tracker module acts as a mediator that receives out-of-sequence log entries, uses merge properties and regular expressions to identify related entries, and reconstructs complete events even when log entries arrive interspersed or incomplete

Inventive Principle:

Principle #24Intermediary (Mediator)

2Speed

If log entries are sent as they occur, then real-time monitoring is achieved, but incomplete sequences and crashes result in lost event information

Engineering Contradiction:

Improvespeed of log entry transmissionVSAvoidcompleteness of event data

Core Design Contradiction:

SpeedVSReliability

Solution Approach 1:

The system performs preliminary actions by establishing merge properties and regular expression patterns before log entries arrive. The grouping tracker module is pre-configured with criteria for identifying related log entries and reconstructing complete events. This preliminary setup enables the system to handle incomplete sequences and crashes by applying pre-defined rules to reconstruct events from partial data, maintaining reliability without sacrificing real-time transmission speed

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The grouping tracker module implements feedback mechanisms by continuously monitoring incoming log entries against merge properties and adjusting event reconstruction accordingly. When log entries are received out of sequence or incomplete due to crashes, the feedback loop uses regular expressions to identify patterns and determine which entries belong to which merged events, ensuring complete event data collection despite transmission issues

Inventive Principle:

Principle #23Feedback

3Adaptability or versatility

If configurable merge properties are used to group log entries, then flexibility in event definition is improved, but system configuration complexity increases

Engineering Contradiction:

Improveflexibility in defining merged eventsVSAvoidconfiguration complexity of merge properties

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent employs parameter changes by allowing merge properties to be configured with various parameters such as regular expression patterns, time windows, and device identifiers. These parameters can be adjusted to define different types of merged events without changing the underlying system architecture. The mapping module uses these configurable parameters to dynamically adapt to different event definitions, providing flexibility while maintaining manageable configuration complexity through a standardized parameter interface

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS7437359B2Merging multiple log entries in accordance with merge properties and mapping properties

Publication Date: 2008.10.14 MICRO FOCUS LLC

AI summary

A system and method for building merged events from log entries received from multiple devices. Multiple log events generally contribute to a single merged event. In the described embodiment, the mapping module receives log entries associated with specific merged events and maps them to fields in the merged event data structure in accordance with mapping properties. The described embodiments of the invention use regular expressions in the merge properties to describe values that are searched for in the received log entries. A described embodiment of the present invention gives the mapping module access to the event under construction. A new conditional operator, _oneOf, is introduced that selects the first token that is bound to a value out of a list of tokens.