Log Event Normalization via Three-Staged Regular Expressions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The management and analysis of computer-generated log messages are cumbersome due to their freeform nature and varying formats, requiring significant manual effort and resources to extract meaningful information, especially in compliance with regulations like PCI DSS, SOX, and HIPAA, which demand secure archiving for extended periods, resulting in high storage and operational costs.
Innovation Solution
A system and method for normalizing log messages by determining the originating program, using a signature and parsing expression to parse and extract information, allowing for real-time processing and reduction in storage volume through automated filtering, prioritization, compression, encryption, and digital signing, enabling efficient archiving and compliance processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If log messages are archived for extended periods to meet regulatory requirements, then compliance is achieved, but storage costs and operational complexity increase significantly
Solution Approach 1:
The patent extracts only the essential and relevant log data elements that are necessary for compliance, rather than archiving complete log messages. By identifying and retaining only critical fields (such as timestamps, event types, and key identifiers), the system reduces storage requirements while maintaining regulatory compliance capability.
Solution Approach 2:
Instead of archiving all log data and then searching through it, the patent inverts the approach by first identifying what compliance requirements need to be met, then selectively capturing only the necessary data elements from the beginning. This inversion of the traditional archiving approach minimizes storage while ensuring compliance.
2Measurement precision
If manual review of log messages is performed to extract meaningful information, then accurate analysis is achieved, but labor time and effort increase massively
Solution Approach 1:
The patent implements automated systems that perform log analysis without requiring manual intervention. The system self-services by automatically parsing, normalizing, and analyzing log messages using predefined rules and patterns, thereby eliminating the need for manual review while maintaining accurate information extraction.
Solution Approach 2:
The patent replaces the mechanical process of manual log review with automated computational systems. By using software-based parsing, pattern matching, and normalization techniques, the system substitutes human manual analysis with automated processing, achieving both accuracy and efficiency.
3Loss of information
If complete log messages are retained for analysis, then full information availability is maintained, but processing speed decreases due to data volume
Solution Approach 1:
The patent segments log messages into distinct components and fields, identifying and retaining only the critical segments that are necessary for compliance and analysis. By dividing the log data structure and selectively processing only essential portions, the system maintains information availability for compliance purposes while improving processing speed by reducing the total data volume that must be handled.
Solution Approach 2:
The patent applies different processing qualities to different parts of log data. Critical fields that are necessary for compliance receive full processing and retention, while non-critical fields are summarized, normalized, or discarded. This local differentiation of data quality requirements maintains necessary information availability while optimizing processing efficiency.
Data Source
AI summary
Methods and systems for normalizing log messages. Some methods include obtaining a freeform log message from one of many disparate programs. The methods can include determining which program originated the message and, based on that, determining a signature which matches the message. Using the signature, a parsing expression may be determined with which to extract information from a portion of the message. The time from obtaining the message to extracting the information can be about the same for all messages and can be about 1/40,000th of a second. In some embodiments, a generic signature of the message may be output. A version of the message may be reconstructed based on the generic signature and information. When more than one message signatures matches the reconstructed message, one of the matching signatures can be adjusted. The parsing expression can be the first of an ordered list of expressions which successfully evaluates the log message.