Universal Link Parser for Log Data Extraction and Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current tools require specific connectors and software to parse log data from various formats, making them inefficient in detecting network breaches, especially when log formats change or new systems are deployed, and they struggle to handle the vast number of known or potential malicious IP addresses and domains.
Innovation Solution
A universal link-based system that uses a cascading filter or sieve approach to parse log data line-by-line, identifying candidate values through high-level patterns and progressively more specific classifications, with heuristics to distinguish between 'source' and 'destination' IP addresses, and discarding unclassified candidates, allowing for detection of network security breaches across different log formats and sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If format-specific connectors are used to parse log data, then parsing accuracy for known formats is improved, but adaptability to new or changed log formats deteriorates
Solution Approach 1:
The patent implements a universal link parser that can handle multiple log formats through a single unified structure. The parser uses a common link parsing framework that accommodates different log formats without requiring separate connectors for each format, thereby achieving both parsing accuracy and adaptability to new formats.
Solution Approach 2:
The patent employs configurable parameters within the universal link parser that can be adjusted to match different log formats. By changing parsing parameters rather than replacing the entire connector, the system maintains high parsing accuracy while adapting to new or modified log formats from different sources.
2Adaptability or versatility
If multiple format-specific connectors are maintained, then support for existing log formats is improved, but device complexity increases
Solution Approach 1:
The patent merges multiple format-specific parsing functions into a single universal link parser. Instead of maintaining separate connectors for different log formats, the system combines them into one unified parser that handles all formats through a common architecture, thereby reducing device complexity while preserving support for existing formats.
Solution Approach 2:
The universal link parser serves as a multi-functional component that replaces multiple specialized connectors. This single parser can process various log formats from different sources, eliminating the need for multiple separate connectors and reducing overall system complexity.
3Measurement precision
If regular expressions are provided for each log line type, then extraction precision is improved, but difficulty of maintaining updates increases
Solution Approach 1:
The patent uses configurable extraction parameters within the universal link parser that can be easily modified to adapt to changes in log formats. Instead of maintaining separate regular expressions for each format, the system uses parameter-based configuration that simplifies updates while preserving extraction precision.
Solution Approach 2:
The patent implements a dynamic parsing approach where the universal link parser can adapt its behavior based on the input log format. This dynamic capability allows the system to maintain high extraction precision while making it easier to update and adapt to changing log formats without rewriting entire parsing logic.
4Reliability
If comprehensive threat detection is implemented, then security coverage is improved, but computational resources required increase
Solution Approach 1:
The patent extracts and classifies only the relevant information from log data using the universal link parser, such as IP addresses, URLs, and other security-relevant fields. By extracting only necessary data and discarding unnecessary information, the system achieves comprehensive security coverage while reducing computational resource consumption.
Solution Approach 2:
The patent applies different processing levels to different parts of the log data based on their security relevance. The universal link parser focuses computational resources on extracting and analyzing security-critical fields while using minimal resources on less important data, thereby achieving comprehensive security coverage efficiently.
Data Source
AI summary
A universal link to extract and classify log data is disclosed. In various embodiments, a set of candidate data values that match a top level pattern that is common to two or more types of data value of interest is identified. The candidate data values are processed through a plurality of successive filtering stages, each stage of which includes determining which, if any, of said candidates match a more specific pattern associated more specifically with a specific data value type. Candidates, if any, which match the more specific pattern are classified as being of a corresponding specific data type and are removed from the set of candidate data values. A structured data record that associates each candidate data value determined to be of a corresponding one of said types of data value of interest with said corresponding one of said types of data value of interest is generated and stored.