Automatic Parser Selection for Network Log Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing data from computer networks for security and compliance purposes is difficult and expensive due to the challenge of ensuring appropriate parsers are deployed for varying and new log formats from diverse devices.
Innovation Solution
A data collection and analysis platform that automates the selection and generation of parsers by evaluating raw data against a library of rules, determining confidence measures, and generating parser rules when none exist, while also enabling obfuscation of sensitive data to protect privacy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual parser deployment is used for each log format, then parsing accuracy is improved, but device complexity and time consumption increase significantly
Solution Approach 1:
The system performs self-service by automatically detecting log formats and generating corresponding parsers without manual intervention. The parser generation module autonomously analyzes incoming log data, identifies format patterns, and creates appropriate parsing rules, eliminating the need for manual parser deployment while maintaining high parsing accuracy
Solution Approach 2:
The system changes parameters by dynamically adjusting parser configurations based on detected log formats. Instead of using fixed manual parsers, the system modifies parsing parameters automatically according to the identified log format characteristics, enabling adaptability across different device types and log formats
2Adaptability or versatility
If comprehensive parser library is maintained for all devices, then parsing coverage is improved, but maintenance cost and time increase
Solution Approach 1:
The system performs preliminary action by pre-processing and cataloging log format patterns in the parser library. Common log formats are pre-analyzed and stored as template patterns, enabling rapid parser generation when similar formats are encountered, thus reducing maintenance time while maintaining broad parsing coverage
Solution Approach 2:
The parser library is designed with universal patterns that can handle multiple device types and log formats. By creating multi-functional parser templates that work across different devices, the system achieves broad parsing coverage without needing separate maintenance for each individual device format
3Measurement precision
If manual parser configuration is performed, then data accuracy is improved, but productivity decreases
Solution Approach 1:
The system replaces the mechanical manual configuration process with an automated computational system. The parser generation module uses algorithmic analysis to detect log formats and generate parsers automatically, substituting human manual work with automated processing that maintains data accuracy while significantly increasing productivity
4Ease of operation
If sensitive data is stored in plain text, then data accessibility is improved, but security decreases
Solution Approach 1:
The system applies local quality by selectively obfuscating only the sensitive portions of data while leaving non-sensitive data in plain text. This targeted approach maintains accessibility for general data operations while providing security protection specifically for sensitive fields, balancing both requirements effectively
Data Source
AI summary
The automatic selection and usage of a parser is disclosed. Raw data is obtained from a first remote device. At least a portion of the raw data is evaluated using a plurality of rules. A confidence measure is determined for at least some of the rules. An indication that the raw data pertains to a source is provided as output when the confidence measure exceeds a threshold.