Log Data Parsing via Mapping File Regular Expressions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently storing and parsing diverse types of log data, particularly in identifying the appropriate parser for log files with varying structures and formats.
Innovation Solution
A method and device for storing and parsing log data involve collecting log data based on file paths, storing metadata including file paths and log data in a database, classifying log data to acquire type information of a parser, and parsing the log data through the appropriate parser. This process utilizes a mapping file with regular expression information to determine the correct parser for each type of log data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a parser is created for each type of log data format, then parsing accuracy is improved, but device complexity increases
Solution Approach 1:
The patent implements a universal parser that can handle multiple types of log data formats by using a mapping file that associates regular expressions with parser types. Instead of creating separate parsers for each log format, the system uses a single parser that selects the appropriate parsing logic based on the mapping file, thereby reducing device complexity while maintaining parsing accuracy across diverse log formats
Solution Approach 2:
The mapping file serves as an intermediary between the log data and the parser. It contains regular expression patterns that match log file paths and associate them with appropriate parser types. This intermediary layer enables the system to automatically identify and route different log formats to the correct parsing logic without requiring multiple specialized parsers, thus resolving the contradiction between parsing accuracy and device complexity
2Measurement precision
If manual classification of log data is performed, then parsing accuracy is improved, but loss of time increases
Solution Approach 1:
The system performs self-service classification by automatically matching log file paths against regular expressions in the mapping file. The parser autonomously determines the appropriate parsing logic based on the mapping file without requiring manual intervention or classification. This automated self-service approach maintains high parsing accuracy while significantly reducing the time consumption associated with manual classification
Solution Approach 2:
The mapping file is prepared in advance with regular expression patterns and corresponding parser type associations. This preliminary action allows the system to quickly match log files against predefined patterns during actual processing, eliminating the need for time-consuming manual classification during operation. The mapping file acts as a pre-computed reference that enables rapid automated decision-making
3Adaptability or versatility
If extensive parser generation is performed, then parsing capability is improved, but use of energy increases
Solution Approach 1:
The patent employs a universal parsing approach where a single parser handles multiple log formats by referencing a mapping file. This eliminates the need to generate and maintain multiple specialized parsers, thereby reducing the computational energy required for parser generation and execution while preserving the ability to handle diverse log data formats
Solution Approach 2:
The parsing logic is extracted from the parser itself and placed into a separate mapping file that contains regular expressions and parser type associations. This extraction allows the parser to remain lightweight and low-energy, while the mapping file handles the complex matching logic. The separated mapping file can be loaded into memory once and reused, reducing the continuous energy consumption associated with maintaining extensive parser generation capabilities
Data Source
AI summary
Provided are a terminal and method for storing and parsing log data. The method includes collecting log data on the basis of a file path of the log data, storing metadata including the file path and log data paired with the metadata in a database (DB), classifying the log data on the basis of the metadata, acquiring type information of a parser related to the log data, and parsing the log data through the parser having the type information.