Mixed-Format Log Extraction via Pre-Stored Format Information
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log analysis systems fail to effectively extract specified log data or fields from mixed-format log files, as they rely on format determination based on log file names, which is not feasible when logs from different applications, OSes, and production-apparatus controllers with varying formats are merged into a single file.
Innovation Solution
An information processing system and method that include a format-information recording portion for storing format determination conditions and an extraction-condition recording portion for specifying extraction criteria, allowing the extraction of specific log data or fields from mixed-format log files by using the stored format information and extraction conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If log format is determined based on log file name, then log type identification is simple and fast, but it cannot handle mixed-format log files from different applications and sources
Solution Approach 1:
The patent segments the log analysis process into two distinct phases: format determination phase (using file name) and data extraction phase (using format information). This allows the system to use simple file-name-based format determination for efficiency while maintaining the ability to handle mixed formats through pre-stored format information for different log types.
Solution Approach 2:
The patent performs preliminary action by pre-storing format information (including field definitions and extraction rules) for multiple log formats in advance. When processing mixed-format logs, the system retrieves the appropriate pre-stored format information based on the determined log type, enabling efficient handling of various formats without real-time format parsing.
2Ease of operation
If all log data from different sources is merged into a single log file, then centralized log management is achieved, but format diversity makes extraction difficult
Solution Approach 1:
The patent creates a universal extraction framework that can handle multiple log formats through a single system. The extraction processing portion uses pre-stored format information to adaptively process different log types (OS logs, application logs, production apparatus logs) according to their respective formats, achieving multi-functionality in a unified system.
Solution Approach 2:
The patent introduces format information as an intermediary layer between the raw log data and the extraction process. This intermediary contains pre-defined field structures and extraction rules for each log type, mediating the complexity by translating diverse log formats into a unified extraction framework without requiring complex real-time format detection.
3Measurement precision
If manual log analysis is performed to extract specific fields, then accurate extraction is achieved, but labor and time consumption increase
Solution Approach 1:
The patent implements self-service by enabling the extraction system to automatically determine log format and select appropriate extraction rules based on pre-stored format information. The system serves itself by autonomously identifying log types and applying correct field extraction logic without requiring manual intervention or expert knowledge, thereby maintaining accuracy while reducing time consumption.
Solution Approach 2:
The patent changes the parameter of extraction accuracy from being dependent on manual operator skill to being determined by pre-configured format information. By transforming the extraction process into a parameter-driven automated operation using stored format definitions, the system maintains high accuracy while eliminating the time cost of manual analysis.
Data Source
AI summary
An information processing system that extracts a specified piece of log data or a specified field that constitutes the specified piece of log data, from a log file in which pieces of log data having different recording formats are mixed, the information processing system includes a format-information recording portion configured to store format information in which a determination condition to identify a recording format of the specified piece of log data is contained, an extraction-condition recording portion configured to store an extraction condition to extract the specified piece of log data or the specified field, and an extraction processing portion configured to extract the specified piece of log data or the specified field, from the log file by using the format information stored in the format-information recording portion and the extraction condition.