Log Auditing System Using Kafka and Elastic Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of logs from various applications in computer information security poses a challenge for real-time analysis, as existing technologies require opening multiple log files in different formats, making it inefficient to trace issues and mine correlations between them.
Innovation Solution
A method involving transmitting log files to a Kafka message queue, storing them in two databases, and using an Elastic Search engine to extract and count fields, with a distributed processing engine determining abnormal log information, facilitating efficient log auditing across different formats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If log files from various applications are collected and analyzed, then comprehensive audit capability is improved, but system complexity increases due to handling multiple log formats and sources
Solution Approach 1:
The patent introduces Kafka as an intermediary message queue system between log collection and analysis components. This mediator standardizes the handling of diverse log formats by providing a unified message queue interface, thereby improving comprehensive audit capability while managing system complexity through abstraction.
Solution Approach 2:
The system is segmented into distinct functional modules: log collection agents, Kafka message queue, Elasticsearch storage, and analysis engines. This segmentation allows each component to handle specific aspects of log processing independently, making the overall complex system more manageable and adaptable to different log sources.
2Speed
If real-time analysis of massive log data is performed, then detection speed is improved, but computational resources are consumed heavily
Solution Approach 1:
The patent extracts and stores only essential log fields in Elasticsearch after initial processing, separating detailed log data from key analytical fields. This extraction approach enables faster real-time analysis by focusing computational resources on critical fields while reducing overall data processing requirements.
Solution Approach 2:
Log data undergoes preliminary processing and field extraction before being stored in Elasticsearch. This preliminary action prepares the data in advance for rapid querying and analysis, improving detection speed while reducing the computational burden during actual security analysis events.
3Loss of information
If multiple log files in different formats are opened for problem tracing, then analysis completeness is improved, but operational efficiency deteriorates
Solution Approach 1:
The patent merges multiple log files from different applications and formats into a unified view within the Elasticsearch-based analysis interface. This merging allows operators to access comprehensive log data from multiple sources simultaneously, improving analysis completeness while maintaining operational efficiency through a single integrated interface rather than requiring manual opening of multiple files.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present disclosure provides a method and an apparatus of auditing a log, an electronic device, and a medium, which relates to a field of a computer technology, in particular to a field of an artificial intelligence technology and a security technology. The method of auditing the log specifically includes: transmitting a collected log file to a Kafka message queue, so as to arrange the log file in the Kafka message queue; storing the log file in the Kafka message queue directly in a first database, extracting a plurality of fields of the log file in the Kafka message queue, and storing the log file in a second database and transmitting the log file to an elastic search engine according to the plurality of fields extracted; and counting each field of the log file stored in the second database by a distributed processing engine, so as to determine an abnormal log field information.