Event-responsive selective log collection system and method

Abstract

An event-responsive selective log collection system and method are disclosed. According to an aspect of the present invention, an event-responsive selective log collection system comprises: a log buffer in which streaming log data is temporarily stored; an event monitor module for monitoring a preset event within the log data stored in the log buffer; a filter module for selecting a log filter corresponding to event information transferred from the event monitor module and filtering the log data stored in the log buffer; and a storage for storing the log data filtered by the filter module.

Description

Event-Responsive Selective Log Collection System and Method

[0001] The present invention relates to log collection, and more specifically, to an event-responsive selective log collection system and method.

[0002] The term "log" is known to have originated from logbooks related to navigation; in the past, wooden pieces (logs) were used to record ship speeds and navigation information, and these records were stored in logbooks. As this concept evolved, the recording of events in computer systems came to be called a log.

[0003] Common log processing methods output events occurring in systems and applications in real time and store and manage them in various ways. Log levels are generated based on predefined parameters and collected through agents or loggers. The generated (collected) logs are saved to local files or transmitted to a log server (remote server). The collected logs are stored in a database or a log management system (e.g., ElasticSearch, Splunk).

[0004] Although a log preprocessing process exists, its primary purpose is to maintain data consistency through data normalization, filtering, and the removal of unnecessary data. Collected logs are loaded into a database or log management system based on predefined log levels and preprocessing criteria. When the scope of the logs to be collected is broad and the log content is extensive, significant file system and database capacity is required.

[0005] Accordingly, the present invention has been devised to solve the aforementioned problems and aims to provide an event-responsive selective log collection system and method that can reduce the burden of storing all unnecessary logs and provide detailed logs at the necessary time.

[0006] In addition, the present invention is intended to provide an event-responsive selective log collection system and method that does not require pre-defining the storage range or level of the log for collection.

[0007] Other objects of the present invention will become more apparent through the preferred embodiments described below.

[0008] According to one aspect of the present invention, an event-responsive selective log collection system is provided, comprising: a log buffer in which streaming log data is temporarily stored; an event monitor module for monitoring a preset event within the log data stored in the log buffer; a filter module for filtering the log data stored in the log buffer by selecting a log filter corresponding to event information transmitted from the event monitor module; and a storage unit for storing the filtered log data by the filter module.

[0009] Here, the log buffer includes a light buffer where generated log data accumulates and a filter buffer where filtering is performed by the filter module, and the functions of the light buffer and the filter buffer are alternately switched.

[0010] In addition, the functions of the light buffer and the filter buffer are switched at the time when a certain amount or more of log data is stored in the light buffer, or at the time when the event monitor module recognizes an event of the log data stored in the light buffer.

[0011] Alternatively, the log buffer includes a light buffer where streaming log data accumulates, and a filter buffer where log data monitored by the event monitor module is stored and filtering is performed by the filter module.

[0012] In addition, the capacity of the filter buffer may be larger than that of the light buffer, and filtering is performed when a certain amount or more of log data is stored in the filter buffer or when log data corresponding to a detected event is stored.

[0013] Additionally, the filter module includes: a filter selector for selecting a log filter corresponding to the event information; a filtering log buffer for temporarily storing filtering log data using the log filter selected by the filter selector; and a log batch scheduler for storing the filtering log data stored in the filtering log buffer in the storage in the form of a batch write.

[0014]

[0015] According to another aspect of the present invention, a method for collecting logs performed in a computing system comprises the steps of: temporarily storing streaming log data in a log buffer; monitoring a preset event in the log data stored in the log buffer; selecting a log filter corresponding to the detected event; filtering the log data stored in the log buffer using the selected log filter; and storing the filtered log data in a storage. An event-responsive selective log collection method and a program for executing the method are provided.

[0016] Here, the log buffer includes a light buffer where generated log data accumulates and a filter buffer where filtering is performed by the filter module.

[0017] In addition, the functions of the light buffer and the filter buffer are switched at the time when a certain amount or more of log data is stored in the light buffer or at the time when the event monitor module recognizes an event of the log data stored in the light buffer.

[0018] Additionally, the step of storing the filtering log data in a storage unit includes: the step of temporarily storing the filtering log data in a filtering log buffer; and the step of storing the filtering log data stored in the filtering log buffer in the storage unit in a batch write format.

[0019] Other aspects, features, and advantages other than those described above will become clear from the following drawings, claims, and detailed description of the invention.

[0020] According to the present invention, the storage system capacity can be reduced by reducing the burden of storing all unnecessary logs through event-responsive variable log filtering, and detailed logs can be provided at the necessary time.

[0021] In addition, according to the present invention, there is an effect that eliminates the need to collect logs by pre-defining the storage range or level of the logs through variable log filtering.

[0022] FIG. 1 is a functional block diagram schematically illustrating the configuration of an event-responsive selective log collection system according to one embodiment of the present invention.

[0023] FIG. 2 is a flowchart illustrating an event-responsive selective log collection process according to an embodiment of the present invention.

[0024] FIG. 3 is a block diagram illustrating the detailed configuration of a log collection system according to one embodiment of the present invention.

[0025] FIGS. 4 to 6 are exemplary diagrams illustrating an event-responsive selective log collection process according to an embodiment of the present invention.

[0026] FIG. 7 is an exemplary diagram illustrating a log buffer operation method according to another embodiment of the present invention.

[0027] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0028] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0029] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, terms such as "first threshold," "second threshold," etc., to be described later may be pre-designated as thresholds that are substantially different or partially identical; however, since there is a possibility of confusion when expressed using the same word "threshold," the terms "first," "second," etc., will be used together for the convenience of distinction.

[0030] The terms used herein are merely for describing specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0031] Furthermore, the components of the embodiments described with reference to each drawing are not limited to the respective embodiments and may be implemented to be included in other embodiments within the scope of maintaining the technical spirit of the present invention. It is also obvious that multiple embodiments may be re-implemented as a single embodiment that integrates multiple embodiments, even if a separate description is omitted.

[0032] Furthermore, in the description referring to the attached drawings, identical components are assigned the same or related reference numerals regardless of drawing symbols, and redundant descriptions thereof are omitted. In describing the present invention, if it is determined that a detailed description of related prior art could unnecessarily obscure the essence of the present invention, such detailed description is omitted.

[0033]

[0034] FIG. 1 is a functional block diagram schematically illustrating the configuration of an event-responsive selective log collection system according to one embodiment of the present invention, and FIG. 2 is a flowchart illustrating an event-responsive selective log collection process according to one embodiment of the present invention.

[0035] Referring to FIG. 1, the log collection system according to the present embodiment includes a log buffer (10), an event monitor module (20), a filter module (30), and a storage (40).

[0036] Streaming log data is temporarily stored in the log buffer (10).

[0037] The event monitor module (20) monitors whether there is a preset event in the log data stored in the log buffer (10).

[0038] The filter module (30) is provided with a respective log filter corresponding to each event in advance, selects a log filter corresponding to the event information transmitted from the event monitor module (20) to filter the log data stored in the log buffer (10), and stores the filtered log data (hereinafter referred to as filtered log data) in the storage (40).

[0039] To reiterate the general process of log collection with reference to FIG. 2, the log data input via streaming is temporarily stored in a log buffer (10) (S10), the presence of events in the log data is monitored (S20), and a log filter corresponding to the detected events is selected and filtered (S30). Here, the filtered log data is temporarily stored in a filtering log buffer (32) and then stored in a storage (40) in a batch write format.

[0040] According to the present embodiment, by storing log data after filtering, there is no need to collect logs by pre-defining the storage range or log level. In other words, the entity transmitting or generating logs (application or system) does not need to significantly consider the volume or log level of the generated logs.

[0041] Furthermore, it is not always necessary to store log messages at every level to retrieve detailed logs at the time of an error or a specified event. Compared to general log collection and storage systems, the storage system capacity required to store logs is small.

[0042] In a typical system, the average proportion of messages by log level is estimated as follows based on various log management documents.

[0043] DEBUG : 60% ~ 70%

[0044] INFO : 15% ~ 20%

[0045] WARNING : 5% ~ 10%

[0046] ERROR : 2% ~ 5%

[0047] CRITICAL: Less than 1%

[0048] Storing all logs in preparation for cases where logs such as ERROR or CRITICAL occur, or when a specific event requires the analysis of overall logs, is inefficient in various aspects, including system performance and storage capacity burden. Therefore, through the event-responsive variable log filtering method according to the present embodiment, the burden of storing all unnecessary logs is reduced, and detailed logs can be provided when needed.

[0049]

[0050] FIG. 3 is a block diagram illustrating the detailed configuration of a log collection system according to one embodiment of the present invention, and FIGS. 4 to 6 are exemplary diagrams illustrating an event-responsive selective log collection process according to one embodiment of the present invention.

[0051] Referring to FIG. 3, the log buffer (10) includes a write buffer (10-1, write buffer) and a filter buffer (10-2, filter buffer). The write buffer (10-1) is a memory where log data is loaded via streaming, and the filter buffer (10-2) is a memory where filtering is performed on the loaded log data. According to one example, when a certain amount of log data is loaded into the write buffer (10-1), it is switched to the filter buffer (10-2), and the filter buffer (10-2), having finished filtering, is switched back to the write buffer (10-1). That is, the write buffer (10-1) and the filter buffer (10-2) alternately switch their functions.

[0052] The Event Monitor Module (20) includes an Event Monitor that detects events in the logs of the Light Buffer (10-1) and an Event Data Manager that manages the event data to be detected.

[0053] A filter module (30, Log Filter Module) includes a filter selector (31, Filter Selector) that specifies a log filter to use according to a detected event among log filters, a filtered log buffer (32, Filtered Log Buffer) in which the filtered log data of the filter buffer (10-2) is filtered by the selected log filter is temporarily stored, and a log batch scheduler (33, Log Batch Scheduler) for storing the temporarily stored filtered log data in a storage (40) in a batch write manner.

[0054] Referring to FIGS. 4 to 6, the log collection process is described in more detail. First, as shown in FIG. 4, when log data is transmitted, the log data is primarily stored in a write buffer (10-1). For the logs loaded in the write buffer (10-1), detection (monitoring) is performed to determine if there is a specified event target. A log filter is specified to adjust the log collection range according to the detected event.

[0055] That is, the Event Monitor continuously monitors the log data stored in the write buffer (10-1), and in order to perform a log filtering operation corresponding to the detected event, it transmits filter information to the filter selector (31) to filter the corresponding log buffer.

[0056] Next, referring to FIG. 5, Buffer 1, which functions as a light buffer (10-1) where an event is detected, and Buffer 2, which functions as a filter buffer (10-2), are swapped. That is, Buffer 1 becomes the filter buffer (10-2), and Buffer 2 performs the function of the light buffer (10-1).

[0057] That is, the light buffer (10-1) and the filter buffer (10-2) are swapped so that log data is continuously added to the new light buffer, the filter buffer (10-2) is prepared to apply a log filter, and the filter selector (31) selects a log filter that matches the event type received from the event monitor and prepares to filter the log information in the filter buffer (10-2).

[0058] Log data stored in the new filter buffer (buffer 1) is filtered through a selected log filter based on event detection. By separating the buffer for storing log data (write buffer (10-1)) and the buffer for filtering (filter buffer), I / O bottlenecks are prevented.

[0059] Referring to FIG. 6, data filtered by a log filter is stored in a log storage (40), and according to one example, the filtered log data is processed in batches using batch writing. Since a large volume of refined logs is stored in batches through batch operations, network overhead reduction and database I / O efficiency improvement can be achieved.

[0060] That is, the log data filtered by the newly configured log filter is stored in the filtering log buffer (32) and is stored in the storage (40) in the form of a batch write by the log batch scheduler (33), which is separated into a separate process.

[0061] Here, according to one example, a transition to the filter buffer (10-2) may be performed when a certain amount or more of log data is stored in the light buffer (10-1). According to another example, since it may be inefficient to filter using a single log filter when different events exist, a transition to the filter buffer (10-2) may be performed whenever an event is detected. That is, the event monitor module (20) switches the functions of the light buffer (10-1) and the filter buffer (10-2) when it recognizes an event of the log data stored in the light buffer (10-1). Therefore, when an event is detected, the corresponding log filter is selected to perform filtering immediately.

[0062]

[0063] Up until now, the method of the first buffer and the second buffer being interchangeable as the write buffer (10-1) and the filter buffer (10-2) has been mainly explained. According to another embodiment, the first buffer may be fixed as the write buffer (10-1) and the second buffer may be fixed as the filter buffer (10-2).

[0064] FIG. 7 is an exemplary diagram illustrating a log buffer operation method according to another embodiment of the present invention.

[0065] Referring to FIG. 7, the event monitor module (20) monitors events for log data stored in the light buffer (10-1), and the log data that has been monitored is transferred to the filter buffer (10-2).

[0066] Here, filtering may be performed when a certain amount or more of log data is stored in the filter buffer (10-2) or when log data corresponding to a detected event is stored. That is, filtering is performed when a certain amount or more of log data is loaded into the filter buffer (10-2), or when log data detected as an event is transferred to the filter buffer (10-2), filtering is performed on the log data stored in the filter buffer (10-2). Until filtering is completed, the transfer of log data from the write buffer (10-1) to the filter buffer (10-2) is stopped.

[0067] And, according to one example, the capacity of the filter buffer (10-2) is formed to be larger than that of the write buffer (10-1). Since the write buffer (10-1) only detects and transmits whether an event exists for the stored log data and only needs to store the log data while filtering is performed in the filter buffer (10-2), it does not need to have a larger capacity than the filter buffer (10-2). Therefore, a larger amount of log data can be filtered at once for log data where no event is detected.

[0068]

[0069] A computer program stored on a computer-readable medium may be provided to perform the event-responsive selective log collection method according to the present invention described above.

[0070] In addition, the aforementioned event-responsive selective log collection method can be implemented as computer-readable code on a computer-readable recording medium. Computer-readable recording media include all types of recording media in which data that can be decoded by a computer system is stored. Examples include ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, optical data storage devices, etc. Furthermore, computer-readable recording media can be distributed across computer systems connected via a computer network and stored and executed as code that can be read in a distributed manner.

[0071] Furthermore, although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims.

Claims

1. A log buffer where streaming log data is temporarily stored; An event monitor module that monitors pre-configured events within log data stored in the above log buffer; A filter module that filters log data stored in the log buffer by selecting a log filter corresponding to event information transmitted from the event monitor module; and An event-responsive selective log collection system comprising a storage for storing filtered log data by the above filter module.

2. In Claim 1, The above log buffer includes a light buffer where generated log data is accumulated and a filter buffer where filtering is performed by the filter module, and the light buffer and the filter buffer alternate functions, in an event-responsive selective log collection system.

3. In Claim 2, An event-responsive selective log collection system in which the functions of the light buffer and the filter buffer are switched at the point when a certain amount or more of log data is stored in the light buffer, or at the point when the event monitor module recognizes an event of the log data stored in the light buffer.

4. In Claim 1, An event-responsive selective log collection system comprising a log buffer in which streaming log data is accumulated, a light buffer in which log data monitored by the event monitor module is stored, and a filter buffer in which filtering is performed by the filter module.

5. In Claim 4, An event-responsive selective log collection system in which the capacity of the filter buffer is larger than that of the light buffer.

6. In Claim 5, An event-responsive selective log collection system in which filtering is performed when a certain amount or more of log data is stored in the filter buffer or when log data corresponding to a detected event is stored.

7. In Claim 1, The above filter module is, A filter selector that selects a log filter corresponding to the above event information; A filtering log buffer for temporarily storing filtering log data using a log filter selected by the filter selector above; and An event-responsive selective log collection system comprising a log batch scheduler that stores filtering log data stored in the filtering log buffer in the storage in the form of a batch write.

8. In a log collection method performed in a computing system, Step of temporarily storing streaming log data in a log buffer; A step of monitoring pre-configured events within the log data stored in the above log buffer; A step of selecting a log filter corresponding to a detected event; A step of filtering the log data stored in the log buffer using a selected log filter; and An event-responsive optional log collection method comprising the step of storing filtered filtered log data in a repository.

9. In Claim 8, An event-responsive selective log collection method comprising a log buffer in which generated log data is accumulated, a light buffer in which filtering is performed by the filter module.

10. In Claim 9, An event-responsive selective log collection method that switches the functions of the light buffer and the filter buffer at a time when a certain amount or more of log data is stored in the light buffer or when the event monitor module recognizes an event of the log data stored in the light buffer.

11. In Claim 8, The step of storing the above filtering log data in a storage is, Step of temporarily storing the above filtering log data in a filtering log buffer; and An event-responsive selective log collection method comprising the step of storing filtering log data stored in the filtering log buffer in the storage in the form of a batch write.

12. A computer program stored on a computer-readable medium for performing an event-responsive selective log collection method, wherein the computer program causes the computer to perform the following steps, the steps being, Step of temporarily storing streaming log data in a log buffer; A step of monitoring pre-configured events within the log data stored in the above log buffer; A step of selecting a log filter corresponding to a detected event; A step of filtering the log data stored in the log buffer using a selected log filter; and A computer program stored on a computer-readable medium, comprising a step system for storing filtered filtering log data in a storage.

Images