Method and apparatus for processing log, device, and product

Abstract

The present disclosure relates to a method and apparatus for processing logs, a device, and a product. The method comprises: in response to that an event recorded into a log has been triggered, a client classifying the event into a target event category. The method further comprises: in response to determining that a log analysis task has been triggered, the client acquiring a plurality of events of the target event category. The method further comprises: the client generating event aggregation data on the basis of the plurality of events. In addition, the method further comprises: the client uploading the event aggregation data to a server.

Description

Methods, apparatus, devices, and products for processing logs Technical Field

[0001] This disclosure relates to the field of data security, and more specifically to methods, apparatus, devices, and products for processing logs. Background Technology

[0002] With the rapid popularization of mobile internet, mobile applications have gradually integrated into users' daily lives and become irreplaceable tools. For example, video applications have not only redefined people's entertainment methods, but also provided a key platform for information dissemination, social interaction, and creative expression.

[0003] In the actual operation of mobile applications, a large amount of log data is generated. This data covers system status, operation records, application performance indicators, network communication records, and information on errors and anomalies. If this data is not properly collected and securely managed, it may lead to data leaks or misuse, thus posing a potential threat to overall data security. Summary of the Invention

[0004] In a first aspect of the embodiments of this disclosure, a method for processing logs is provided. The method includes, in response to an event being triggered in the log, a client classifying the event into a target event category. The method further includes, in response to determining that a log analysis task is triggered, the client acquiring multiple events having the target event category. The method also includes, the client generating event aggregation data based on the multiple events. Furthermore, the method includes, the client uploading the event aggregation data to a server.

[0005] In a second aspect of the embodiments of this disclosure, an apparatus for processing logs is provided. The apparatus includes an event classification module configured to, in response to the triggering of an event recorded in the log, classify the event into a target event category by a client. The apparatus also includes an event acquisition module configured to, in response to the triggering of a log analysis task, acquire multiple events having the target event category by the client. The apparatus further includes an event aggregation module configured to, by the client, generate event aggregation data based on the multiple events. Furthermore, the apparatus includes an event upload module configured to, by the client, upload the event aggregation data to a server.

[0006] In a third aspect of embodiments of this disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a method for processing logs. The method includes, in response to an event being logged in the log being triggered, a client classifying the event into a target event category. The method further includes, in response to determining that a log analysis task is triggered, the client acquiring multiple events having the target event category. The method also includes, the client generating event aggregation data based on the multiple events. Furthermore, the method includes, the client uploading the event aggregation data to a server.

[0007] In a fourth aspect of embodiments of this disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to implement a method for processing logs. The method includes, in response to an event being logged in the log being triggered, a client classifying the event into a target event category. The method also includes, in response to a determination that a log analysis task is triggered, the client acquiring multiple events having the target event category. The method further includes, the client generating event aggregation data based on the multiple events. Furthermore, the method includes, the client uploading the event aggregation data to a server.

[0008] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 illustrates a schematic diagram of an example environment in which various embodiments of the present disclosure may be implemented;

[0011] Figure 2 shows a flowchart of a method for processing logs according to some embodiments of the present disclosure;

[0012] Figure 3 shows a schematic diagram of an example system for processing logs according to some embodiments of the present disclosure;

[0013] Figure 4 illustrates a schematic diagram of an example of using an event aggregator to generate event aggregation data according to some embodiments of the present disclosure;

[0014] Figure 5 illustrates a schematic diagram of an example of using a data filter to process data according to some embodiments of the present disclosure;

[0015] Figure 6 shows a block diagram of an apparatus for processing logs according to some embodiments of the present disclosure;

[0016] Figure 7 shows a block diagram of a device capable of implementing several embodiments of the present disclosure. Detailed Implementation

[0017] It is understood that all user-related data involved in this technical solution should be obtained and used only after authorization from the user. This means that if it is necessary to use a user's personal information in this technical solution, the user's explicit consent and authorization are required before obtaining this data; otherwise, no related data collection and use will be carried out. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly followed in the process of data collection, use, and storage, and necessary technical measures should be taken to protect user data security and ensure the secure use of data.

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0020] Mobile applications generate a large amount of log data during operation, including but not limited to system information, operation records, application performance data, network communication data, and error and exception data. These logs are crucial tools for development engineers, operations engineers, and data analysts to understand the application's operational status, troubleshoot problems, optimize performance, and make critical decisions. Depending on the needs, logs record various events from application startup to runtime, such as user actions, background tasks, exceptions, and network requests. To analyze the application's operational status and operation records, application logs can be uploaded to a server for access by data analysis systems.

[0021] In this article, events refer to various operations or state changes triggered during the operation of an application. For example, events can include user button clicks, page transitions, video playback, network requests, and error exceptions. Application engineers can pre-define event triggering conditions (also known as "event tracking") in the application's code. When the event triggering conditions are met, the application can capture the event and log the data associated with it. In the logs, the data associated with these events may include specific types of target data (e.g., user-related data). If this data is not properly protected, it may lead to data leaks or misuse.

[0022] In some related technologies, all logs are uploaded to a server, where the target data within the logs is anonymized. However, the potential risk of interception or leakage remains during data transmission to the server. In other technologies, applications collect only a small amount of data that does not contain the target data. While this approach effectively ensures data security, it also hinders necessary performance monitoring and troubleshooting. Still other related technologies allow users to decide whether to upload data, but users may lack the relevant technical background and are prone to making inappropriate decisions, negatively impacting the user experience.

[0023] To address this, embodiments of this disclosure propose a scheme for processing logs. In this scheme, if an event to be logged is triggered, the client can categorize the event. When a log analysis task is triggered, the client can acquire multiple events with a target event category. The client can then perform aggregation operations on the multiple events to transform the raw log data into high-dimensional event aggregation data (e.g., probability statistics, trend analysis, etc.). The client can then upload the event aggregation data to the server. In this document, aggregation operations refer to generating event aggregation data by performing operations such as statistical calculations and log analysis on multiple events. The event aggregation data can indicate the statistical characteristics of the aggregated events or the log analysis results for multiple events.

[0024] This approach reduces the likelihood of target data being included in aggregated event data, thereby decreasing the amount of target data uploaded to the server and improving data security. Furthermore, aggregated event data is smaller than raw log data; therefore, converting raw log data into aggregated event data reduces the amount of data that needs to be uploaded to the server, thus saving network resources.

[0025] Figure 1 illustrates a schematic diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented. As shown in Figure 1, environment 100 includes a client 102 and a server 104. The client 102 can be any device capable of running applications thereon. For example, the client 102 can be a mobile phone, a smart wearable device, a tablet computer, a laptop computer, a desktop computer, or an Internet of Things device, etc. The server 104 can be any device with processing or computing capabilities. For example, the server 104 can be a cloud server, a local server, a virtual server, a laptop computer, or a desktop computer, etc.

[0026] In environment 100, application 106 can run on client 102. Application 106 can be any application that generates logs during its runtime. For example, application 106 could be a lifestyle service application, a video application, a music application, or a social application. Logs are recording files automatically generated during application runtime, typically including application status, operations, error messages, performance metrics, user behavior, and system events. Logs help development engineers, maintenance engineers, and data analysis engineers understand the application's working status, troubleshoot problems, and make appropriate decisions.

[0027] In environment 100, application 106 can generate log 108 during runtime. Log 108 can include multiple events 110-1, 110-2, ..., and 110-N (collectively referred to as event 110). For example, event 110 can include normal operation events in the application (e.g., page entry events, page exit events, button click events, form submission events, etc.), error events encountered by the application during runtime (e.g., crash events, exception events, etc.), and events related to application performance (e.g., application startup events, page load events, API call events, etc.).

[0028] In environment 100, when event 110 is triggered, client 102 can categorize event 110. For example, client 102 can categorize event 110 based on log analysis tasks associated with it. As shown in Figure 1, after categorization, event 110 can have corresponding event categories 112-1, 112-2, ..., and 112-N (collectively referred to as event category 112). Event category 112 can include business events and engineering events. Business events can be used to collect operational data related to business operations, helping the product operations team analyze historical operation records, optimize feature design, and improve user experience. Engineering events can be used to collect data related to system and application performance, helping the technical team locate problems, optimize performance, and monitor the application's operational status.

[0029] In environment 100, when client 102 determines that log analysis task 114 has been triggered, it can obtain multiple events 116-1, 116-2, ..., and 116-K (collectively referred to as events 116) with a target event category from event 110. In some embodiments, the target event category can be a pre-set category. For example, the target event category can be an engineering event. Since engineering events are typically used to troubleshoot faults or defects in applications and optimize application performance, aggregating these events at client 102 (e.g., probability statistics, trend analysis, etc.) before uploading them to server 104 will not have a significant impact on subsequent log analysis. In some embodiments, the target event category can be an event category set for log analysis task 114.

[0030] In environment 100, after identifying event 116 with a target event category, client 102 can generate event aggregate data 118 based on event 116. For example, client 102 can perform aggregation or statistical analysis on event 116 to generate event aggregate data 118. For instance, client 102 can calculate statistical data such as the frequency of occurrence of various events in event 116, the distribution or average value of performance indicators, and the pattern of interaction behavior, to reduce the target data included in event aggregate data 118. Then, client 102 can upload event aggregate data 118 to server 104 without uploading the original data of event 116.

[0031] In this way, the likelihood of target data being included in the event aggregation data 118 can be reduced, thereby reducing the amount of target data uploaded to the server 104 and improving data security. Furthermore, the event aggregation data 118 is smaller in size than the original data of event 116; therefore, converting event 116 into event aggregation data 118 reduces the amount of data that needs to be uploaded to the server 104, thus saving network resources.

[0032] Figure 2 illustrates a flowchart of a method 200 for processing logs according to some embodiments of the present disclosure. Method 200 can be performed by a client, for example, by client 102 in Figure 1. As shown in Figure 2, at block 202, in response to an event being triggered in the log, the client can classify the event into a target event category. For example, in environment 100 as shown in Figure 1, when event 110 is triggered, client 102 can classify event 110. For example, client 102 can classify event 110 based on a log analysis task associated with event 110. After being classified, event 110 can have a corresponding event category 112. In some embodiments, the target event category can be a pre-set category (e.g., engineering event). In some embodiments, the target event category can be an event category set for log analysis task 114.

[0033] In box 204, in response to determining that a log analysis task has been triggered, the client can retrieve multiple events with a target event category. For example, in environment 100 as shown in Figure 1, when client 102 determines that log analysis task 114 has been triggered, it can retrieve multiple events 116 with a target event category from event 110.

[0034] In box 206, the client can generate event aggregate data based on multiple events. For example, in environment 100 as shown in Figure 1, client 102 can generate event aggregate data 118 based on event 116. For example, client 102 can perform aggregation or statistical analysis on event 116 to generate event aggregate data 118. For example, client 102 can calculate statistical data such as the frequency of occurrence of various events in event 116, the distribution or average value of performance indicators, and the pattern of interaction behavior, to reduce the target data included in event aggregate data 118.

[0035] In box 208, the client can upload event aggregate data to the server. For example, in environment 100 as shown in Figure 1, client 102 can upload event aggregate data 118 to server 104 without uploading the original data of event 116.

[0036] This approach reduces the likelihood of target data being included in aggregated event data, thereby decreasing the amount of target data uploaded to the server and improving data security. Furthermore, aggregated event data is smaller than raw log data; therefore, converting raw log data into aggregated event data reduces the amount of data that needs to be uploaded to the server, thus saving network resources.

[0037] In some embodiments, events are stored in a storage area located on the client side. This storage area is configured to store event streams based on time series. The events stored in the storage area include the event's trigger time, event type, and event content. When retrieving multiple events of a target event category, the client can retrieve multiple events from the event stream stored in this storage area. In some embodiments, the client can determine the time difference between the event's trigger time and the current time. If the time difference exceeds a predetermined retention time threshold, the client can delete the event from the storage area.

[0038] In some embodiments, the event is a first event, and the client can determine that a second event has been triggered when it is determined that the log analysis task has been triggered. The client can determine that the log analysis task has been triggered in response to the determination that the triggering of the second event satisfies the pre-set triggering conditions of the log analysis task.

[0039] In some embodiments, when uploading event aggregate data to the server, in response to receiving a log upload request from the server, the client may upload event aggregate data to the server, wherein the log upload request is sent by the server to the client when it receives a fault or defect reported by the user.

[0040] Figure 3 illustrates a schematic diagram of an example 300 of a system for processing logs according to some embodiments of the present disclosure. The system shown in example 300 can be deployed on a client, for example, on client 102 in Figure 1. Example 300 includes an event collector 302, an event classifier 304, an event stream storage center 306, an event aggregator 308, a data filter 310, and a data upload module 312.

[0041] Event collector 302 can be configured to collect various events triggered during application runtime. For example, event collector 302 can utilize event tracking tools or software development kits (SDKs) to capture user-triggered action events (e.g., click events, swipe events, etc.) and application-generated runtime events (e.g., startup events, error events, etc.). Event collector 302 can also structure the captured events (e.g., structure the events into a JSON data structure) to facilitate subsequent processing and storage of the events.

[0042] Event classifier 304 can be configured to classify events so that clients can execute various processing logic based on event categories. In some embodiments, event classifier 304 can classify events according to a pre-set mapping table of events and event categories. Event classifier 304 can classify events into two event categories: one event category indicates that the raw data of the event needs to be uploaded to the server, and the other event category indicates that the event can be aggregated at the client before being uploaded to the server. For example, in the pre-set mapping table, event categories can include business events that require the raw data to be uploaded and engineering events that can be aggregated. Business events can be used to collect operational data related to business operations, helping product operations teams analyze historical operation records, optimize feature design, and improve user experience. For example, events such as clicking the "like" button and submitting a comment can be mapped to business events. Engineering events can be used to collect data related to system and application performance, helping technical teams locate problems, optimize performance, and monitor the application's running status. For example, page load events and API call failure events can be mapped to engineering events.

[0043] After the event classifier 304 classifies the events, events that can be further aggregated can be sent to the event stream storage center 306, while events requiring the upload of raw data can be sent to the data filter 310. In this way, different processing logic can be executed for events with different event categories, thereby maintaining the accuracy of data analysis while reducing the amount of target data and improving data security.

[0044] Event stream storage center 306 can be configured to store event streams based on time series. Event stream storage center 306 can be implemented using, for example, an SQLite database. Events stored in event stream storage center 306 can include the event's trigger time, event type, and event content. Event stream storage center 306 can periodically check the time difference between the event's trigger time and the current time; this time difference indicates the duration the event has been stored at the client. If an event is stored at the client for more than a preset time threshold (e.g., 24 hours), event stream storage center 306 can automatically delete the event. In this way, it is ensured that log data is not stored long-term at the client, thereby improving data security.

[0045] Event aggregator 308 can be configured to retrieve multiple events from event stream storage center 306 when a trigger event for a log analysis task is detected, and perform aggregation operations or statistical analysis on the retrieved events to generate event aggregation data. For example, event aggregator 308 can retrieve pre-set trigger conditions for the log analysis task, and trigger the log analysis task when it determines that the captured events meet the trigger conditions. Log analysis tasks can, for example, calculate statistical data such as the frequency of occurrence of various events, the distribution or average value of performance metrics, or patterns of interaction behavior, to reduce the amount of target data included in the calculated statistical data and improve data security.

[0046] In some embodiments, the event aggregator 308 can store multiple log analysis tasks, each of which can consist of a series of executable rules that can be executed by the rule engine of the event aggregator 308. In some embodiments, the event aggregator 308 can determine the time to perform the aggregation operation in real time, so as to perform the aggregation operation when the client is relatively idle, thereby reducing the impact of the aggregation operation on the normal operation of the application. In some embodiments, the log analysis tasks can be triggered by different events. In some embodiments, the probability of executing the log analysis task can be set for the triggering event, thereby reducing the frequency of executing the log analysis task and saving processing resources.

[0047] Data filter 310 can be configured to perform filtering operations on data from event aggregator 308 and data from event classifier 304 to de-identify the data. Data filter 310 can utilize machine learning models deployed at the client or string processing rules (e.g., regular expressions) stored at the client to identify target data. In some embodiments, data filter 310 can combine machine learning models and string processing rules to identify target data to improve the accuracy of the identification results. The identified target data can be replaced with predetermined values ​​or deleted by data filter 310. In this way, target data can be filtered at the client level before being uploaded to the server, thereby reducing the possibility of data leakage during network transmission.

[0048] The data upload module 312 can be configured to upload data to the server using different control logics based on the data source and requirements. In some embodiments, the data upload module 312 can store logs associated with the fault or defect being checked at the client end instead of uploading them to the server. Then, when the data upload module 312 determines that the user has provided feedback on the fault or defect, it uploads the associated logs to the server. In this way, the amount of data uploaded can be reduced while ensuring normal business operation, thus saving network resources. In some embodiments, when the business requirement only requires event aggregation data, the data upload module 312 can use anonymization or pseudonymization to remove data such as user identifiers and device identifiers, further reducing the amount of user-related data in the uploaded data.

[0049] In some embodiments, when generating event aggregation data based on multiple events, the client can obtain the aggregation strategy of the log analysis task. The client can then generate event aggregation data by applying the aggregation strategy to the multiple events. In some embodiments, the client can obtain the aggregation strategy of the log analysis task from the server, where the aggregation strategy is configured by the user in real time on the server. In some embodiments, the client can determine its resource utilization. In response to determining that the resource utilization is less than a predetermined resource utilization threshold, the client can generate event aggregation data based on multiple events.

[0050] Figure 4 illustrates a schematic diagram of an example 400 using an event aggregator to generate event aggregation data according to some embodiments of the present disclosure. Example 400 includes a client 402 and a server 404, wherein an event aggregator 408 (e.g., event aggregator 308 in Figure 3) is deployed on the client 402. As shown in Figure 4, log analysis tasks 416-1, 416-2, ..., and 416-N (collectively referred to as log analysis tasks 416) and corresponding aggregation policies 418-1, 418-2, ..., and 418-N (collectively referred to as aggregation policies 418) can be set at the client 402. Each aggregation policy 418 may include a series of executable rules. When a log analysis task 416 is triggered, the rule engine of the event aggregator 408 can execute the corresponding aggregation policy 418 on multiple events to generate event aggregation data.

[0051] In Example 400, server 404 can receive user input 410, which may include log analysis task 412 and corresponding aggregation policy 414. Log analysis task 412 can be one of the log analysis tasks 416 set in client 402, or it can be a new log analysis task. After receiving user input 410, server 404 can send log analysis task 412 and aggregation policy 414 to client 402 in real time. After receiving log analysis task 412 and aggregation policy 414, client 402 can determine whether the already set log analysis tasks 416 include log analysis task 412. If log analysis task 412 is included in log analysis task 416, client 402 can update the aggregation policy of that log analysis task to aggregation policy 414. If log analysis task 412 is not included in log analysis task 416, log analysis task 412 and aggregation policy 414 can be set in client 402.

[0052] For example, if client 402 determines that log analysis task 412 and log analysis task 416-N are the same task, then client 402 can update aggregation policy 418-N to aggregation policy 414. When log analysis task 416-N is triggered, event aggregator 408 can obtain an event stream including events 420-1, 420-2, ..., and 420-K (collectively referred to as event 420) from the event stream storage center (e.g., event stream storage center 306 in Figure 3), and apply aggregation policy 418-N to the event stream to generate event aggregation data 422.

[0053] In this way, even if the code or version of the client 402 is not updated, users can update the log analysis task and aggregation strategy at the client 402 in real time, thereby simplifying the process of adjusting event aggregation operations and improving the flexibility of data analysis and data security.

[0054] In some embodiments, when uploading event aggregate data to a server, the client can identify specific types of target data within the event aggregate data. The client can then generate processed event aggregate data by deleting or replacing the target data. The client can then upload the processed event aggregate data to the server. In some embodiments, the client can utilize a natural language processing model to identify potential target data from the event aggregate data. The client can then use regular expressions to determine if the potential target data is of a specific type.

[0055] Figure 5 illustrates a schematic diagram of an example 500 using a data filter to process data according to some embodiments of the present disclosure. As shown in Figure 5, a data filter 510 (e.g., data filter 310 in Figure 3) and a data processing module 518 are deployed on the client. In example 500, the data filter 510 may receive data 502 from an event classifier (e.g., event classifier 304 in Figure 3) or an event aggregator (e.g., event aggregator 308 in Figure 3). Data 502 may include event aggregated data 504 (e.g., from the event aggregator), unaggregated raw data 506 (e.g., from the event classifier), or both.

[0056] In Example 500, to filter target data (e.g., user-related data) from data 502, data filter 510 can input the received data 502 into a Natural Language Processing (NLP) model 512 to identify the target data in data 502. A Natural Language Processing model is a machine learning model capable of performing natural language understanding tasks. It can process and analyze text to accomplish various language tasks, such as text classification, sentiment analysis, machine translation, question answering systems, etc. In Example 500, the Natural Language Processing model 512 can be a language model for identifying target data, fine-tuned based on a pre-trained language model (e.g., the BERT model). However, the Natural Language Processing model 512 exhibits high recall and low precision in the task of identifying target data. Recall and precision are commonly used evaluation metrics in classification tasks, where recall represents the proportion of positive samples identified by the model out of the total number of actual positive samples, and precision represents the proportion of samples predicted as positive by the model that are actually positive samples.

[0057] In Example 500, based on the characteristics of Natural Language Processing (NLP) Model 512, the target data identified by NLP Model 512 can be determined as potential target data. This potential target data can then be input into String Processing Module 514. String Processing Module 514 can utilize string processing rules (e.g., regular expressions) to identify target data 516 from the potential target data. Compared to NLP Model 512, string processing rules have a lower recall and a higher precision. Therefore, by combining NLP Model 512 and String Processing Module 514 to identify target data 516 in data 502, the comprehensiveness and accuracy of target data identification can be improved.

[0058] After identifying target data 516 in data 502, data processing module 518 can replace target data 516 with a predetermined value (e.g., a placeholder) or delete target data 516 from data 502, thereby generating processed data 520. Processed data 520 includes event aggregate data 522 or raw data 524, in which the target data has been replaced or deleted. Then, processed data 520 can be uploaded to the server by a data upload module (e.g., data upload module 312 in Figure 3).

[0059] In this way, the data filter 510 can combine the natural language processing model 512 and the string processing module 514 to identify the target data 516 in the data 502, thereby improving the comprehensiveness and accuracy of target data identification. By processing the target data 516, the possibility of the target data being included in the data uploaded to the server can be reduced, thereby reducing the risk of target data leakage during network transmission and improving data security.

[0060] Figure 6 shows a block diagram of an apparatus 600 for processing logs according to some embodiments of the present disclosure. As shown in Figure 6, the apparatus 600 includes an event classification module 602, configured to classify events into a target event category by a client in response to an event being recorded in the log. The apparatus 600 also includes an event acquisition module 604, configured to acquire multiple events having the target event category by a client in response to a log analysis task being determined to be triggered. The apparatus 600 also includes an event aggregation module 606, configured to generate event aggregation data by the client based on the multiple events. Furthermore, the apparatus 600 includes an event upload module 608, configured to upload the event aggregation data from the client to a server.

[0061] In some embodiments, events are stored in a storage area located on the client, the client's storage area is configured to store event streams based on time series, the events stored in the storage area include the event's trigger time, the event's type, and the event's content, and the event acquisition module 604 includes: acquiring multiple events from the event stream stored in the storage area.

[0062] In some embodiments, the apparatus 600 further includes: a time difference determination module configured to determine the time difference between the trigger time of an event and the current time; and a time difference comparison module configured to delete the event from the client's storage area in response to the time difference being greater than a predetermined storage time threshold.

[0063] In some embodiments, the event is a first event, and the event aggregation module 606 includes: an event triggering module configured to determine that a second event is triggered; and a task triggering module configured to determine that a log analysis task is triggered in response to determining that the triggering of the second event satisfies the triggering conditions of a pre-set log analysis task.

[0064] In some embodiments, the event aggregation module 606 includes: an aggregation strategy module configured to obtain an aggregation strategy for a log analysis task; and an aggregation data generation module configured to generate event aggregation data by applying the aggregation strategy to multiple events.

[0065] In some embodiments, the aggregation strategy acquisition module includes an aggregation strategy transmission module configured to acquire the aggregation strategy of the log analysis task from the server, wherein the aggregation strategy is configured by the user in real time on the server.

[0066] In some embodiments, the event aggregation module 606 includes: a resource utilization determination module configured to determine the resource utilization of a client; and a resource utilization usage module configured to generate event aggregation data based on multiple events in response to determining that the resource utilization is less than a predetermined resource utilization threshold.

[0067] In some embodiments, the event upload module includes: a target data identification module configured to identify a specific type of target data in the event aggregate data; a target data processing module configured to generate processed event aggregate data by deleting or replacing the target data; and a data upload module configured to upload the processed event aggregate data to a server.

[0068] In some embodiments, the target data identification module includes: a model usage module configured to use a natural language processing model to identify potential target data from event aggregation data; and a regular expression usage module configured to use regular expressions to determine whether the potential target data is target data of a specific type.

[0069] In some embodiments, the event upload module 608 includes an upload request receiving module configured to upload event aggregate data to the server in response to receiving a log upload request from the server, wherein the log upload request is sent by the server to the client when it receives a fault or defect reported by the user.

[0070] It is understood that by utilizing the apparatus 600 of this disclosure, at least one of the many advantages achievable by the methods or processes described above can be realized. For example, the likelihood of target data being included in the event aggregate data can be reduced, thereby reducing the amount of target data uploaded to the server and improving data security. Furthermore, since the event aggregate data is smaller in size than the original log data, converting the original log data into event aggregate data can reduce the amount of data that needs to be uploaded to the server, thereby saving network resources.

[0071] Figure 7 shows a block diagram of a device 700 capable of implementing various embodiments of the present disclosure. Device 700 may be, for example, a client 102 or a server 104 as shown in Figure 1. As shown in Figure 7, device 700 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 702 or loaded from storage unit 708 into random access memory (RAM) 703. Various programs and data required for the operation of device 700 may also be stored in RAM 703. The CPU / GPU 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704. Although not shown in Figure 7, device 700 may also include a coprocessor.

[0072] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0073] The various methods or processes described above can be executed by CPU / GPU 701. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by CPU / GPU 701, one or more steps or actions in the methods or processes described above can be performed.

[0074] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0075] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0076] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0077] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0078] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0079] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0081] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing logs, comprising: In response to an event being logged and triggered, the client categorizes the event into a target event category; In response to the determination that a log analysis task has been triggered, the client retrieves multiple events with the target event category; The client generates event aggregation data based on the multiple events; as well as The client uploads the event aggregation data to the server.

2. The method of claim 1, wherein the events are stored in a storage area located on the client, the storage area of ​​the client is configured to store event streams based on time series, the events stored in the storage area include the event trigger time, the event type, and the event content, and the client obtaining the plurality of events having the target event category includes: The plurality of events are obtained from the event stream stored in the storage area.

3. The method according to claim 2, further comprising: Determine the time difference between the trigger time of the event and the current time; as well as In response to the time difference being greater than a predetermined retention time threshold, the event is deleted from the storage area of ​​the client.

4. The method of claim 1, wherein the event is a first event, and determining that the log analysis task is triggered includes: It is confirmed that the second event has been triggered; as well as In response to determining that the triggering of the second event satisfies the pre-set triggering conditions of the log analysis task, it is determined that the log analysis task has been triggered.

5. The method of claim 1, wherein generating the event aggregation data by the client based on the plurality of events comprises: Obtain the aggregation strategy for the log analysis task; as well as The event aggregation data is generated by applying the aggregation strategy to the multiple events.

6. The method according to claim 5, wherein obtaining the aggregation strategy of the log analysis task includes: The aggregation strategy for the log analysis task is obtained from the server, wherein the aggregation strategy is configured by the user in real time on the server.

7. The method of claim 1, wherein generating the event aggregation data by the client based on the plurality of events comprises: Determine the resource utilization rate of the client; as well as In response to determining that the resource utilization rate is less than a predetermined resource utilization rate threshold, the event aggregation data is generated based on the multiple events.

8. The method according to claim 1, wherein uploading the event aggregation data to the server comprises: Identify specific types of target data within the aggregated event data; Processed event aggregation data is generated by deleting or replacing the target data; as well as The processed event aggregation data is uploaded to the server.

9. The method of claim 8, wherein identifying the specific type of target data in the event aggregation data comprises: Natural language processing models are used to identify potential target data from the event aggregation data; as well as Regular expressions are used to determine whether the potential target data is target data of the specific type.

10. The method according to claim 1, wherein uploading the event aggregation data to the server comprises: In response to receiving a log upload request from the server, the event aggregation data is uploaded to the server, wherein the log upload request is sent by the server to the client when it receives a fault or defect reported by the user.

11. An apparatus for processing logs, comprising: The event classification module is configured to, in response to an event being logged and triggered, have the client classify the event into a target event category; The event acquisition module is configured to acquire multiple events with the target event category from the client in response to the determination that a log analysis task is triggered; The event aggregation module is configured to generate event aggregation data by the client based on the multiple events; as well as The event upload module is configured to upload the event aggregate data from the client to the server.

12. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 10.

13. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 10.

Images