Structured Log Labeling for Predictive Incident Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional log data analytic systems fail to efficiently capture and label useful log data for long-term storage, leading to retroactive analysis and excessive storage resource usage, which impacts the ability to audit past application behaviors and anticipate errors.
Innovation Solution
A method for managing structured log data involves retrieving raw log data, parsing it to discover key tokens, applying predetermined labels, generating structured log data, and using models to predict potential incidents based on timestamped and host-name-tagged data, enabling proactive error anticipation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional log data analytic systems store all raw log data indiscriminately, then complete audit capability is maintained, but storage resource consumption increases excessively
Solution Approach 1:
The system extracts only relevant information from raw log data by identifying key tokens and their associations with incidents. Instead of storing all raw log data, the system extracts key tokens, determines their relevance to incidents, and stores only this structured information in a condensed format, significantly reducing storage requirements while maintaining audit capability.
Solution Approach 2:
The system transforms raw log data into a different parameter representation by converting unstructured log entries into structured records with key tokens, incident associations, and metadata. This parameter transformation allows the system to store condensed information that retains essential audit capabilities while occupying minimal storage space.
2Quantity of substance
If old log data are periodically purged to free up space, then storage resources are optimized, but ability to audit applications past a point in time is impacted
Solution Approach 1:
The system creates a condensed copy of essential log information by extracting key tokens and their incident associations. This condensed representation serves as a sufficient copy for audit purposes, allowing the system to purge original raw log data while maintaining the ability to audit application behaviors through the structured stored information.
3Measurement precision
If all raw log data are retained for predictive analytics, then analysis completeness is improved, but storage costs increase
Solution Approach 1:
The system extracts only the essential elements needed for predictive analytics by identifying key tokens in log data and determining their associations with incidents. This extraction process captures the critical information patterns required for predictive analysis while discarding redundant data, achieving both analysis completeness and storage efficiency.
Solution Approach 2:
The system transforms raw log data into a condensed parameter representation that includes key tokens, incident associations, timestamps, and host information. This parameter transformation maintains the analytical value needed for predictive analytics while reducing storage requirements by orders of magnitude.
4Quantity of substance
If conventional systems examine log data only after incidents occur, then storage resources are conserved by not retaining historical data, but predictive capability is lost
Solution Approach 1:
The system performs preliminary structuring and labeling of log data as it is generated, creating a condensed structured format with key tokens and incident associations in advance. This preliminary action enables future predictive analytics by having the data ready in an analyzable format, allowing the system to predict potential incidents before they occur while maintaining efficient storage.
Data Source
AI summary
A method for providing predictive analytics based on structured log data is disclosed. The method includes retrieving, via a network interface, raw log data from a data storage component, the raw log data including application information; parsing the raw log data to discover a key token, the key token corresponding to an application; identifying a predetermined label that corresponds to the key token; generating structured log data based on the key token and the predetermined label; retrieving incident data corresponding to the application; and predicting, by using a model, a potential incident based on the structured log data and the incident data.