Log Identity Classification for Selective Log Data Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Centralized log management platforms face challenges in efficiently managing large volumes of log data due to varying value and format, leading to increased costs, storage requirements, and performance degradation.
Innovation Solution
A log management engine and agent that identify log entries' syntax to determine a log identity, enabling intelligent storage handling decisions such as blocking, sampling, and compression based on log identities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If log management platforms collect and store all log data from multiple sources, then log data availability and accessibility are improved, but storage costs and computational resources increase
Solution Approach 1:
The patent segments log data into different categories based on their syntax structure (e.g., JSON, CSV, XML formats) and handles each segment differently. By dividing the log data into distinct syntactic groups, the system can apply selective storage strategies - storing only necessary log entries in appropriate formats while reducing storage volume for redundant or less critical data, thus resolving the contradiction between data availability and storage requirements
Solution Approach 2:
The patent applies local quality by treating different log entries differently based on their specific syntax characteristics. Instead of uniform processing, the system identifies log entries with known versus unknown syntax and applies corresponding handling rules - such as storing only essential fields for complex formats or applying transformations only when necessary. This selective approach maintains data availability for critical logs while reducing overall storage volume
2Speed
If log management platforms process and analyze log data in real-time, then operational visibility and responsiveness are improved, but processing time and computational overhead increase
Solution Approach 1:
The patent performs preliminary action by pre-processing log data at the point of ingestion to identify and categorize syntax types before full analysis. The system quickly determines whether log entries have known or unknown syntax and applies preliminary handling rules, such as immediate storage of critical logs or deferred processing of complex formats. This preliminary classification reduces the time required for subsequent analysis and improves response time without significantly increasing processing overhead
Solution Approach 2:
The patent applies partial action by processing only the necessary portions of log data - specifically, the syntax identification and classification aspects - rather than fully analyzing every log entry in detail immediately. This selective processing approach maintains real-time responsiveness for critical operations while reducing overall computational overhead and processing time
3Loss of information
If log management platforms store all log data regardless of format, then data completeness is improved, but storage costs and bandwidth requirements increase
Solution Approach 1:
The patent extracts only the essential information needed for log management - specifically, the syntax identification and key data fields - while leaving out redundant or less critical data. By extracting only necessary log entries and fields based on their syntax characteristics, the system maintains data completeness for essential operations while significantly reducing bandwidth consumption and storage costs associated with transmitting and storing all log data regardless of format
Data Source
AI summary
A computing platform may be configured to (i) receive a log entry that was produced by a log producer, wherein the log entry comprises one or more data elements, (ii) produce a restructured representation of the log entry, the restructured representation comprising a sequence of one or more tokens that represent the one or more data elements of the log entry, (iii) based on the restructured representation of the log entry, determine a log identity of the log entry, and (iv) handle the log entry in accordance with a handling rule for the determined log identity.