Log Data Classification System for Sparse Record Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing vast quantities of log data generated by computing systems and applications is inefficient due to the large size and sparsity of interesting records, making it difficult to identify rare events and hidden trends within the data.
Innovation Solution
A method for classifying and grouping log records based on similarity thresholds, using data subsetting and tokenization to optimize processing, allowing for efficient identification of sparse records and detection of anomalies, and applying analytics such as anomaly detection and forecasting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If log data is stored in its original format, then all information is preserved, but the data becomes difficult to analyze and process efficiently
Solution Approach 1:
The patent segments log data into distinct fields (timestamp, host, app, proc, message, etc.) and stores them in separate columns in a structured format. This segmentation allows efficient querying and analysis of specific fields while preserving all original information, resolving the contradiction between information preservation and analysis efficiency.
Solution Approach 2:
The patent transforms unstructured log data into a structured format with defined parameters and data types for each field. By changing the parameter structure from free-text to organized columns with specific data types, the system enables efficient processing while maintaining all information through proper field decomposition.
2Reliability
If all log records are analyzed, then complete coverage is achieved, but the time and resources required increase significantly
Solution Approach 1:
The patent extracts and stores metadata information (timestamp, host, application, process) separately from the main message content. This extraction allows the system to quickly filter and search based on metadata fields without processing the entire log content, achieving complete coverage through structured storage while reducing processing time through targeted queries.
3Ease of manufacture
If log data is stored in unstructured format, then storage is simpler, but searching and analyzing specific information becomes inefficient
Solution Approach 1:
The patent divides log entries into structured fields (timestamp, host, app, proc, message, etc.) that can be stored in organized columns. This segmentation maintains storage efficiency while dramatically improving information retrieval by enabling direct querying of specific fields without parsing entire log lines.
Solution Approach 2:
The structured log format serves multiple functions simultaneously: it preserves complete log information, enables efficient searching by any field, supports various analysis operations, and maintains storage compactness. The universal structured format resolves the contradiction between storage simplicity and retrieval efficiency.
Data Source
AI summary
Disclosed is a system, method, and computer program product for analyzing sets of data in an efficient manner, such that analytics can be effectively performed over that data. Classification operations can be performed to generate groups of similar log records. This permits classification of the log records in a cohesive and informative manner.