Log File Anomaly Detection via Frequency Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analyzing large volumes of data from log-files in servers, such as those from online gaming communities, is impractical for manual detection of anomalies and requires an automated, resource-efficient method to process and identify anomalies effectively.
Innovation Solution
A computer-implemented method that processes data to provide frequency information for combinations of data, compares this information with a reference set, and identifies anomalies based on differences exceeding a threshold, using hash tables to store and compare occurrence counts across log files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual analysis of log-file data is performed, then detailed anomaly detection is possible, but it becomes highly impractical when dealing with thousands to millions of events per day
Solution Approach 1:
The patent replaces manual mechanical analysis with automated computer-based processing. The system uses processors to automatically parse log files, extract data combinations, generate frequency information, and compare against reference data, eliminating the need for human operators to manually examine thousands to millions of events while maintaining anomaly detection capability
Solution Approach 2:
The patent creates frequency information copies of the actual log data that can be efficiently processed and compared. By generating frequency information that represents the statistical distribution of data combinations, the system enables automated analysis without requiring direct manual examination of the raw voluminous log data
2Measurement precision
If comprehensive automated analysis of all log data is performed, then anomaly detection capability is improved, but processing loads and memory requirements increase significantly
Solution Approach 1:
The patent extracts only the essential frequency information from the complete log data set. Instead of processing all raw log events, the system extracts data combinations, counts their occurrences, and stores only these frequency statistics. This extraction reduces the data volume significantly while preserving the information needed for anomaly detection
Solution Approach 2:
The patent transforms the raw log data into frequency information, changing the parameter representation from individual events to aggregated counts. This parameter transformation reduces the complexity and volume of data that needs to be processed, allowing automated analysis with reduced processing loads and memory requirements
3Measurement precision
If frequency information for all data combinations is stored, then complete anomaly detection is achieved, but memory requirements increase
Solution Approach 1:
The patent merges multiple occurrences of the same data combination into a single frequency count. Instead of storing each individual log event, the system combines identical data combinations and stores only their aggregated frequency information. This merging significantly reduces the quantity of data stored while maintaining the ability to detect anomalies through frequency comparison
Data Source
AI summary
A method for determining one or more anomalies associated with a first set of data, said method comprising: processing said first set of data to provide frequency information for a plurality of combinations of data in said first set of data; comparing said frequency information for the first set of data with frequency information for said combinations of data in a reference set of data; and in dependence on said comparison determining one or more anomalies.