Big Data Anomaly Detection via Feature Permutation Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In enterprise-wide data processing systems, the vast amounts of data from server event loggings, known as 'Big Data,' are challenging to analyze for meaningful and actionable information due to their volume, varying formats, and duplicative records, making it difficult for human administrators to identify anomalies in real-time, which can lead to missed or delayed detection of system faults and failures.
Innovation Solution
A Big Data mining and reporting system that automatically identifies relevant features for anomaly detection by using an expert knowledge database and statistical clustering analysis, such as Kernel Density Estimation, to distinguish between duplicative and non-duplicative records, and to determine the underlying causes and relations of anomalies, thereby providing actionable alerts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all event records are monitored and analyzed, then complete anomaly detection coverage is achieved, but the system complexity and processing difficulty increase significantly
Solution Approach 1:
The patent segments the event data stream into multiple channels based on event types, sources, and characteristics. Each channel is processed independently with specialized filters and anomaly detection rules, dividing the complex monolithic processing task into manageable segments that can be handled with simpler, more efficient algorithms.
Solution Approach 2:
The patent extracts and separates relevant anomaly-indicating features from the voluminous event data using filtering mechanisms. By taking out only the critical information needed for anomaly detection and discarding redundant data, the system achieves comprehensive coverage without processing the entire data set, thereby reducing complexity.
2Measurement precision
If all fields in event records are analyzed, then complete feature information is obtained, but the processing time and computational resources increase
Solution Approach 1:
The patent applies local quality by analyzing different fields with different levels of detail based on their relevance to anomaly detection. Critical fields such as error codes and exception types are analyzed in depth, while less relevant fields are processed with simpler methods or skipped entirely, optimizing the balance between information completeness and processing speed.
Solution Approach 2:
The patent implements partial action by selectively analyzing only the necessary subset of fields rather than processing all fields uniformly. The system identifies and focuses on the minimum essential fields required for effective anomaly detection, avoiding the time cost of analyzing redundant information while maintaining detection precision.
3Loss of information
If duplicate anomaly records are processed individually, then each anomaly is fully documented, but redundant information increases alert volume and reduces actionable insights
Solution Approach 1:
The patent merges duplicate or highly similar anomaly records into consolidated alert groups. By combining multiple records that represent the same underlying issue, the system preserves complete anomaly information within each group while presenting a reduced number of actionable alerts to administrators, eliminating redundancy without losing critical details.
Solution Approach 2:
The patent creates representative copies or summaries of duplicate anomaly patterns rather than processing each individual record separately. A single representative record or aggregated summary captures the essential characteristics of multiple similar anomalies, maintaining information completeness while significantly reducing the volume of alerts requiring human attention.
4Adaptability or versatility
If data from multiple sources with varying formats is integrated, then comprehensive system monitoring is achieved, but data consistency and processing difficulty increase
Solution Approach 1:
The patent implements a universal data processing framework that can handle multiple data sources and formats through a common interface and standardized processing pipeline. The system employs format normalization and adapter patterns that allow diverse event sources to be processed uniformly, achieving comprehensive multi-source monitoring without proportionally increasing processing complexity.
Data Source
AI summary
A big data processing system includes a features permutations testing function that separates out from among a set of identified compound features, those compound feature permutations that have better capabilities for distinguishing between anomalies observed in respective multi-dimensional feature spaces having as their axes the features of the identified compound features.