Event Clustering System for Infrastructure Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for managing and organizing vast amounts of digital information, such as emails and messages, face challenges in effectively clustering events for anomaly detection and monitoring, particularly due to the complexity of spam detection and the difficulty in extracting application dependencies in data centers, leading to inefficiencies in network traffic management and productivity.
Innovation Solution
An agent technology system with a statistical analytical engine and monitoring policy that clusters events by determining common characteristics and producing actionable clusters, using techniques like Shannon entropy, NMF decomposition, and k-means clustering to identify failures or errors in managed infrastructure, and includes a signalizer engine to generate alerts and optimize event processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional folder-based systems are used to organize digital information, then information can be stored in a structured manner, but retrieval efficiency decreases and user productivity deteriorates due to manual organization requirements and inability to handle massive information volumes
Solution Approach 1:
The system performs preliminary automated clustering of events and information into meaningful groups before user retrieval is needed. The clustering engine pre-processes incoming information streams, organizing them into clusters based on similarity metrics, so that when users need information, it is already organized and easily accessible without manual intervention.
Solution Approach 2:
The system creates multiple views and representations of the same information cluster, allowing users to access the same clustered information through different interfaces and perspectives. This copying mechanism enables efficient retrieval without duplicating the underlying organization work, as the same clustered data can be viewed through multiple access points.
2Productivity
If automated clustering techniques are implemented to organize information, then information organization efficiency improves, but system complexity increases due to the need for sophisticated algorithms and processing resources
Solution Approach 1:
The clustering system is divided into separate functional modules: event extraction components, similarity calculation modules, clustering algorithm engines, and visualization interfaces. Each module handles a specific aspect of the clustering process independently, reducing overall system complexity while maintaining high organization efficiency. This segmentation allows each component to be optimized separately and makes the system more maintainable.
3Reliability
If comprehensive monitoring of all digital information flows is performed, then anomaly detection capability improves, but processing overhead and system resource consumption increase
Solution Approach 1:
The monitoring system applies different levels of analysis to different information streams based on their characteristics and importance. High-priority or anomaly-prone streams receive intensive monitoring with detailed clustering and analysis, while routine streams receive lighter processing. This local quality approach ensures reliable anomaly detection where needed while conserving processing resources on less critical data flows.
Data Source
AI summary
A system is provided for clustering events. A first engine is configured to receive message data from a managed infrastructure that includes managed infrastructure physical hardware that supports the flow and processing of information, The at least one engine is configured to determine common characteristics of events and produce clusters of events relating to the failure of errors in the managed infrastructure. Membership in a cluster indicates a common factor of the events that is a failure or an actionable problem in a physical hardware of the managed infrastructure directed to supporting the flow and processing of information. The first engine is configured to create one or more situations that is a collection of one or more events or alerts representative of the actionable problem in the managed infrastructure. A second engine is configured to determine one or more common steps from events and produces clusters relating to events. The second engine determines one or more common characteristics of events and producing clusters of events relating to the failure or errors in the managed infrastructure. A statistical analytical engine is included.