Machine Learning Alert Prioritization for IT Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume and diversity of machine data in IT environments pose challenges in efficiently analyzing and searching massive quantities of data, as conventional systems discard minimally processed data during pre-processing, limiting the ability to derive insights and requiring administrators to sift through numerous alerts for actionable information.
Innovation Solution
An event-based data intake and query system with a late-binding schema allows for flexible data processing and storage of minimally processed machine data, enabling users to extract and analyze all data at search time, and an alert prioritization subsystem using machine learning models to prioritize alerts based on user feedback and behavior.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If pre-processing data by extracting specified data items and discarding the remainder, then data retrieval and analysis efficiency is improved, but data flexibility and completeness are worsened
Solution Approach 1:
The system performs preliminary actions by extracting and storing only the most critical data items (such as host, source, sourcetype, and message) during data ingestion, while retaining the ability to access the full original data through the data lake. This selective pre-processing improves retrieval efficiency for common queries while maintaining flexibility to access complete data when needed.
Solution Approach 2:
The patent segments data into two storage locations: a data lake for storing full raw data and a processed data store for storing extracted key fields. This segmentation allows the system to serve different query patterns efficiently - using the processed store for quick retrievals of common fields and the data lake for complete data analysis, thus resolving the contradiction between efficiency and flexibility.
2Adaptability or versatility
If storing massive quantities of raw data for later analysis, then data flexibility and analysis comprehensiveness are improved, but system complexity and storage management difficulty are worsened
Solution Approach 1:
The system performs preliminary extraction and transformation of data into a standardized format during the ingestion phase, creating a processed data store that simplifies later queries. This preliminary action reduces the complexity of managing massive raw data by pre-organizing it into meaningful structures while preserving the original data in the data lake for comprehensive analysis when needed.
3Device complexity
If providing alerts in randomized or time-sequential manner, then system simplicity is maintained, but alert value and administrator productivity are worsened due to time required to parse through alerts
Solution Approach 1:
The system implements feedback mechanisms where administrators can mark alerts as false positives or adjust priorities, and this feedback is used to refine future alert prioritization. The machine learning model continuously learns from user interactions to improve alert relevance over time, significantly reducing the time administrators spend parsing through alerts while maintaining system simplicity.
Solution Approach 2:
The alert prioritization system performs self-service by automatically learning from user feedback and adjusting its own prioritization logic without requiring manual reconfiguration. The machine learning model autonomously optimizes alert ordering based on administrator behavior patterns, improving productivity while keeping the system operationally simple.
Data Source
AI summary
Systems and methods are disclosed that are directed to improving the prioritization, display, and viewing of system alerts through the use of machine learning techniques to group the alerts and further to prioritize the groupings. Additionally, a graphical user interface is generated that illustrates the prioritized listing of the plurality of groupings. Thus, a system administrator or other user receives an improved experience as the number of notifications provided to the system administrator are reduced due to the grouping of individual alerts into related groupings and further due to the prioritization of the groupings. Previously, or in current technology, system alerts may be automatically generated and provided immediately to a system administrator. In some instances, any advantage of detecting system errors or system monitoring provided by the alerts is negated by the vast number of alerts and provision of minimally important alerts in a manner that concealed more important alerts.