Stateful Flow Identification for Proactive Issue Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The sheer volume and rapid changes in system-generated structured data from storage systems make it challenging for support teams to identify issues efficiently, as conventional methods are time-consuming and prone to errors, hindering effective analysis of stateful flows across large numbers of systems.
Innovation Solution
The technology processes structured data into stateful flows, combines similar flows, and determines unique flows using hash values and machine learning models to identify anomalous flows, enabling proactive issue detection and notification, thus reducing the need for extensive dataset examination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional manual analysis methods are used to examine system-generated structured data, then support teams can identify issues, but the process becomes extremely time-consuming and error-prone due to the sheer volume of data
Solution Approach 1:
The patent replaces manual mechanical analysis with automated machine learning models and algorithms. The system uses trained ML models to automatically analyze structured data, identify stateful flows, and detect anomalies, eliminating the need for human reviewers to manually examine vast volumes of data while maintaining or improving accuracy.
Solution Approach 2:
The patent introduces an intermediary automated analysis system that sits between the raw structured data and the support teams. This intermediary system processes data through multiple stages including flow identification, state machine generation, and anomaly detection, delivering pre-processed insights to support teams rather than raw data requiring manual analysis.
2Reliability
If support teams manually review large datasets to identify issues, then problems can be detected, but the process becomes impracticable and error-prone due to data volume
Solution Approach 1:
The patent segments the complex task of issue detection into distinct automated stages: data ingestion, flow identification using state machines, pattern recognition, anomaly detection, and result generation. Each stage is handled by specialized automated components, making the overall process reliable and operationally feasible without human intervention at the data analysis level.
Solution Approach 2:
The system enables self-service automated analysis where the platform independently processes structured data, identifies flows, detects anomalies, and generates findings without requiring human operational input. The machine learning models automatically adapt and improve through continuous learning from new data patterns.
3Adaptability or versatility
If conventional analysis methods are used on rapidly changing system data from new software versions, then issues can be identified, but the process becomes extremely time-consuming and not practicable
Solution Approach 1:
The patent implements dynamic adaptive systems where machine learning models continuously learn from new data patterns arising from software version changes. The state machine generators automatically adapt to new flow patterns, and the system re-trains on emerging data characteristics, maintaining high adaptability while processing data at automated speeds.
Solution Approach 2:
The system automatically adjusts analysis parameters and thresholds based on changing data characteristics from new software versions. The machine learning models modify their internal parameters and decision boundaries to accommodate evolving data patterns, maintaining productivity while adapting to change.
Data Source
AI summary
The described technology is generally directed towards processing structured data corresponding to system information such as alerts, logs, events, health check data and the like, to identify stateful flows from the structured data. Identical flows are combined, similar flows, based on similarity scores obtained from stateful flows are combined, and incremental stateful flows are combined. Neural networks can be used to identify the similar flows and incremental flows. A distribution can be obtained based on counts of the different stateful flows that remain after combining the identical, similar and incremental stateful flows; a neural network that accounts for subtle differences can be used to provide a more accurate distribution than simple counts. Anomalous stateful flows can be identified from the distribution, with some action taken for an anomalous stateful flow, e.g., to send a notification or other output to a support engineer of the like for proactive issue detection.