Dynamic Graph Novelty Detection for Changing Streaming Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern anomaly detection techniques are limited in handling streaming data, non-numerical data, or data with a large or changing number of features, particularly in batch-processed data with small feature sets.
Innovation Solution
A graph-based novelty detection system that processes streaming data by constructing a dynamic graph model representing observed components, calculating a novelty score based on conditional probability, and incrementally updating the graph to handle categorical, numerical, and structural data without conversion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional batch-processed anomaly detection is used, then it can handle data with small feature sets, but it cannot handle streaming data, non-numerical data, or data with large or changing feature sets
Solution Approach 1:
The patent implements dynamic graph structures that adapt to streaming data in real-time. The graph model continuously evolves by adding new nodes and edges as new features are observed, allowing the system to handle changing feature sets without retraining. This dynamic adaptation enables the system to process streaming data, non-numerical categorical data, and varying feature dimensions while maintaining anomaly detection capability.
Solution Approach 2:
The graph-based novelty detection system serves multiple functions: it handles streaming data, processes non-numerical categorical data, accommodates varying feature sets, and performs anomaly detection. The unified graph structure universally represents all these different data types and processing requirements through nodes (representing features) and edges (representing relationships), eliminating the need for separate processing pipelines for different data types.
2Speed
If complex mathematical relationships are computed for anomaly detection, then anomaly scores can be calculated for batch data, but the system cannot process streaming data in real-time
Solution Approach 1:
The system pre-computes and stores the graph structure representing normal data patterns before anomaly detection is needed. During streaming data processing, the system only needs to traverse the pre-built graph and compare new observations against stored patterns, rather than computing complex mathematical relationships from scratch. This preliminary graph construction enables real-time processing while maintaining detection accuracy.
Solution Approach 2:
The patent replaces traditional mathematical computation-based anomaly detection with a graph traversal and pattern matching approach. Instead of computing complex statistical relationships in real-time, the system uses graph-based representations to encode normal patterns and performs simpler operations like node traversal, path matching, and frequency counting, which can be executed rapidly on streaming data while preserving detection precision.
3Reliability
If the system processes all observed components in streaming data, then it can detect novel anomalies, but the computational complexity increases with data volume
Solution Approach 1:
The system extracts only the essential features and patterns from streaming data by representing them as nodes and edges in a graph structure. Instead of processing all raw data components, the graph abstraction captures the critical relationships and frequencies needed for anomaly detection. This extraction reduces computational complexity while maintaining detection reliability by focusing on the most informative aspects of the data.
Solution Approach 2:
The patent transforms the data representation parameters from raw streaming values to graph-based structural parameters (node frequencies, path patterns, edge weights). This parameter transformation enables more efficient processing by converting continuous streaming data into discrete, countable graph elements that can be processed using simpler operations like incrementing counters and traversing predefined paths, thereby increasing throughput while maintaining anomaly detection capability.
Data Source
AI summary
A graph-based novelty detection system processes a stream of observations by storing a compressed representation of the observations in a graph. Each node of the graph represents a component of an observation and a count of a number of times that component has been observed. The system calculates a novelty score based, at least in part, on a conditional probability of one or more components of the observation.