State-Based API Anomaly Detection for Scalable Real-Time Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for detecting anomalous data in real-time data streams require large memory storage, are difficult to scale, and struggle with real-time updates, especially in complex, high-volume environments, often relying on discrete queries and third-party tools.
Innovation Solution
A mechanism that filters and analyzes real-time data streams using a stored state to detect anomalies, allowing for efficient detection and remediation without storing the entire data set, and is scalable.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the complete data set is stored in a database for later review, then data can be queried using structured searches, but a large amount of memory is required to store the complete data set
Solution Approach 1:
The patent extracts only the essential state information from the complete data set rather than storing all raw data. By maintaining a condensed state representation that captures only the necessary information for anomaly detection, the system achieves reliable querying capability with minimal memory storage requirements.
Solution Approach 2:
The patent segments the data processing into two distinct phases: real-time state modification based on incoming data streams, and subsequent anomaly detection queries based on state changes. This segmentation allows the system to maintain only essential state information while preserving the ability to perform comprehensive anomaly detection.
2Measurement precision
If discrete queries are made for each condition to detect anomalies, then specific conditions can be searched, but the update rate is limited by the number of queries that must be run for each update
Solution Approach 1:
The patent performs preliminary action by continuously maintaining an updated state representation as data arrives, rather than waiting to accumulate data before analysis. The state is proactively modified with each incoming data point, enabling real-time anomaly detection without requiring discrete batch queries, thus achieving both high precision and high update rates.
Solution Approach 2:
The patent ensures continuity of useful action by continuously modifying the state as data streams arrive and continuously monitoring for state changes that indicate anomalies. This continuous process eliminates the need for discrete query interruptions, maintaining both detection accuracy and high update throughput.
3Reliability
If the system processes complex, high-volume data streams with multiple query conditions, then comprehensive anomaly detection is achieved, but real-time updating becomes infeasible
Solution Approach 1:
The patent segments the complex anomaly detection task into continuous state modification operations and state change detection operations. By separating data ingestion from analysis, the system can process high-volume data streams in real-time while maintaining comprehensive anomaly detection capabilities through the state representation.
Solution Approach 2:
The patent changes the fundamental parameter from storing complete data sets to maintaining a condensed state representation. This parameter change enables real-time processing of complex, high-volume data streams while preserving comprehensive anomaly detection capability, as the state captures only the essential information needed for detection.
Data Source
AI summary
Media, methods, and systems are disclosed for detecting anomalous API calls within a real-time data flow. A real-time data stream is received. The real-time data stream is filtered. A stored state is modified based on a log entry data element within the filtered real-time data stream. An API call associated with the modified stored state is detected to be anomalous. Finally, a remediation action is taken for the API call.