Unified Event Stream Storage for Real-Time Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently managing and analyzing event streams for anomaly detection across diverse compute assets in cloud and non-cloud environments, particularly in terms of data ingestion, processing, and unified storage, which hinders real-time anomaly detection and response.
Innovation Solution
A data platform is implemented to ingest, process, and store event data from compute assets using agents that collect and report information, with data processing resources performing real-time anomaly detection and user interface resources providing results, while utilizing a unified storage system for event streams.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If event streams are stored in multiple separate databases for different compute assets, then each database can be optimized for its specific asset type, but the system complexity increases and unified anomaly detection becomes difficult
Solution Approach 1:
The patent merges event streams from multiple compute assets into a single unified storage system. The storage system receives event streams from various compute assets (virtual machines, containers, physical servers) and stores them in a centralized location, enabling unified anomaly detection across all assets without requiring separate optimized databases for each asset type.
Solution Approach 2:
The unified storage system is designed to handle multiple types of event streams from different compute asset types using a single storage mechanism. The system can store, manage, and query event data from virtual machines, containers, and physical servers uniformly, eliminating the need for multiple specialized storage systems while maintaining detection accuracy across diverse environments.
2Speed
If real-time processing of event streams is implemented, then anomaly detection speed improves, but data processing resources and computational overhead increase
Solution Approach 1:
The system performs preliminary actions by storing event streams in a unified structure that is pre-organized for efficient querying and analysis. The event streams are normalized and stored with appropriate indexing before anomaly detection is needed, so that when anomalies are detected, the data is already in the optimal format for rapid processing, reducing the computational overhead during real-time analysis.
Solution Approach 2:
The unified storage system acts as an intermediary between event stream ingestion and anomaly detection processing. It receives raw event streams from various compute assets, normalizes and stores them in a standardized format, and provides efficiently queryable data to the anomaly detection system, thereby reducing the processing burden on real-time analysis while maintaining fast detection speeds.
3Reliability
If comprehensive monitoring of all compute assets is implemented, then security and compliance coverage improve, but data ingestion and processing volume increases
Solution Approach 1:
The system segments the monitoring function by compute asset type (virtual machines, containers, physical servers) while maintaining a unified storage structure. Each asset type's event streams are collected separately during ingestion but are stored in a standardized unified format, allowing comprehensive coverage of all asset types without creating separate storage systems that would increase overall data volume management complexity.
Data Source
AI summary
Providing unified storage for event streams in an anomaly detection framework, including: receiving, by an event streaming platform, first event data encoded in a first file format; converting, by the event streaming platform, the first event data into second event data encoded in a second file format for storage in a first remote storage system; and providing, to the first remote storage system, a command to load the second event data into the first remote storage system.