Cloud Data Ingestion Platform for Real-Time Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data analytics platforms face challenges in efficiently monitoring and detecting anomalies within cloud environments, particularly in datacenters, due to the complexity of network activities and the need for real-time data processing and security monitoring.
Innovation Solution
A data platform is configured to ingest data from cloud environments, process it in real-time, and generate polygraphs to model normal behaviors, allowing for the detection of deviations and anomalies through the use of agents, data aggregators, and microservices that collect and analyze network and process-level information, enabling continuous monitoring and threat detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is collected and processed in real-time from cloud environments, then anomaly detection capability is improved, but system complexity increases
Solution Approach 1:
The system divides the data processing function into separate components: agents deployed at data sources collect local data, data aggregators receive and pre-process data from multiple agents, and polygraph generators create behavior models. This segmentation allows each component to handle specific tasks independently, reducing overall system complexity while maintaining real-time anomaly detection capability.
Solution Approach 2:
Data aggregators act as intermediaries between agents and polygraph generators, buffering and pre-processing data before it reaches the model generation component. This intermediary layer decouples the real-time data collection from the computational model generation, allowing the system to handle complexity more effectively.
2Measurement precision
If polygraphs are generated to model normal behaviors, then detection precision is improved, but processing time increases
Solution Approach 1:
The system performs preliminary data collection and aggregation by agents before polygraph generation is needed. Agents continuously gather network and process data locally, so when anomaly detection is required, the data is already prepared and waiting, reducing the time needed to generate polygraphs and detect anomalies.
Solution Approach 2:
Agents operate continuously, constantly collecting and aggregating data from cloud environments. This continuous operation ensures that data is always available when polygraph generation is triggered, eliminating idle processing time and maintaining steady detection precision without time losses.
3Adaptability or versatility
If agents collect network and process-level information, then monitoring coverage is improved, but data volume increases
Solution Approach 1:
Agents selectively extract only the necessary data elements from cloud environments - specifically network connections and process information - rather than collecting all possible data. This targeted extraction maintains comprehensive monitoring coverage of critical security-relevant activities while significantly reducing the overall data volume that needs processing.
Solution Approach 2:
Each agent tailors its data collection to the specific characteristics of its local environment, focusing on network and process data that are most relevant for anomaly detection at that location. This localized approach ensures monitoring coverage is optimized for each environment while minimizing unnecessary data collection.
Data Source
AI summary
An example method includes accessing, by a data platform via a network, data from one or more cloud environments; identifying, by the data platform and in the data, first data associated with a first entity and a first data type and second data associated with a second entity and a second data type; mapping, by the data platform and based on the first entity and the first data type, the first data to a first data stream of a data streaming platform; mapping, by the data platform and based on the second entity and the second data type, the second data to a second data stream of the data streaming platform, the second data stream different from the first data stream of the data streaming platform; and generating, based on the first data stream, a graph representing activity associated with the first entity in the one or more cloud environments.