Edge Data Platform Offline Workflow Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data analytics platforms face challenges in efficiently monitoring and detecting anomalies within cloud environments, particularly in datacenters, due to the complexity of network activities and the need for real-time data processing and security monitoring.
Innovation Solution
A data platform is configured to ingest data from cloud environments, process it for anomaly detection, and provide real-time insights through a polygraph model that aggregates and analyzes network activities, using agents deployed on compute assets to collect and report information, and a data aggregator to minimize security exposure and optimize data transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is collected and processed in real-time from cloud environments, then anomaly detection capability is improved, but system complexity and data transmission overhead increase
Solution Approach 1:
The system segments data processing by deploying agents on individual compute assets to perform local data collection and preliminary processing, separating this function from the central data aggregator. This segmentation reduces the complexity burden on the central system while maintaining real-time anomaly detection capability through distributed processing.
Solution Approach 2:
Agents act as intermediary components between compute assets and the data aggregator. These agents perform local data collection, filtering, and preliminary analysis, then transmit only relevant information to the data aggregator. This intermediary layer simplifies the overall system architecture by distributing complexity while enabling real-time anomaly detection.
2Object-affected harmful factors
If data transmission is optimized to minimize security exposure, then security is improved, but data processing speed may be reduced
Solution Approach 1:
The system extracts and processes sensitive data locally at the agent level before transmission to the data aggregator. By performing data filtering, aggregation, and preliminary analysis locally, the system minimizes the amount of data transmitted over networks, thereby reducing security exposure surface area while maintaining processing speed through efficient local operations.
Solution Approach 2:
Processing operations are performed locally at each agent with customized data collection and filtering logic adapted to specific compute asset environments. This local quality approach enables security-optimized data transmission by processing only necessary data locally, reducing transmission volume and exposure risk while maintaining processing efficiency.
3Measurement precision
If polygraph model aggregates and analyzes network activities, then anomaly detection accuracy is improved, but computational resources required increase
Solution Approach 1:
The polygraph model analysis is segmented and performed locally at agents rather than requiring all data to be processed centrally. Each agent maintains a local polygraph model for its compute asset, performing anomaly detection locally with reduced computational resource requirements, while the data aggregator maintains a global polygraph model for cross-asset analysis.
Solution Approach 2:
The system performs partial polygraph analysis at the agent level for immediate anomaly detection, reserving more comprehensive global analysis for the data aggregator. This partial action approach enables accurate local anomaly detection with reduced computational resource consumption, while the data aggregator performs extensive global correlation analysis only when necessary.
Data Source
AI summary
Methods, systems, and products for offline workflows in an edge-based data platform, including: accessing log data describing activity associated with a user; generating, based on the log data, one or more alerts; and initiating, based on the one or more alerts, a workflow to acknowledge the one or more alerts by the user.