Flow Aggregation Keys for Host Data Collection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Software-defined data centers (SDDCs) face challenges in analyzing fragmented data, making it difficult for users to assess and visualize their security posture effectively.
Innovation Solution
A method is introduced that collects and reports attributes of data flows from machines executing on host computers, utilizing a logical network managed by a virtualization manager, and processes this data through a policy, analytics, and correlation engine appliance for analysis and visualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If data is collected from multiple sources including flow exporters, guest introspection agents, and third-party programs, then the comprehensiveness of security analysis is improved, but the system complexity increases
Solution Approach 1:
The system divides data collection into separate modular components: flow exporters for network flow data, guest introspection agents for VM-level data, and third-party program interfaces for external security data. Each component operates independently and publishes to standardized topics, allowing comprehensive data collection while maintaining manageable system complexity through clear separation of concerns.
Solution Approach 2:
The anomaly detection engine serves multiple functions by processing data from diverse sources (flow data, context data, configuration data) through a unified analysis pipeline. It can detect various types of anomalies including security threats, performance issues, and policy violations, making the system versatile without requiring separate specialized components for each detection type.
2Measurement precision
If flow data is aggregated using multiple keys specifying different attributes, then the precision of security analysis is improved, but the processing time increases
Solution Approach 1:
The system performs preliminary aggregation of flow data by key attributes (such as source/destination IP, port, protocol) before detailed anomaly detection. Flow exporters pre-process and publish aggregated flow records to Kafka topics, reducing the volume of raw data that requires detailed analysis later and enabling faster detection while maintaining precision through pre-computed aggregates.
Solution Approach 2:
The anomaly detection engine operates periodically, processing batches of aggregated flow data at scheduled intervals rather than continuously analyzing every individual packet. This periodic batch processing approach balances analysis precision with processing time, allowing comprehensive multi-key aggregation while avoiding real-time performance degradation.
3Measurement precision
If all flow data is published to the analysis appliance, then the accuracy of security posture assessment is improved, but the data transmission volume increases
Solution Approach 1:
The system extracts and publishes only the most relevant flow attributes to Kafka topics rather than transmitting complete raw flow records. Flow exporters extract key attributes (source/destination addresses, ports, protocols, packet/byte counts) and publish these extracted features, reducing transmission volume while maintaining sufficient data for accurate security assessment by the anomaly detection engine.
Data Source
AI summary
Some embodiments provide a novel method for collecting and reporting attributes of data flows associated with machines executing on a plurality of host computers to an analysis appliance. Each host computer, in some embodiments, is responsible for collecting and reporting attributes of data flows associated with machines executing on a host computer. The host computer, in some embodiments, first eliminates duplicative flow group records and then aggregates the flow data according to a set of received keys that specify attributes that define the aggregation. For example, a simple key that specifies a set of machine identifiers (e.g., a VM ID) as attribute values will, for each machine identifier, aggregate all flows with that machine identifier into a single aggregated flow group record. In some embodiments, the host computer includes a flow exporter that processes and publishes flow data to the analysis appliance.