Virtual Firewall Flow Record Scheduling Under Queue Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In virtualized environments with distributed firewalls, the management and collection of firewall flow records are inefficient due to resource constraints, leading to data loss and suboptimal analysis capabilities in software-defined networking (SDN) managers.
Innovation Solution
Implement a dynamic collection schedule for firewall flow records that adapts based on processing capacity, uses heuristic-based graceful degradation, and employs controlled lossy aggregation to prioritize data retention and coalesce identical records, ensuring efficient storage and analysis while optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If firewall flow records are collected from all data end nodes at high frequency, then analysis capability is improved, but resource constraints cause data loss
Solution Approach 1:
The system dynamically adapts the collection schedule based on processing capacity and queue status. When the collection queue is near capacity, the system reduces collection frequency or skips certain data end nodes, transforming a static collection approach into a dynamic one that responds to real-time resource conditions, thereby preventing data loss while maintaining analysis capability.
Solution Approach 2:
The system changes operational parameters (collection frequency, selected data end nodes) based on resource availability and queue status. By adjusting these parameters dynamically, the system optimizes the balance between collecting sufficient data for analysis and avoiding resource exhaustion that would cause data loss.
2Loss of information
If collection frequency is increased to reduce data loss, then data completeness is improved, but processing capacity is exceeded
Solution Approach 1:
The system uses feedback from queue status monitoring to adjust collection frequency. When the collection queue approaches capacity, the system receives feedback and automatically reduces collection frequency or skips data end nodes, preventing processing capacity overload while minimizing data loss through intelligent adaptation.
Solution Approach 2:
The system performs partial collection actions by selectively skipping certain data end nodes when resources are constrained, rather than attempting to collect from all nodes at maximum frequency. This partial action approach prevents processing overload while still maintaining acceptable data completeness through strategic sampling.
3Loss of information
If all firewall flow records are retained, then data completeness is improved, but storage requirements increase
Solution Approach 1:
The system implements graceful degradation by selectively discarding less critical flow record data when storage or processing capacity is constrained, while recovering and prioritizing retention of more valuable data. This selective discarding approach manages storage requirements while maintaining essential data completeness for analysis.
Data Source
AI summary
In a computer-implemented method for collecting firewall flow records, firewall flow records are received from a plurality of data end nodes of a virtualized infrastructure comprising a distributed firewall according to a collection schedule, wherein the collection schedule defines which data end nodes of the plurality of data end nodes from which firewall flow records are collected, a frequency of collection of firewall flow records from the data end nodes, and an amount of firewall flow records collected from the data end nodes. Firewall flow records received at a firewall flow record collection queue are processed, such that the received firewall flow records are prepared for storage at a flow record data store. The collection schedule is dynamically adapted based at least in part on the processing of the received firewall flow records, such that the firewall flow record collection queue is available for processing firewall flow records prior to receiving additional firewall flow records from the data end nodes.