Network Flow Record Archiving for Loss-Free Export and Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies struggle to efficiently manage network flow information records in large data centers, often leading to data loss due to processing limitations, which is unacceptable for environments requiring data integrity for auditing and other purposes.
Innovation Solution
Network devices operate in an archiving mode to store records locally and upload them to remote storage at scheduled intervals, while also enabling a streaming mode to prioritize real-time availability, ensuring data integrity and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If network devices export records at high rate to monitor network performance, then productivity is improved, but data integrity deteriorates due to record loss
Solution Approach 1:
The system segments the record export process into two independent paths: a streaming path for real-time monitoring and an archiving path for data integrity. The archiving path includes a local buffer store and remote storage, separating the high-speed export function from the data integrity function, allowing each to optimize independently without compromising the other.
Solution Approach 2:
The patent introduces a local buffer store as an intermediary between the network device and remote storage. This buffer acts as a mediator that receives records at high speed from the network device and ensures their safe transfer to remote storage, preventing record loss while maintaining the high export rate capability.
2Loss of time
If network monitoring appliance processes records in real-time, then response time is improved, but processing capacity is exceeded leading to record drop
Solution Approach 1:
The system segments the record processing function by providing a dedicated archiving path that operates independently from the real-time monitoring path. The archiving path handles bulk record transfer to remote storage without competing for processing capacity with real-time monitoring functions, allowing both high-speed processing and complete record capture.
3Reliability
If network devices buffer records locally before exporting, then data integrity is improved, but storage requirements increase
Solution Approach 1:
The local buffer store serves as a temporary intermediary that holds records only briefly before they are transferred to remote storage. This buffer is designed for high-speed temporary storage rather than long-term retention, minimizing the volume of data stored locally at any given time while ensuring data integrity during the export process.
Solution Approach 2:
The system maintains continuous transfer of records from the local buffer to remote storage, ensuring that the buffer is constantly being emptied. This continuous action prevents the buffer from accumulating large volumes of data, reducing storage requirements while maintaining the ability to buffer records temporarily for integrity purposes.
Data Source
AI summary
A network device exports network flow information records. The network device obtains network traffic data sent to the network device. The network device generates network flow information records characterizing the network traffic data and including information relating to firewall policies applied by the network device. The network device stores the network flow information records to a local storage of the network device. At a scheduled refresh time, the network device uploads a set of the network flow information records over a network to a remote network storage for archiving in a database accessible to a network monitoring appliance. The network device may include an optional streaming mode. While the streaming mode is enabled, the network device streams the network flow information records to the network monitoring appliance over a network.