Network Flow Record Caching and Selective Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As networks grow in size and complexity, traditional methods for analyzing network traffic flow data face bottlenecks due to the large volume of data generated, leading to inefficiencies in storage, bandwidth usage, and processing power, with existing solutions often resorting to dropping data that cannot be handled.
Innovation Solution
A method and system that cache network flow records temporarily and selectively forward only the records representing the largest data transfers to permanent storage, prioritizing the storage of flows that account for the majority of network traffic, thereby reducing storage, bandwidth, and processing requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all flow records are stored in permanent storage, then complete network traffic data is preserved, but storage requirements and processing overhead increase significantly
Solution Approach 1:
The patent extracts only the most significant flow records (those representing the largest data transfers) from the complete set of flow records. By identifying and extracting top flows based on byte volume, the system retains the most important network traffic information while discarding less significant records, thereby reducing storage requirements while maintaining data value.
Solution Approach 2:
The patent changes the selection parameter from treating all flow records equally to selecting based on byte volume threshold. By using a parameter (byte count) to differentiate and select only flows exceeding a certain threshold, the system transforms the storage approach from comprehensive to selective, reducing storage quantity while preserving information about significant traffic patterns.
2Productivity
If flow records are cached temporarily and selectively forwarded, then storage and processing requirements are reduced, but some network traffic information is lost
Solution Approach 1:
The patent applies local quality by treating different flow records differently based on their byte volume characteristics. Instead of uniform handling, flows are categorized into significant (large byte volume) and insignificant (small byte volume), with only the significant ones being forwarded to permanent storage. This localized differentiation optimizes the balance between information retention and system productivity.
3Quantity of substance
If traditional hardware improvements are made to handle larger datasets, then more flow records can be stored and processed, but system complexity and cost increase
Solution Approach 1:
The patent applies partial action by storing only a portion (the most significant) of the available flow records rather than attempting to store and process all records. This partial approach achieves sufficient data handling capacity for meaningful analysis without requiring hardware improvements, as the selected subset contains the majority of network traffic information.
Data Source
AI summary
Embodiments of the invention include an apparatus, method, and computer program for collecting and analyzing network traffic flow records. The method includes, in one example, receiving a plurality of flow records which include data about traffic in a network. The method may further include caching the plurality of flow records in temporary storage, analyzing the plurality of flow records to determine a number of bytes of traffic that each of the plurality of flow records represents, and selecting a subset of the plurality of flow records to forward to permanent storage based on the determined number of bytes of each of the flow records.