Network Traffic Flow Analysis with Dynamic Threshold Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high volume of network traffic flows poses challenges for efficient near-real-time processing and analysis, requiring optimizations to enable interactive display and report generation in computer networks.
Innovation Solution
A method and system that determine a statistical traffic flow profile, set a traffic flow size threshold, and retain only flow records above this threshold, while discarding smaller ones, using a database structure with memory buckets and dynamic threshold adjustments to maintain analysis accuracy and reduce data volume.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all network traffic flow records are retained for analysis, then analysis accuracy is improved, but processing efficiency and data volume management deteriorate
Solution Approach 1:
The patent extracts and retains only the most significant flow records above a dynamically determined threshold, removing insignificant records that consume processing resources. This selective extraction maintains analysis accuracy by preserving flows that contribute meaningfully to traffic patterns while discarding noise from minimal flows.
Solution Approach 2:
The patent dynamically adjusts the flow size threshold parameter based on statistical analysis of the traffic distribution. By changing this parameter adaptively rather than using a fixed value, the system optimizes the balance between retaining sufficient data for accurate analysis and filtering out excessive volumes of insignificant flows that hinder processing efficiency.
2Productivity
If a fixed flow size threshold is used, then processing efficiency is improved, but adaptability to different traffic patterns deteriorates
Solution Approach 1:
The patent implements a dynamic threshold determination mechanism that continuously adapts to changing traffic patterns. The threshold is recalculated based on the statistical distribution of flow sizes in the current traffic stream, allowing the system to automatically adjust to different network conditions, applications, and time periods while maintaining processing efficiency.
Solution Approach 2:
The system uses feedback from statistical analysis of the traffic flow distribution to adjust the threshold parameter. By monitoring the distribution characteristics and using this information to refine the threshold, the system creates a closed-loop control mechanism that adapts to varying traffic patterns while preserving processing efficiency.
Data Source
AI summary
Network traffic flow records received from a network probe are recorded in multiple sets of buckets of different granularity, optimized for the purpose of almost instant analysis and display as well as for longer term report generation. The flow data is pre-processed and stored redundantly in parallel in multiple bucketized data base tables of different time window sizes. Denormalized tables keyed on different combinations of traffic flow attributes are precomputed and stored in parallel tables redundantly to facilitate a near real time display of summarized network traffic data, and a capability to rapidly generate reports for different monitoring periods.