Flow-Level Anomaly Detection via ML Rule Translation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing network security systems face challenges in scaling packet inspection across large networks due to computational bandwidth constraints and high licensing costs, especially when dealing with complex queries and high-speed interfaces, making it impractical to monitor every network packet effectively.
Innovation Solution
A machine learning-based architecture that translates packet-level rules into flow-level rules, leveraging correlations between packet payload and flow header information to detect unwanted traffic, allowing for real-time or near real-time alerts without the need for exhaustive packet inspection, using flow statistics like NetFlow and ML algorithms to approximate packet signatures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If packet inspection is performed at network edge with many monitoring interfaces, then detection coverage is improved, but computational bandwidth constraints are exceeded
Solution Approach 1:
The system segments packet-level inspection into flow-level processing by grouping packets into flows based on 5-tuple characteristics. Instead of inspecting every packet individually at multiple interfaces, the system aggregates packet data into flow records and processes them at a lower computational level, reducing the computational burden while maintaining detection coverage.
Solution Approach 2:
The patent introduces flow records as an intermediary data structure between packet inspection and detection rules. Flow records aggregate packet information and serve as the unit for rule evaluation, allowing the system to maintain comprehensive monitoring while reducing computational complexity through this intermediate representation.
2Measurement precision
If complex queries are executed for comprehensive packet inspection, then detection accuracy is improved, but computational bandwidth constraints are hit
Solution Approach 1:
The system changes the fundamental parameter of inspection granularity from packet-level to flow-level. By operating on flow records instead of individual packets, the system can execute complex detection queries with reduced computational cost, as flow records aggregate multiple packet characteristics into a single processing unit.
3Ease of operation
If packet inspection is deployed as router feature, then detection capability is integrated, but licensing costs increase
Solution Approach 1:
The patent creates a simplified copy of packet inspection functionality that operates at flow level rather than requiring full packet inspection capabilities. This flow-based approach can be implemented with lower-cost router features or software solutions, avoiding the need for expensive proprietary packet inspection licenses while maintaining detection functionality.
Data Source
AI summary
A system to detect anomalies in internet protocol (IP) flows uses a set of machine-learning (ML) rules that can be applied in real time at the IP flow level. A communication network has a large number of routers equipped with flow monitoring capability. A flow collector collects flow data from the routers throughout the communication network and provides them to a flow classifier. At the same time, a limited number of locations in the network monitor data packets and generate alerts based on packet data properties. The packet alerts and the flow data are provided to a machine learning system that detects correlations between the packet-based alerts and the flow data to thereby generate a series of flow-level alerts. These rules are provided to the flow time classifier. Over time, the new packet alerts and flow data are used to provide updated rules generated by the machine learning system.