Correlating Network and Container Traffic via Hash Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud platforms, correlating network level and application level traffic is challenging due to the use of dynamic or ephemeral IP addresses, which obscure the source or destination of network traffic, and typical network level traffic capture systems lack visibility into application or container level traffic, making it difficult to attribute network traffic to specific applications, containers, or hosts for threat detection, troubleshooting, and forensic investigations.
Innovation Solution
The solution involves correlating network level traffic data with container level traffic data by hashing parameters such as IP addresses, timestamps, and data transfer amounts, allowing identification of the originating application or container, and merging this information for enhanced visibility and analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If network level traffic capture is performed using traditional systems, then network traffic can be monitored at the network layer, but the originating application or container cannot be identified due to dynamic IP addresses and lack of application level visibility
Solution Approach 1:
The patent introduces an intermediary correlation system that receives both network level traffic data and application level traffic data, matches them using hashing algorithms on common parameters (IP addresses, timestamps, data transfer amounts), and produces correlated traffic information that identifies the originating application or container. This intermediary layer resolves the information loss without requiring complete system redesign.
Solution Approach 2:
The patent segments the traffic analysis into two distinct data streams: network level traffic capture and application level traffic capture. Each stream is processed independently and then correlated through hashing matching. This segmentation allows each component to specialize in its own data type while the correlation mechanism integrates them to solve the identification problem.
2Measurement precision
If application level traffic capture is implemented to identify originating applications, then application visibility is improved, but the system complexity increases due to the need for multi-level data collection and correlation
Solution Approach 1:
The patent creates hash copies of the original traffic data parameters (IP addresses, timestamps, data transfer amounts) from both network and application level captures. These hash copies serve as matching keys that preserve the essential identification information without requiring storage or processing of the complete original data sets, thus improving measurement precision while controlling system complexity.
Solution Approach 2:
The patent transforms the original traffic parameters into hashed values for correlation purposes. This parameter transformation maintains the uniqueness and matchability of the data while reducing the complexity of direct comparison. The hashing process converts complex multi-parameter correlation into simpler hash value matching.
3Adaptability or versatility
If dynamic IP addresses are used in cloud environments, then resource flexibility and scalability are improved, but the ability to trace and attribute network traffic to specific applications or containers deteriorates
Solution Approach 1:
The patent establishes a feedback mechanism where application level traffic data provides the missing attribution information that network level data alone cannot provide. By capturing application level data and correlating it with network level data through hashing, the system feeds back the originating application or container identification to the traffic analysis process, thereby restoring reliability to traffic source attribution while preserving the benefits of dynamic IP addressing.
Data Source
AI summary
Methods, systems, and devices supporting network and container level traffic analysis and correlation are described. An application server may receive network traffic data from a network-level data capture system and receive container-level application traffic data from a container-level data capture system. The application server may then hash the destination addresses, the time stamp information, and the data amount information from the network traffic data to create a first set of hash values and hash the destination addresses, the time stamp information, and the data amount information from the application traffic data to create a second set of hash values. The application server may then identify matching hash values from the first set of hash values and the second set of hash values and then merge into a data queue the corresponding network traffic with metadata associated with the corresponding application traffic data to create a merged data set.