Network Recorder Packet Truncation by Entropy and Header Value
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer network data packet storage devices face limitations in retention time due to finite storage capacity, requiring a balance between data retention and storage space usage, especially when dealing with high entropy packets like encrypted and compressed data which have low utility value.
Innovation Solution
Implementing entropy-based packet truncation methods that categorize data packets based on their entropy values, allowing for the truncation of high entropy packets while preserving low entropy packets, thereby optimizing storage space usage and increasing retention time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full and complete capture of network packets is performed, then data completeness for incident investigation is improved, but storage space consumption increases
Solution Approach 1:
The patent segments network packets into different categories based on entropy values. High entropy packets (encrypted/compressed) are separated from low entropy packets (plain text), allowing different storage strategies to be applied to each segment. This segmentation enables selective truncation of high entropy packets while preserving low entropy packets in full, resolving the contradiction between data completeness and storage space consumption.
Solution Approach 2:
The patent applies different quality levels of data retention to different types of packets. Low entropy packets receive full retention (high quality) while high entropy packets receive truncated retention (lower quality). This local quality differentiation allows the system to maintain high data completeness for valuable plain text packets while reducing storage consumption through truncation of less valuable encrypted packets.
2Duration of action of stationary object
If retention time is increased to capture more historical data, then data retention capability is improved, but storage space requirements increase
Solution Approach 1:
The patent extracts and removes the payload portion of high entropy packets, retaining only the packet headers. This extraction approach significantly reduces the amount of data that needs to be stored for extended retention periods. By taking out the redundant encrypted payload data and keeping only the structural header information, the system can achieve longer retention times with reduced storage space requirements.
Solution Approach 2:
The patent discards the payload data of high entropy packets during the capture process, recovering only the essential header information. This selective discarding allows the system to extend retention time by storing less data, while still maintaining the ability to perform network analysis through the preserved header information.
3Productivity
If high entropy packets are truncated to save storage space, then storage efficiency is improved, but data utility for incident investigation may be reduced
Solution Approach 1:
The patent changes the parameter of data retention from full preservation to selective truncation based on entropy parameters. By using entropy as a classification parameter, the system identifies which packets to truncate (high entropy) and which to preserve (low entropy). This parameter-based approach maintains storage efficiency while minimizing information loss, as the truncation decision is made intelligently rather than arbitrarily.
Solution Approach 2:
The patent discards only the payload portion of high entropy packets while recovering and preserving the packet headers. This selective discarding approach maintains storage efficiency by removing redundant encrypted data, while preserving data utility through retention of header information that contains essential network analysis metadata such as source/destination addresses, ports, and protocol information.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
In one embodiment, a computer-implemented method of conserving storage space in a network recorder includes receiving a computer packet including a header and payload data; estimating entropy of the payload data in the computer packet; determining if storage of the computer packet is of value or not based on the header of the computer packet; and storing all or a portion of the computer packet into a storage device based on the estimated entropy and the value determination.