Network Packet Truncation Using Entropy and Value Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer network data packet storage devices face limitations in retention time due to finite storage capacity, requiring a balance between data retention and storage space usage, especially when dealing with high entropy packets that have low storage value.
Innovation Solution
Implementing entropy and value-based packet truncation methods, where packets with high entropy (e.g., encrypted or compressed data) are truncated to conserve storage space, while packets with low entropy (e.g., plain text) are preserved, using techniques like compression and selective truncation based on entropy thresholds and packet utility values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If full packet capture is performed to maintain data coverage for incident investigation, then data completeness is improved, but storage space consumption increases
Solution Approach 1:
The patent applies local quality by differentiating treatment of packet portions based on their entropy characteristics. High-entropy packets (encrypted/compressed) are truncated to save space, while low-entropy packets (plain text) are preserved in full. This selective approach ensures data completeness is maintained only where necessary, optimizing the balance between information retention and storage efficiency.
Solution Approach 2:
The patent changes the parameter of packet retention based on entropy measurement. By calculating entropy as a parameter, the system dynamically adjusts whether to truncate or preserve each packet's payload. This parameter-driven approach allows the system to adapt storage allocation to the actual information content of each packet, resolving the contradiction between maintaining data completeness and reducing storage consumption.
2Duration of action of stationary object
If retention time is increased to capture more historical data, then data retention is improved, but storage space requirements increase
Solution Approach 1:
The patent applies discarding and recovering by selectively discarding high-entropy packet payloads that can be reconstructed or are less valuable for investigation, while preserving low-entropy packets. This allows the system to extend retention time within fixed storage constraints by recovering storage efficiency through intelligent selection of what to retain and what to discard.
Solution Approach 2:
The patent implements partial action by truncating only the payload portion of high-entropy packets while retaining packet headers and metadata. This partial retention approach maintains sufficient information for incident investigation while significantly reducing storage requirements, thereby enabling extended retention periods within available storage capacity.
3Quantity of substance
If high entropy packets are truncated to save storage space, then storage efficiency is improved, but data value may be reduced
Solution Approach 1:
The patent introduces entropy calculation as an intermediary mechanism to evaluate packet value before truncation decisions. This intermediary assessment allows the system to objectively determine which packets are suitable for truncation based on their entropy characteristics, ensuring that data value is preserved where necessary while achieving storage efficiency through truncation of appropriate candidates.
Data Source
AI summary
In one embodiment, a computer-implemented method of conserving storage space in a network recorder includes receiving a computer packet including a header and payload data; estimating entropy of the payload data in the computer packet; determining if storage of the computer packet is of value or not based on the header of the computer packet; and storing all or a portion of the computer packet into a storage device based on the estimated entropy and the value determination.