Network Recorder Packet Truncation Using Payload Entropy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Network recorders face challenges in efficiently storing computer network data packets due to limited storage capacity, requiring a balance between retention time and storage space, as full packet capture consumes significant space and incurs economic costs.
Innovation Solution
Implementing entropy and value-based packet truncation, where packets are categorized by entropy and utility value, allowing for partial or complete truncation of low-value packets, especially encrypted and compressed data, while preserving high-value plain text data, to conserve storage space and extend retention time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If full packet capture is implemented, then data completeness is improved, but storage space consumption increases
Solution Approach 1:
The patent applies local quality by treating different packets differently based on their characteristics. High-value packets (containing plain text, unencrypted data) are preserved completely, while low-value packets (encrypted, compressed data) are truncated. This selective treatment optimizes storage space while maintaining data completeness for important information.
Solution Approach 2:
The patent changes the parameter of packet retention by introducing entropy-based classification. Packets are evaluated using entropy metrics to determine their value, and the retention parameter is adjusted accordingly - high entropy packets are truncated while low entropy packets are preserved. This dynamic parameter adjustment resolves the contradiction between storage efficiency and data completeness.
2Duration of action of stationary object
If retention time is increased, then historical data availability is improved, but storage space requirements increase
Solution Approach 1:
The patent implements discarding and recovering by selectively discarding low-value packet data (high entropy portions) while preserving high-value data. This allows the system to extend retention time for valuable information without proportionally increasing total storage requirements, as redundant or low-value data is discarded.
Solution Approach 2:
The patent applies partial action by retaining only the necessary portions of packets based on their value. Instead of storing complete packets uniformly, the system performs partial retention - keeping full high-value packets and truncated low-value packets - thereby extending effective retention time for important data while controlling overall storage consumption.
3Reliability
If encrypted and compressed data is preserved, then data security is maintained, but storage efficiency decreases
Solution Approach 1:
The patent applies inversion by reversing the conventional approach to encrypted data. Instead of preserving all encrypted data for security, the system identifies that high-entropy encrypted data cannot be meaningfully analyzed without decryption keys and therefore has low investigative value. The inversion is to truncate this data rather than preserve it, thereby improving storage efficiency without compromising security or analytical capability.
Data Source
AI summary
In one embodiment, a network recorder includes a storage device; a motherboard; and daughter-cards. Each of the daughter-cards includes an input port, a packet analyzer, an entropy calculator, a comparator, and a processor. The input port receives a plurality of computer data packets. The packet analyzer identifies header fields and a starting point of the payload data in the plurality of computer data packets. The entropy calculator examines the payload data of the plurality of the computer data packets to respectively generate an entropy estimate for each of the plurality of computer data packets. The comparator compares each entropy estimate with an entropy truncation threshold to generate an entropy exceed signal to indicate that the payload data of a computer data packet can be truncated to conserve storage space. The processor compresses at least some of the payload data of the plurality of computer data packets that are not truncated.