Compressed Data Stream Intrusion Detection Using Statistical Signatures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage technologies are insufficient to keep pace with the rapidly increasing demand for data storage, and existing intrusion detection systems face limitations such as inability to process encrypted packets, frequent false positives, and reliance on continually updated signature libraries.
Innovation Solution
A system and method for data compression with intrusion detection that measures the probability distribution of encoded data streams, compares it to a reference distribution, and uses statistical algorithms to detect divergence, thereby identifying potential data intrusions. This system includes a signature generating component that correlates anomalous events with known vulnerabilities to create statistical signatures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression is used to increase storage capacity, then storage efficiency is improved, but intrusion detection capability deteriorates
Solution Approach 1:
The system separates intrusion detection from the data compression process by operating on the compressed bitstream itself. The detector divides the compressed data into blocks and analyzes statistical properties without requiring decompression, enabling independent detection functionality that doesn't interfere with storage efficiency.
Solution Approach 2:
The patent introduces an intermediary detection layer that operates on the compressed bitstream between the compression process and storage. This intermediary analyzes statistical properties of the compressed data using techniques like entropy calculation and block comparison, allowing intrusion detection without decompression while maintaining storage efficiency.
2Reliability
If traditional intrusion detection systems are used to detect attacks, then detection capability is improved, but processing speed and efficiency deteriorate
Solution Approach 1:
The system extracts only the essential statistical properties from compressed data blocks for analysis, such as entropy values and frequency distributions. By taking out only these key features rather than fully decompressing and analyzing all data, the system achieves fast detection with minimal processing overhead.
Solution Approach 2:
The patent uses lightweight statistical metrics that are computationally inexpensive to calculate, such as simple entropy estimates and block frequency counts. These disposable statistical features can be rapidly computed and discarded, enabling high-speed processing without the need for complex decompression or analysis.
3Measurement precision
If signature libraries are continually updated to detect latest threats, then detection accuracy is improved, but system complexity and maintenance burden increase
Solution Approach 1:
The system performs self-adaptation by automatically learning the statistical characteristics of normal compressed data from the specific compression algorithm being used. This self-service approach eliminates the need for manual signature library updates, as the detector adapts to new compression methods and threats automatically through statistical analysis.
Solution Approach 2:
The patent changes the detection parameters from fixed signature patterns to dynamic statistical properties that adapt to different compression algorithms and data types. By using parameters like entropy, frequency distributions, and block variance that can be recalculated automatically, the system maintains detection accuracy without requiring manual signature updates.
Data Source
AI summary
A system and method for data compression with intrusion detection, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system further comprises a signature generating component which correlates anomalous event data with known vulnerabilities and exploits to create a signature based on statistical information of the anomalous event. Computed statistics may be compared against a signature database to determine if a data intrusion has occurred.