Compressed Data Stream Anomaly Detection Using Probability Divergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage technologies are insufficient to meet the rapidly increasing demand for data storage, as evidenced by the exponential growth of digital data storage exceeding the growth in storage capacity, and existing intrusion detection systems face limitations such as inability to process encrypted packets, frequent false positives, and vulnerability to protocol-based attacks.
Innovation Solution
A system and method for data compression with intrusion detection that measures the probability distribution of an encoded data stream in real-time, compares it to a reference probability distribution, and uses statistical algorithms to determine divergence, thereby detecting unusual distributions that may indicate data intrusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression is applied to increase storage capacity, then storage efficiency is improved, but intrusion detection capability deteriorates because compressed data loses its original structure and patterns
Solution Approach 1:
The system segments the intrusion detection process into two parts: (1) detecting anomalies in the compressed data stream using probability distribution analysis, and (2) optionally decompressing only the suspicious segments for detailed inspection. This allows intrusion detection to operate on compressed data without requiring full decompression, thus maintaining storage efficiency while preserving detection capability.
Solution Approach 2:
The patent introduces probability distribution analysis as an intermediary method between compressed data and intrusion detection. Instead of directly analyzing compressed data patterns or fully decompressing data, the system uses statistical properties (probability distributions) of the compressed stream as a mediator to detect anomalies, thereby bridging the gap between compression and detection.
2Reliability
If traditional intrusion detection systems are used to detect anomalies, then security monitoring is improved, but processing speed deteriorates due to the need to decompress and analyze entire data streams
Solution Approach 1:
The system applies partial action by performing intrusion detection on only the compressed data stream without full decompression. It calculates probability distributions and detects anomalies directly from the compressed format, using just enough processing to identify suspicious patterns while avoiding the excessive computational cost of complete decompression and analysis.
Solution Approach 2:
The patent changes the detection parameter from traditional pattern matching on decompressed data to probability distribution analysis on compressed data. By transforming the detection approach to work with statistical parameters of the compressed stream rather than requiring full data reconstruction, processing speed is maintained while security monitoring effectiveness is preserved.
3Manufacturing precision
If lossless compression is used to maintain data integrity, then data quality is improved, but compression ratio deteriorates resulting in minimal storage savings
Solution Approach 1:
The system uses a disposable approach to data representation by creating a statistical model (probability distribution) of the compressed data that can be analyzed for intrusion detection without requiring the original high-quality data format. This allows the system to work with lower-quality compressed representations for detection purposes while maintaining data integrity only when needed.
Data Source
AI summary
A system and method for data compression with intrusion detection, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system comprises both encoding and decoding machines, an intrusion detection module, a codebook training module, and various databases which perform various analyses on encoded data streams.