Filesystem Codebook Compression With Statistical Intrusion Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rapid growth of data storage demand outstrips the capacity to store it, and existing data compression and intrusion detection systems are inadequate, particularly for encrypted data and dynamic threat detection.
Innovation Solution
A system and method for filesystem data compression using codebooks that measure real-time probability distributions, integrate intrusion detection, and use statistical algorithms to detect anomalies, enabling efficient compression and threat detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data compression methods are used, then storage capacity is doubled, but compression ratio is limited and cannot solve the global data storage problem
Solution Approach 1:
The patent changes the fundamental parameter of data representation by converting raw data into probability distribution models. Instead of compressing individual data points, the system models the statistical properties of data streams, achieving compression ratios of 10:1 or higher by storing only the essential probabilistic characteristics rather than all original data.
Solution Approach 2:
The patent applies a phase transition in the data processing domain by transforming deterministic data into probabilistic models. This transition from exact data representation to statistical representation enables dramatically higher compression ratios while maintaining the ability to reconstruct and analyze data patterns.
2Reliability
If signature-based intrusion detection systems are used, then known attacks can be detected, but the systems cannot detect new attacks and require frequent updates
Solution Approach 1:
The patent performs preliminary action by establishing baseline probability distributions of normal data streams before attacks occur. These baselines are created in advance and enable the system to detect anomalies representing new attacks without requiring prior knowledge or signature updates, as any deviation from the established baseline triggers detection.
Solution Approach 2:
The system implements continuous feedback by constantly comparing incoming data streams against the established probability distributions. This real-time feedback mechanism automatically adapts to new attack patterns by detecting statistical deviations, eliminating the need for manual signature library updates while maintaining high detection accuracy for both known and unknown threats.
3Productivity
If codebook-based compression is used, then compression efficiency is improved, but the system cannot detect intrusions or data tampering
Solution Approach 1:
The patent applies universality by making the probability distribution modeling serve multiple functions simultaneously. The same statistical models that enable high-ratio compression also provide the foundation for intrusion detection, as any data tampering or intrusion manifests as a deviation from the expected probability distributions. This multi-functionality eliminates the need for separate compression and security systems.
Data Source
AI summary
A system and method for filesystem data compression using codebooks, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system comprises both encoding and decoding machines, an intrusion detection module, a codebook training module, and various databases which perform various analyses on encoded data streams. Further, the system comprises a system for integrating the compression into a filesystem for both system-wide compression on a per-file or filegroup basis, and intrusion or alteration detection of files.