Deep Packet Inspection De-Duping With Smaller Hash Histories
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data de-duplication techniques lack protocol-specific optimization, leading to inefficient data processing due to generic algorithms, large hash tables, and excessive hashing and chunking, which affects compression ratios and processing efficiency.
Innovation Solution
Implementing deep packet inspection (DPI) to identify protocols, partition data into chunks, apply hash algorithms, and store selected hash values in a smaller hash table, allowing for efficient matching and comparison of data chunks with historical data without re-chunking or re-hashing, and using linked containers for storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a larger hash table is used to store more hash values, then the likelihood of finding matching data chunks increases, but the processing time and memory usage increase
Solution Approach 1:
The patent segments the hash table into multiple smaller hash tables, each responsible for storing hash values of specific data chunks. This segmentation allows parallel processing of hash lookups across multiple tables, reducing the time required to search while maintaining comprehensive coverage of all hash values. The segmented structure enables the system to achieve both high matching accuracy and efficient processing speed.
2Productivity
If longer histories are stored to enable more frequent matching, then compression ratios improve, but the amount of data to be processed and stored increases
Solution Approach 1:
The patent extracts only the essential hash values from historical data and stores them in specialized hash tables, rather than storing complete historical data sequences. This extraction approach allows the system to maintain long historical contexts for frequent matching while keeping the stored data volume minimal. The hash tables contain only the critical identifying information needed for matching, discarding redundant historical data.
Solution Approach 2:
The patent changes the parameter representation of historical data by converting complete data chunks into compact hash values. This parameter transformation enables the system to retain information about extensive historical data sequences in a compressed form, allowing long histories to be maintained with reduced storage requirements while still enabling frequent pattern matching.
3Productivity
If more hash values are stored in the hash table, then the compression ratio improves, but the hash table size and processing complexity increase
Solution Approach 1:
The patent divides the comprehensive hash value storage task across multiple segmented hash tables, where each table manages a specific portion of the data stream. This segmentation reduces the complexity of individual tables while collectively maintaining comprehensive hash coverage. The modular structure simplifies processing by allowing independent management and optimization of each segment.
Solution Approach 2:
Instead of storing all hash values in a single comprehensive table, the patent inverts the approach by creating multiple specialized hash tables, each optimized for specific data patterns or time periods. This inversion distributes the complexity across multiple simpler structures rather than concentrating it in one complex table, achieving both high compression ratios and manageable complexity.
Data Source
AI summary
The efficiency of data de-duplication may be improved by storing related file data in a single container, or in multiple linked containers, of a history. Additionally, the efficiency of data de-duplication may be improved when shorter hash tables are used to reference historical data in a history. Shorter hash tables may be achieved by storing fewer than all the hash values obtained for a given amount of historical data. Further, the efficiency of data de-duplication may be improved by comparing related incoming file data with historical data from a container without hashing/chunking the remaining file data upon matching an earlier chunk of the incoming file data to the container.