History Buffer Decompression Across Local and Remote Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current decompression technologies face memory pressure and network traffic limitations in handling large numbers of concurrent connections, particularly in stateless decompression, where the entire compressed data stream must be available, leading to inefficiencies and potential vulnerabilities in systems like intrusion prevention systems.
Innovation Solution
A method and system that utilize a history buffer to asynchronously fetch and store data dictionary elements from remote memory, allowing for stalled decompression operations to proceed using locally available data, and prefetching recent history to reduce memory access delays, enabling efficient stateful decompression and minimizing memory overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If stateless decompression is used to handle large numbers of concurrent connections, then the system can process multiple connections simultaneously, but significant memory pressure is created requiring the entire compressed data stream to be available
Solution Approach 1:
The patent segments the data dictionary into multiple pages that can be independently managed and loaded. Instead of requiring the entire data dictionary to be available in memory simultaneously, the system divides it into manageable pages that are loaded on-demand, reducing the memory pressure while maintaining the ability to handle multiple concurrent connections
Solution Approach 2:
The patent implements preliminary loading of data dictionary pages into a history buffer before they are needed for decompression operations. By pre-loading pages into local memory buffers, the system reduces the need to access remote memory during active decompression, thereby reducing memory pressure while maintaining high concurrent connection处理能力
2Reliability
If decompression engines are dedicated to particular streams to enable per-packet decompression, then packets can be decompressed one after another, but the number of concurrent connections is limited to the number of decompression engines
Solution Approach 1:
The patent introduces a shared history buffer as an intermediary structure that multiple decompression engines can access simultaneously. This shared buffer allows packets from different streams to be decompressed in parallel by different engines while maintaining the stateful decompression capability, thereby increasing concurrent connection capacity without sacrificing per-packet processing reliability
Solution Approach 2:
The patent makes the history buffer a universal resource that serves multiple decompression engines simultaneously. Instead of dedicating separate history buffers to each engine, the same history buffer is shared across multiple engines, allowing them to collaborate on decompressing multiple concurrent streams without requiring proportional increases in memory resources
3Ease of operation
If the entire compressed data stream is made available for stateless decompression, then independent processing of different streams is enabled, but service attacks can be exploited through memory pressure
Solution Approach 1:
The patent implements partial loading of the data dictionary into the history buffer, loading only the necessary pages rather than the entire data stream. This partial action approach maintains independent stream processing capability while limiting the memory exposure to attack vectors, as only portions of the data dictionary are loaded into accessible memory at any given time
Data Source
AI summary
A method for decompressing a stream of a compressed data packet includes determining whether first data of a data-dictionary for a first decompression copy operation is located in a history buffer on a remote memory or a local memory, and when it is determined that the first data is located in the remote memory, stalling the first decompression copy operation, performing a second decompression operation using second data that is located in the history buffer on the local memory and fetching the first data from the remote memory to the history buffer on the local memory. The method further includes performing the first decompression operation using the first data in the history buffer on the local memory.