Compression Engine Buffer Bypass for Consistent Data Throughput
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression systems experience inconsistent throughput, leading to undesirable performance in flash storage systems due to varying compressibility of data and increased latency, which can be unacceptable in data storage and transmission applications.
Innovation Solution
The method involves using a buffer to store input data, distributing it to compression engines while monitoring latency, and bypassing the engines if latency exceeds a threshold, allowing simultaneous reading and writing to the output stream, and evaluating data for compressibility to bypass both the buffer and compression engines for non-compressible data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is compressed using traditional compression engines, then data storage requirements and transmission bandwidth requirements are reduced, but throughput becomes inconsistent and latency increases
Solution Approach 1:
The patent segments the data stream into fixed-size blocks and processes them through multiple compression engines in parallel. This segmentation allows the system to maintain consistent throughput by distributing work across multiple engines, preventing any single engine from becoming a bottleneck while still achieving compression for storage and transmission.
Solution Approach 2:
The patent implements a look-ahead buffer that pre-processes and evaluates incoming data blocks before they reach the compression engines. By performing preliminary analysis to identify compressible patterns and prepare data blocks in advance, the system can maintain steady throughput without waiting for compression results, thus resolving the throughput inconsistency problem.
2Quantity of substance
If compression is performed to reduce data storage requirements, then storage efficiency improves, but latency increases which is unacceptable in many applications
Solution Approach 1:
The patent introduces an intermediary buffer system between the data source and compression engines. This buffer acts as a mediator that decouples the data input rate from the compression processing rate, allowing compression to occur at optimal speeds without introducing variable latency. The buffer absorbs timing variations and ensures steady data flow to compression engines, reducing overall system latency while maintaining storage efficiency.
3Productivity
If data is evaluated for compressibility before compression, then non-compressible data can be bypassed, but system complexity increases
Solution Approach 1:
The patent implements a partial evaluation approach where only essential compressibility checks are performed on data blocks before compression. Rather than进行全面 analysis, the system performs targeted checks for common compression patterns. This partial action approach maintains compression efficiency by identifying obviously compressible data while avoiding the complexity of comprehensive analysis, thus resolving the contradiction between efficiency and complexity.
Data Source
AI summary
Systems and method provide for consistent throughput of one or more compression engines. Data received from an input stream is stored in a buffer. Data is read from the buffer and distributed to the compression engines. Latency of the compression engines is monitored. If latency exceeds a threshold, data is read from the buffer and written to an output stream simultaneously with reading of data and inputting it to the compression engines. Data from the input stream may be evaluated for likely compressibility and non-compressible data may be written to the output stream bypassing both the buffer and the compression engines.