Parallel Compression Engine Selection for Fixed-Block Data Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression systems for fixed block sizes, such as those in storage products, achieve low compression ratios due to limited data history, which complicates optimization of storage performance and compression speed in applications like streaming data services.
Innovation Solution
An information handling system with multiple compression engines and a Smart Data Accelerator Interface (SDXI) that utilizes a unified source buffer and multiple destination buffers, allowing parallel operation of different compression algorithms to select the best compression ratio or speed based on criteria such as compression threshold or data type.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single compression engine is used with fixed block size, then device complexity is reduced, but compression ratio deteriorates due to limited data history
Solution Approach 1:
The system divides the compression task into multiple parallel compression engines, each processing different portions of the data stream. This segmentation allows each engine to maintain its own data history buffer, effectively increasing the total data history available for compression without requiring a single complex engine.
Solution Approach 2:
Multiple compression engines are merged into a unified compression system that processes data in parallel. The results from multiple engines are combined through a selection mechanism that chooses the best compressed output, achieving higher overall compression ratios while maintaining manageable device complexity.
2Productivity
If multiple compression algorithms are used in parallel, then compression efficiency is improved, but device complexity increases
Solution Approach 1:
The system dynamically selects which compression engine's output to use based on real-time performance metrics such as compression ratio and processing speed. This dynamic selection allows the system to adapt to different data types and compression requirements without requiring all engines to operate simultaneously at full capacity, thereby managing device complexity.
Solution Approach 2:
Each compression engine operates independently with its own data history buffer and processing logic, making the system self-organizing. The selection mechanism automatically chooses the best performing engine without external intervention, allowing the system to scale by adding more engines without proportionally increasing control complexity.
3Speed
If fixed block size compression is used, then processing speed is improved, but compression ratio deteriorates due to limited data history
Solution Approach 1:
The data stream is segmented into fixed blocks that are processed in parallel by multiple compression engines. Each engine receives a portion of the data and maintains its own history buffer, allowing fixed block processing speed to be maintained while collectively achieving better compression ratios through the combined processing capacity of multiple engines.
Data Source
AI summary
An information handling system for compressing data includes multiple compression engines, a source data buffer to provide compression data to the compression engines, at least one destination data buffer to receive compressed data from the compression engines, and a compression engine driver. Each compression engine is configured to provide a different compression function. The compression engine driver directs each compression engine to compress data from the source data buffer, and retrieves select compressed data from a first one of the compression engines from the at least one destination data buffer. The selection is based upon a selection criterion.