Parallel Dictionary Compression for Sequential Coding Bottlenecks

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Dictionary-based sequential coders, such as LZMA, have slow compression speeds due to sequential processing, which prevents parallelization and makes them slower than other compression methods like block sorting coders.

Innovation Solution

Implementing parallel data compression on a multi-processor system by determining a dictionary size and block size, then partitioning input data into blocks for simultaneous processing across multiple cores using dictionary-based sequential compression algorithms like Lempel-Ziv variants.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of substance

If dictionary-based sequential coders process data sequentially byte-by-byte, then compression ratio is improved, but compression speed deteriorates

Engineering Contradiction:

Improvecompression ratioVSAvoidcompression speed

Core Design Contradiction:

Loss of substanceVSProductivity

Solution Approach 1:

The patent divides the input data into multiple blocks that can be processed in parallel. Each block is independently compressed using dictionary-based sequential coding, allowing multiple processing units to work simultaneously on different segments of the data, thereby improving compression speed while maintaining compression ratio.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent transitions from single-threaded sequential processing to multi-threaded parallel processing by adding a temporal dimension to the compression process. Multiple threads execute compression operations concurrently on different data blocks, transforming the single-dimensional sequential approach into a multi-dimensional parallel approach that achieves both high compression ratio and fast speed.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Device complexity

If sequential processing is used in dictionary-based coders, then algorithm simplicity is maintained, but parallelization capability is lost

Engineering Contradiction:

Improvealgorithm simplicityVSAvoidparallelization capability

Core Design Contradiction:

Device complexityVSAdaptability or versatility

Solution Approach 1:

The patent segments the compression task into independent blocks that can be processed by multiple threads simultaneously. Each block maintains the simplicity of dictionary-based sequential coding while the overall system gains parallelization capability through the segmented structure, allowing the algorithm to scale across multiple processing units.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary actions by pre-processing the input data into suitable blocks and preparing the compression parameters before actual compression begins. This preliminary segmentation and setup enables subsequent parallel processing to proceed efficiently while each individual compression operation remains simple and straightforward.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS8279096B2Parallel compression for dictionary-based sequential coders

Publication Date: 2012.10.02 RED HAT INC

AI summary

Dictionary-based sequential compression is performed in parallel on input data using multiple cores of a computer system. The compression can be performed by first determining a dictionary size for data compression, followed by determining a block size for partitioning the input data. The block size is greater than the dictionary size. The multiple cores of the computer system then execute multiple threads in parallel, with each thread compressing one block of the input data according to a dictionary-based sequential compression scheme.