SIMD Data Compression Using Parallel Hash Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression methods require significant processing resources and time due to their sequential nature, which limits their efficiency in reducing data volume for storage and transfer.
Innovation Solution
The use of a Single Instruction Multiple Data (SIMD) engine for concurrent execution of operations such as calculating hash values, searching for matches in a hash table, and updating the hash table, allowing for the compression of data streams by replacing repetitive sequences with pointers, thereby reducing processing resources and time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential data compression methods are used, then data volume is reduced for storage and transfer, but processing resources and time are significantly increased
Solution Approach 1:
The patent segments the input data stream into multiple parallel data streams, allowing simultaneous processing of multiple data chunks through the compression algorithm. This segmentation enables the system to process N pieces of data in parallel, effectively increasing compression throughput and reducing overall processing time while maintaining the same compression ratio.
Solution Approach 2:
The patent transitions from sequential single-stream processing to parallel multi-stream processing by adding a temporal dimension to the compression operation. Multiple data streams are processed concurrently in different time slices, transforming the compression process from a one-dimensional sequential operation to a multi-dimensional parallel operation, thereby improving productivity without sacrificing compression efficiency.
2Productivity
If traditional compression algorithms are used, then data is compressed effectively, but processing resources such as CPU and memory are heavily consumed
Solution Approach 1:
By dividing the compression task into multiple independent parallel streams, each stream processes a portion of the data with reduced computational burden. This segmentation distributes the processing load across multiple cores or processing units, increasing overall throughput while reducing the resource consumption per individual processing unit.
Solution Approach 2:
The patent merges multiple compression operations into a single parallel processing framework. By combining several compression tasks that operate simultaneously on different data streams using the same algorithmic logic, the system achieves economies of scale in resource utilization, improving throughput while optimizing the use of processing resources.
Data Source
AI summary
A system for compressing an input data stream to create a compressed output data stream is provided. The system comprises a memory storing a hash table comprising hash entries each comprising a hash value of an associated subset of following data items of an input data stream and a pointer to a memory location of the associated subset. A processor coupled to the memory executes operations while instructing an SIMD engine to execute concurrently one or more of the operations for consecutive subsets: calculate the hash value for each subset, search the hash table for a match of each calculated hash value and update the hash table according to the match result. The processor then updates the compressed output data stream according to the match result and a comparison result depending on the match result and operations for the plurality of associated subsets to create the compressed output data stream.