Redundancy Removal Engine Hash Indexing for Bandwidth Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Bandwidth-constrained network connections lead to delayed data transmission and inefficient data de-duplication due to the tradeoff between block size for data compression, where small block sizes increase signature data storage needs but degrade compression efficiency, and large block sizes reduce network bandwidth usage.
Innovation Solution
A method involving a redundancy removal engine that generates hash identifiers for contiguous data blocks, selects indexed hash identifiers based on sparsity constraints, and replaces redundant data with compression instructions, using a bidirectional cache memory to optimize data transmission by reducing memory consumption and enhancing compression efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a small block size is used for data compression, then compression efficiency is improved, but signature data storage requirements increase
Solution Approach 1:
The patent divides the data stream into fixed-size blocks and generates multiple hash identifiers (fingerprints) for each block. This segmentation allows the system to use smaller effective block sizes for compression while managing signature storage through selective hashing. The data block is processed in segments with different hash functions applied to create a manageable set of signatures.
Solution Approach 2:
The patent transitions from storing entire data blocks or large signatures to storing compact hash identifiers. By converting data blocks into hash space (another dimension), the system achieves efficient storage and comparison. Multiple hash identifiers per block create a dimensional expansion that enables better matching while controlling storage requirements.
2Quantity of substance
If a large block size is used for data compression, then signature data storage requirements are reduced, but compression efficiency deteriorates
Solution Approach 1:
The patent segments large data blocks into smaller portions for hash generation, allowing efficient storage of multiple hash identifiers while maintaining the ability to compress large blocks effectively. This segmentation enables the system to balance storage requirements with compression performance.
Solution Approach 2:
The patent changes parameters such as hash identifier length, number of hash identifiers per block, and block size to optimize the tradeoff between storage requirements and compression efficiency. By adjusting these parameters, the system can adapt to different storage constraints while maintaining effective compression.
3Measurement precision
If multiple hash identifiers are generated for each data block, then compression accuracy is improved, but memory consumption increases
Solution Approach 1:
The patent optimizes parameters including the number of hash identifiers per block, hash identifier length, and block size to achieve the desired balance between compression accuracy and memory consumption. By carefully selecting these parameters, the system maintains high accuracy while controlling memory usage.
Solution Approach 2:
The patent generates a limited number of hash identifiers (not all possible combinations) to achieve sufficient compression accuracy without excessive memory consumption. This partial action approach provides a practical balance between accuracy and resource usage.
Data Source
AI summary
A method, an apparatus and/or a system of data compression through redundancy removal in an application acceleration environment is disclosed. In one aspect, a method includes generating through a fingerprint module of a redundancy removal engine associated with a first network element a hash identifier of at least one portion of data of a contiguous data block stored in a bidirectional cache memory of the redundancy removal engine associated with a first network element. The size of contiguous data block is larger than the size of the at least one portion of data of the contiguous data block for which the hash identifier is generated. The method further includes compressing a data stream through a transmission compression module of the redundancy removal engine based on the at least one a hash identifier that is indexed.