Dictionary Compression Using Bloom Filters for Mesh Network Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing symmetric data deduplication technologies face scaling challenges in large mesh networks, making it difficult to efficiently manage bandwidth across wide area networks.
Innovation Solution
A system and method utilizing a data reducing module that performs dictionary compression by selecting data segments with universal identifiers and data length information, querying a global bloom filter to determine if the receiver has stored copies, and sending only the identifier and length information to allow the receiver to retrieve and decode the data segments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If symmetric data deduplication technology is used between network traffic management devices, then bandwidth consumption is reduced, but scaling capability deteriorates in large mesh networks
Solution Approach 1:
The patent segments the data transmission process into multiple components: data segments are divided into fixed-size blocks, each block is independently hashed and checked against a bloom filter. This segmentation allows the system to scale efficiently in large mesh networks while maintaining bandwidth reduction benefits through parallel processing and distributed storage of segment information.
2Reliability
If data segments are transmitted with full content data bytes, then data completeness is ensured, but bandwidth consumption increases
Solution Approach 1:
The patent extracts only the essential identifying information (hash values and bloom filter bits) from the complete data segments for transmission. The receiver uses these extracted identifiers to retrieve the full content data from its local storage, eliminating the need to transmit redundant content data bytes while ensuring data completeness through verification mechanisms.
Solution Approach 2:
The receiver device performs preliminary actions by pre-storing data segments and their associated hash values in local storage before transmission is needed. When a data segment needs to be transmitted, the sender queries the bloom filter to determine if the receiver already has the data, avoiding redundant transmissions and optimizing bandwidth usage.
Data Source
AI summary
A system, medium and method of performing dictionary compression is disclosed. A first data segment received at a receiver device (RD) from a transmiter device (TD) is selected A global bloom filter of the TD is queried to determine if the RD has a stored copy of a first plurality of content data bytes and corresponding first identifier and data length information for the first data segment. A first encoded data packet is prepared and sent which includes the first identifier and data length information without the first plurality of content data bytes. The RD utilizes the received first identifier and data length information to retrieve the first plurality of content data bytes associated with the first data segment from the RD's data store and decodes the first data segment to include the first plurality of content data bytes.