Adaptive Data Segment Compression for Redundant Network Transfers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of dynamic data over networks leads to slow internet speeds, causing delays in business responses, document downloads, and media transmission, which are major complaints among users due to the inefficiency in handling redundant data transfers.
Innovation Solution
A system that selectively revises the boundaries of data segments by identifying matching data within input data and a synchronized store, using a fitness function to determine optimal chunk selection, checksums for indexing, and generating encoded representations with pointers and offsets to reduce data sent over the network, and applying compression techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transmitted over the network without compression, then the transmission is simple and fast, but the network bandwidth is consumed inefficiently due to redundant data
Solution Approach 1:
The patent segments data into fixed-size chunks and identifies redundant segments across multiple transmissions. By dividing data into manageable units (chunks of 256-65535 bytes), the system can efficiently compare and identify duplicates without processing entire files, thereby reducing redundant bandwidth consumption while maintaining transmission productivity.
Solution Approach 2:
The patent performs preliminary actions by maintaining a synchronized store of previously transmitted data chunks on both sending and receiving ends. Before transmitting data, the system checks the synchronized store for existing chunks using checksums, allowing it to identify and eliminate redundant transmissions in advance, thus optimizing bandwidth usage.
2Loss of energy
If compression techniques are applied to reduce data size, then bandwidth consumption is reduced, but the processing time and computational resources increase
Solution Approach 1:
The patent extracts only the essential information needed for compression - specifically, checksums and pointers to synchronized store locations - rather than processing and transmitting entire data sets for comparison. This extraction approach minimizes computational overhead while achieving effective redundancy elimination.
Solution Approach 2:
The patent uses checksums as simplified copies of data content for comparison purposes. Instead of comparing actual data chunks which would be computationally expensive, the system compares lightweight checksum values (32-bit or 64-bit integers), dramatically reducing processing time while maintaining accurate redundancy detection.
3Productivity
If the boundary of data segments is fixed, then the processing is simple and fast, but the compression ratio is suboptimal due to incomplete redundancy identification
Solution Approach 1:
The patent implements dynamic boundary adjustment by allowing chunk boundaries to shift and expand based on synchronization points found in the synchronized store. When matching data is identified, the system dynamically adjusts segment boundaries to include additional redundant content, optimizing compression ratios while maintaining processing efficiency through a structured approach.
4Device complexity
If redundant data is not identified and eliminated, then the system complexity is low, but the network transmission efficiency is poor
Solution Approach 1:
The patent creates a universal synchronized store mechanism that serves multiple functions: storing previously transmitted chunks, enabling redundancy detection, providing compression information, and facilitating efficient retrieval. This multi-functional component achieves high network transmission efficiency without proportionally increasing system complexity.
Solution Approach 2:
The patent introduces checksums as intermediary elements that mediate between the actual data content and the compression logic. These lightweight intermediaries enable efficient redundancy identification by providing a simplified comparison mechanism, achieving high transmission efficiency without complex direct data comparison systems.
Data Source
AI summary
A system, method, and apparatus are directed towards identifying adaptive length segments of redundant data for encoding a data structure. Initial boundaries are identified for an input matching segment within input data and for a candidate store matching segment in a synchronized store. The data prior to and after the boundaries are compared to identify matching data. As matching data is identified, at least one of the boundaries of the matching segments is revised. An encoded representation of the resulting input matching segment is then generated based in part on pointers and offsets into the synchronized store. A data structure is generated based on the encoded representation and unmatched portion, which is sent to a receiver. The receiver uses the data structure to extract matching data from the synchronized store, and together with the unmatched input data in the data structure, reconstruct the input data.