Reference-Token Data Compression for Redundant Transmission
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression and decompression methods fail to efficiently utilize redundancies in data transmission, leading to inefficient bandwidth utilization due to the time-consuming and low-yield process of locating redundant data across large volumes of storage.
Innovation Solution
An apparatus and method that utilize a memory device to store history data and generate reference tokens reflecting the relationship between input data and stored data, allowing for the efficient identification and transmission of redundant data, thereby minimizing redundant data transmission and optimizing bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing compression and decompression methods are used to locate redundant data in large volumes of storage data, then the process is thorough, but it is time-consuming and has low yield
Solution Approach 1:
The patent segments the large volume of storage data into fixed-size data chunks, and further segments each chunk into smaller sub-chunks for signature generation. This segmentation allows the system to process and compare data in manageable units rather than searching through entire large files, significantly reducing the time required to locate redundancies while maintaining detection accuracy through systematic coverage of all data portions.
Solution Approach 2:
The patent performs preliminary actions by generating signatures for data chunks and storing them in a chunk table before actual compression operations. When compression is needed, the system compares incoming data signatures against the pre-generated chunk table signatures, eliminating the need for time-consuming searches through the entire storage volume during compression operations.
2Reliability
If redundant data is transmitted over the network, then all data is preserved for the recipient, but bandwidth utilization becomes inefficient
Solution Approach 1:
The patent extracts only the essential information needed for data reconstruction by generating reference tokens that contain location and size information of redundant data chunks. Instead of transmitting the actual redundant data content, the system transmits these compact reference tokens along with non-redundant data portions, ensuring the recipient can reconstruct the complete data while using minimal bandwidth.
Solution Approach 2:
The patent creates reference tokens that act as pointers or references to the actual data chunks stored in the chunk table. These reference tokens contain sufficient information (location, size, signature) for the recipient to retrieve and reconstruct the original data without transmitting the full data content, thereby preserving data completeness while optimizing bandwidth utilization.
3Loss of information
If the entire input data is transmitted to the recipient device, then no data is lost, but the transmission efficiency is reduced due to redundant data
Solution Approach 1:
The patent applies partial action by transmitting only the necessary portions of data - specifically, non-redundant data portions and reference tokens for redundant portions. The system identifies and transmits exactly what is needed for complete reconstruction without over-transmitting redundant information, achieving optimal transmission efficiency while maintaining full data integrity through the combination of transmitted data and reference tokens.
Data Source
AI summary
An apparatus and a computer-implemented method for processing data are disclosed. The apparatus comprises a memory device that stores first history data, and at least one processor configured to receive input data, determine a relationship between the first history data and one or more portions of the input data, generate one or more reference tokens reflecting the relationship, and transmit the one or more reference tokens to a recipient device.