Delta Compression with Forgetful Cache for Unordered Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Delta compression in data streams with unordered data items is inefficient due to the need for large data stores to store previous values, leading to high latency and variable throughput, especially when related data items are not adjacent, making it costly to maintain a quick-access cache and resulting in non-constant data compression efficiency.
Innovation Solution
A data compressor calculates delta values using both stored related items and a predetermined value, allowing for constant throughput by using a 'forgetful' data store that allocates and evicts storage locations dynamically, and a data formatter packs output values into fixed-size packets with extension formats to handle varying data sizes efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If a large data store is used to store all previously received data items to enable delta calculation between related items in unordered streams, then compression effectiveness is improved, but device complexity and cost increase
Solution Approach 1:
The patent changes the parameter of data store capacity from large to small by introducing a forgetful cache that dynamically evicts older entries. This allows the system to maintain effective compression by calculating deltas between related items using only recently received data, rather than requiring access to all historical data items.
Solution Approach 2:
The patent applies the discarding principle by implementing a forgetful cache that automatically evicts and discards older data items when the cache is full. The system recovers compression effectiveness by continuously receiving new data items and calculating deltas between related items within the limited cache window, ensuring that the most relevant data is retained for compression operations.
2Loss of information
If a large data store is used to ensure related data items are always available for delta calculation, then compression quality is improved, but latency increases
Solution Approach 1:
The patent changes the time parameter by implementing a time-bounded forgetful cache with a fixed capacity. This ensures that the system processes data items within a predictable time frame by evicting older entries, thereby maintaining constant latency while still achieving effective compression by calculating deltas between related items within the cache window.
3Speed
If a quick-access cache is used to store previously received data items, then delta calculation speed is improved, but area cost increases
Solution Approach 1:
The patent changes the area parameter by implementing a forgetful cache with limited capacity. This allows the system to maintain quick access for delta calculation on recently received data items while using minimal cache area, rather than allocating large memory space to store all historical data items.
4Productivity
If data items are stored in allocated locations in the data store, then access efficiency is improved, but the data store requires larger size to handle unordered streams
Solution Approach 1:
The patent changes the volume parameter by implementing a forgetful cache with a fixed, limited capacity. This allows the system to maintain efficient access by storing data items in allocated locations within the limited cache, evicting older entries when full, thereby achieving high access efficiency without requiring large data store volume.
Data Source
AI summary
A data compressor has a delta value calculator which receives data items and determines if a related data item to a received data item is stored in a data store. If the related item is stored, the delta value calculator retrieves the related data item from the data store and calculates a delta value from the received data item and the related data item. If the related item is not stored, then the delta value is calculated from the received data item and a predetermined value. A data store controller accesses the data store in response to receipt of a data item and determines if a storage location is allocated to the data item. If there is an allocated storage location for the data item, the data item is stored in the allocated storage location; and if not then a storage location is allocated to the data item.