In-Place Data Compression Using Small Working Memory Buffers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current compression and decompression systems require distinct source and target buffers in memory, making it impossible to compress data in-place when system memory is overcommitted, as they need additional memory pages which may not be available.
Innovation Solution
An in-place compression system that uses a small working memory with a data buffer and an overflow buffer to compress source data stored in a memory page by temporarily storing it in the data buffer, writing compressed data back to the same page, and aborting compression if there's insufficient space, restoring the original data to maintain memory efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If distinct source and target buffers are used for compression, then compression can be performed reliably, but additional memory pages are required which are not available when system memory is overcommitted
Solution Approach 1:
The patent merges the source buffer and target buffer into a single memory page, allowing compression to be performed in-place. The source data and compressed data coexist in the same memory space, with the source data being read sequentially and compressed data being written to the same page as the source data is being processed.
Solution Approach 2:
The patent introduces a data buffer as an intermediary structure within the single memory page. The data buffer temporarily holds source data that has been read but not yet processed, and also holds uncompressed data that needs to be written back. This intermediary buffer enables the compression operation to proceed without requiring separate source and target pages.
2Quantity of substance
If in-place compression is performed with a small data buffer, then memory efficiency is improved, but the system may encounter insufficient space when write address catches up to read address
Solution Approach 1:
The patent implements a feedback mechanism by continuously monitoring the relationship between the write address and read address. When the write address catches up to or exceeds the read address, indicating insufficient space in the data buffer, the system aborts the compression operation and restores the original data. This feedback loop ensures data integrity while operating with limited buffer space.
Solution Approach 2:
The patent performs partial compression action by processing only the portion of data that can be safely compressed within the available buffer space. When the buffer becomes full, the compression is aborted and the original data is restored, rather than attempting to compress the entire data set. This partial action approach prevents data loss while maximizing the use of limited buffer space.
3Quantity of substance
If compression header is stored in the same memory page, then memory usage is reduced, but the header may overwrite source data that has not been transferred to the data buffer
Solution Approach 1:
The patent performs preliminary action by first transferring the compression header to the overflow buffer before writing it to the final destination. This preliminary placement in the overflow buffer ensures that the header does not overwrite source data that has not yet been processed. The header is then written to the memory page only after sufficient source data has been transferred to the data buffer, preventing data loss.
Data Source
AI summary
Method and apparatus for performing in-place compression is provided. The in-place compression system transfers source data from a partition of a memory to a data buffer based on a read address. Compressed data is created by referencing the source data stored in the data buffer. The system writes the compressed data to the memory partition based on a write address. When the write address points at an address location that stores source data that has not been transferred to the data buffer, the system overwrites the compressed data stored in the memory partition with the source data stored in the data buffer.