Variable-Width Dictionary Compression With Reused Code Partitions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing lossless data compression techniques using variable width codes face challenges in achieving high compression ratios while managing dictionary entries efficiently, leading to rapid increases in dictionary size and unnecessary code width.
Innovation Solution
The proposed solution involves dividing the dictionary into two partitions and reusing unused codes from the second partition when sufficient unassigned codes are not available, allowing for the use of smaller width codes to represent longer strings and dynamically adjusting code width by merging entries from the extended partition to the basic partition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If variable width codes are used to represent repeating strings, then compression ratio is improved, but dictionary size increases rapidly
Solution Approach 1:
The dictionary is divided into two separate partitions: a first partition for storing initial dictionary entries and a second partition for storing extended dictionary entries. This segmentation allows independent management of code assignments in each partition, enabling reuse of codes from the second partition when the first partition exhausts its unassigned codes, thereby controlling overall dictionary size while maintaining compression efficiency.
Solution Approach 2:
The system recovers and reuses codes from the second partition when the first partition runs out of unassigned codes. Instead of continuously expanding the dictionary, unused codes in the second partition are made available for reassignment to new strings, preventing unnecessary dictionary growth while maintaining the ability to represent repeating patterns efficiently.
2Productivity
If more dictionary entries are added to represent longer strings, then compression efficiency improves, but code width increases unnecessarily
Solution Approach 1:
The code width is made dynamic rather than fixed. The system determines the minimum code width required based on the number of unassigned codes available in the dictionary partitions. When fewer codes are needed, smaller code widths are used; when more codes are required, the width expands. This dynamic adjustment prevents unnecessary increases in code width while maintaining the ability to represent all necessary dictionary entries.
Solution Approach 2:
The system changes the parameter of code width based on the current state of the dictionary. By monitoring the number of unassigned codes and adjusting the code width parameter accordingly, the system optimizes the balance between representing longer strings efficiently and avoiding excessive code width. This parameter adjustment occurs dynamically as the dictionary is populated during compression.
Data Source
AI summary
An aspect of the present invention provides for lossless compression of an input stream using variable width codes. In an embodiment, in each of a sequence of iterations, an output code is emitted (to an output stream) corresponding to a longest string from a present symbol based on entries in a dictionary. In each iteration, two entries are thereafter added to the dictionary, with the first entry associating (the longest string+symbol succeeding said longest string in said input stream) to a first available code in the dictionary, and a second entry associating a (previous token+symbol succeeding said previous token in said input stream) to a second available code in the dictionary, wherein the previous token represents the token of a first entry in a previous iteration, and wherein ‘+’ represents a text appending operation.