Dictionary-Based Compression with Dynamic Retraining
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data compression techniques, particularly dictionary-based methods, face inefficiencies due to stale dictionaries and the resource-intensive reference counting mechanism, which degrades their effectiveness over time and increases computational burden.
Innovation Solution
Implement dynamic dictionary retraining and store dictionaries within containers alongside the compressed data, eliminating the need for reference counting by associating each container with a dictionary and using existing garbage collection mechanisms to manage orphan dictionaries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a dictionary is trained using sample data and used to compress new data, then compression ratios are improved for small data sets, but the dictionary becomes stale and ineffective when new data differs significantly from sample data
Solution Approach 1:
The patent implements dynamic dictionary retraining by monitoring compression ratios and automatically retraining the dictionary when performance degrades below a threshold. This transforms the static dictionary approach into a dynamic system that adapts to changing data characteristics, resolving the contradiction between initial compression improvement and long-term effectiveness.
Solution Approach 2:
The system continuously monitors compression ratios as feedback on dictionary performance. When the compression ratio falls below a predetermined threshold, this feedback triggers automatic dictionary retraining. This closed-loop feedback mechanism ensures the dictionary remains effective while maintaining high compression ratios.
2Reliability
If a reference count mechanism is used to track data blocks compressed using a particular dictionary, then dictionary removal is prevented when data blocks exist, but complexity and overhead are significantly increased
Solution Approach 1:
The patent extracts the reference counting functionality from the compression system by storing dictionary metadata directly within each compressed data block. This eliminates the need for separate reference count tracking mechanisms, reducing system complexity while maintaining accurate dictionary management through the metadata embedded in the data blocks themselves.
Solution Approach 2:
The compressed data blocks carry their own dictionary metadata, making them self-sufficient for dictionary identification and management. This self-service approach eliminates the need for external reference counting infrastructure, simplifying the storage system while ensuring accurate tracking of dictionary usage through the embedded metadata.
Data Source
AI summary
Methods, non-transitory machine readable media, and computing devices that provide improved dictionary-based compression are disclosed. With this technology, a first portion of an input data stream is compressed using a first dictionary. A second dictionary is trained when the first dictionary is determined to be stale. The dictionary can be determined to be stale based on a size of the input data stream compressed using the first dictionary or a compression ratio decreasing by a threshold, for example. The first dictionary can be stored with metadata associated with the compressed first portion of the input data stream. Accordingly, this technology improves compression ratios, eliminates the need for reference counting, and facilitates improved reclamation of orphan dictionaries, among other advantages.