Database Key Compression That Preserves LSM Tree Sort Order
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large database systems face a significant storage footprint challenge due to the large size of database keys, which can be mitigated by compressing these keys while maintaining their sorted order, especially in log-structured merge-tree (LSM tree) systems that require key-sorted order for efficient data management.
Innovation Solution
A computer system compresses database keys by replacing portions with data identifying the location and similarity to a reference key, allowing for efficient storage and decompression, and implements a key compression engine to manage this process, ensuring that compressed keys maintain the same sorted order as their uncompressed counterparts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If database keys are compressed to reduce storage footprint, then storage efficiency is improved, but key retrieval and decomposition complexity increases
Solution Approach 1:
The database key is segmented into multiple components: a reference key portion and a delta portion. The compression engine divides the key storage into reference keys stored in an array and compressed keys that reference these array positions, reducing overall storage while maintaining retrieval capability through structured segmentation.
Solution Approach 2:
A reference key array serves as an intermediary between the compressed key storage and the original key representation. Instead of storing full keys, the system stores references to the reference key array, using this intermediary structure to achieve compression while enabling efficient key reconstruction when needed.
2Quantity of substance
If compressed keys are stored in LSM tree structures, then storage space is reduced, but maintaining sorted order becomes more difficult
Solution Approach 1:
The system changes the parameter representation of keys by storing only the delta portion and reference position instead of complete key values. This parameter transformation allows compressed keys to maintain sorted order properties because the compression preserves the relative ordering relationship between keys through consistent reference key selection and delta calculation.
3Productivity
If full database keys are stored, then key retrieval is simple and fast, but storage footprint increases significantly
Solution Approach 1:
Instead of storing complete key copies in every database record, the system creates compressed representations that copy only essential portions (delta values and reference positions). The full key can be reconstructed by copying the reference key from the array and applying the delta, achieving space efficiency while maintaining retrieval capability.
Data Source
AI summary
Techniques are disclosed relating to compressing database keys. A computer system may receive a request to write a database record to a storage medium. The database record may include a database key and a corresponding data value. The computer system may compress the database key by replacing a portion of the database key with particular data that identifies a location of a reference database key and an amount of similarity determined between the database key and the reference database key. The computer system may write the database record to the storage medium. The database record may include the compressed database key and the corresponding data value.