In-Memory Key-Value Block Compression for Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional in-memory key-value systems inefficiently manage memory consumption by storing each key-value pair individually without compression, leading to increased indexing data structure sizes and metadata overhead, and lack effective methods for compressing continuous key-value pairs to reduce storage footprint.
Innovation Solution
A method and system that divide key-value pairs into segments, create blocks, and compress each block, allowing for efficient storage and retrieval by offloading decompression to the front-end server, reducing memory consumption and metadata overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is stored as individual key-value pairs without compression, then retrieval operations are simple and direct, but memory consumption and metadata overhead increase significantly
Solution Approach 1:
The patent segments continuous key-value pairs into fixed-size blocks, where each block contains multiple key-value pairs. This segmentation allows the system to store and manage data in manageable units while enabling compression at the block level, thereby reducing overall memory consumption without completely sacrificing retrieval efficiency.
Solution Approach 2:
The patent changes the storage parameter from individual key-value pairs to compressed blocks of key-value pairs. By transforming the data structure from uncompressed individual entries to compressed blocks, the system achieves significant memory savings while maintaining the ability to perform retrieval operations through block decomposition when necessary.
2Measurement precision
If exact lookup indexing is implemented for each key-value pair, then retrieval precision is high, but the size of indexing data structures increases
Solution Approach 1:
The patent segments the indexing structure to operate at the block level rather than individual key-value pair level. Each block maintains its own indexing information, allowing the system to provide precise lookup within blocks while reducing the overall size of indexing structures by eliminating the need for separate indexes for every individual key-value pair.
Solution Approach 2:
The patent introduces a new dimensional approach by organizing data in blocks with block-level indexing, adding a layer of abstraction between the individual key-value pairs and the indexing structure. This dimensional change allows the system to maintain lookup precision through block-level indexes while significantly reducing the total volume of indexing data.
3Reliability
If data is stored in the received form without compression, then data integrity is maintained, but storage footprint is large
Solution Approach 1:
The patent changes the storage parameter from uncompressed individual key-value pairs to compressed blocks. By applying compression algorithms at the block level while maintaining the original data within those blocks, the system achieves smaller storage footprints while preserving data integrity, as the compressed blocks can be decompressed to retrieve the original data when needed.
4Ease of operation
If individual key-value pairs are managed separately, then access control is precise, but metadata overhead increases
Solution Approach 1:
The patent segments the data management unit from individual key-value pairs to blocks containing multiple key-value pairs. This segmentation reduces metadata overhead by eliminating redundant management information for individual pairs, while access control is maintained at the block level with the ability to access specific key-value pairs within authorized blocks.
Solution Approach 2:
The patent merges multiple individual key-value pairs into blocks, combining their management under a single block-level metadata structure. This merging eliminates the need for separate metadata for each individual pair, significantly reducing overall metadata overhead while maintaining the ability to control access to the contained key-value pairs.
Data Source
AI summary
A method for compressing a group of key-value pairs, the method including dividing the group of key-value pairs into a plurality of segments, creating a plurality of blocks, each block of the plurality of blocks corresponding to a segment of the plurality of segments, and compressing each block of the plurality of blocks.