Server-Side Unified Dictionary for Key-Value Data Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing key-value storage systems face inefficiencies in data compression, as they typically compress data on client devices, leading to high time consumption and additional storage space requirements due to the need for generating a compression dictionary for each piece of data, which results in low overall compression efficiency and increased costs.
Innovation Solution
The proposed solution involves generating a unified dictionary on a server based on a large amount of key-value data, using training and verification data to select the most efficient dictionary, which is then applied to all data, allowing for efficient data compression without the need for individual dictionaries for each piece of data, and automatically updating the dictionary when compression efficiency decreases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If dictionary compression is performed on each piece of data on the client device, then data can be compressed before transmission, but the time consumption and latency of compression increase due to high-frequency word analysis required to generate a corresponding dictionary for each piece of data
Solution Approach 1:
The patent segments the compression process into two phases: (1) offline dictionary generation phase where a comprehensive dictionary is built from historical data on the server, and (2) online compression phase where the pre-built dictionary is applied to compress incoming data. This segmentation eliminates the need for real-time dictionary generation during compression, thus resolving the time consumption issue while maintaining compression effectiveness.
Solution Approach 2:
The patent applies preliminary action by pre-generating compression dictionaries offline using historical data before actual compression operations. The dictionary is built in advance and stored, so when compression is needed, the system can directly use the pre-prepared dictionary without performing time-consuming word analysis during the compression process itself.
2Loss of substance
If a compression dictionary is generated for each piece of data, then the compression can be tailored to specific data characteristics, but additional storage space is required to store the compression dictionary with the compressed data
Solution Approach 1:
The patent makes the compression dictionary universal by building it from aggregated historical data that represents the entire dataset's characteristics. This single universal dictionary can be applied to compress any piece of data in the system, eliminating the need for multiple individual dictionaries. The dictionary serves multiple purposes across different data pieces, thus reducing storage space requirements.
Solution Approach 2:
The patent merges the dictionary generation process from individual data pieces into a unified process that aggregates data from the entire system to create a single comprehensive dictionary. Instead of having separate dictionaries for each data piece, the system combines information from all historical data to build one master dictionary that captures overall patterns, thereby reducing total storage space.
3Ease of operation
If dictionary compression is performed on small pieces of data, then the compression can be applied to individual data units, but the generated compression dictionary has only a few high-frequency words and compression is inefficient
Solution Approach 1:
The patent uses preliminary action by accumulating historical data over time and performing dictionary generation offline before actual compression operations. This allows the system to build a rich dictionary with many high-frequency words by analyzing a large corpus of data, rather than trying to build a dictionary from a single small data piece. The pre-built dictionary contains comprehensive word patterns that improve compression efficiency.
Solution Approach 2:
The patent segments the data accumulation process from the dictionary generation process. Instead of generating a dictionary from each small data piece, the system first accumulates large amounts of historical data, then segments this accumulated data for offline dictionary generation. This segmentation allows the dictionary to be built from sufficient data to capture all high-frequency words and patterns, improving compression efficiency while maintaining applicability to individual small data units.
Data Source
AI summary
Methods, systems, and apparatus for, for data compression based on a key-value store. In one aspect, a method includes generating, at a server, a current dictionary based on a plurality of key-values stored in a storage system of the server; receiving a key-value pair transmitted by a client device; and performing, at the server, data compression on a key-value in the key-value pair by using the current dictionary; and storing the key-value in the storage system of the server.