Shared Compression Dictionary Versioning in Distributed Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing shared compression dictionaries across database servers and clients in distributed databases is complex, leading to issues like data loss due to expired dictionaries and inefficient compression ratios, particularly for small data blocks.
Innovation Solution
A method where a dictionary management server manages shared compression dictionaries on the server side, ensuring all clients use the correct version by transmitting the current primary dictionary and expiring outdated ones, allowing for deterministic re-writing of data with newer versions during compaction processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If server-side compression is used, then compression ratio is improved, but CPU resource consumption increases
Solution Approach 1:
The patent introduces a shared compression dictionary as an intermediary between the client and server. The dictionary is maintained on the server side and shared with clients, enabling clients to perform compression locally using the server-maintained dictionary. This intermediary structure allows clients to achieve better compression ratios without requiring high CPU resources at the server side, as the compression work is distributed to clients who use the shared dictionary for efficient local compression.
2Use of energy by moving object
If client-side compression is used, then CPU resource consumption is reduced, but compression ratio deteriorates
Solution Approach 1:
The shared compression dictionary serves multiple functions: it is used by the server for maintaining and updating the dictionary, by clients for compression, and by both sides for decompression. This universal resource enables clients to perform compression with better ratios than traditional client-side compression while keeping CPU consumption low, as the dictionary provides pre-computed compression patterns that clients can apply efficiently without re-analyzing the data.
3Device complexity
If individual clients manage their own compression dictionaries, then device complexity is reduced, but data reliability decreases
Solution Approach 1:
The server acts as an intermediary that maintains and manages the shared compression dictionary. Instead of each client independently managing their own dictionaries, the server centralizes dictionary management and shares it with all clients. This intermediary approach reduces client complexity while ensuring data reliability, as the server can maintain dictionary versions and ensure consistency across all clients, preventing data loss from mismatched compression/d decompression operations.
4Reliability
If server maintains shared compression dictionary, then data reliability is improved, but device complexity increases
Solution Approach 1:
The system implements self-service mechanisms where the server automatically manages dictionary versions, tracks which clients have which versions, and handles dictionary updates without manual intervention. The server can autonomously determine when to create new dictionary versions, distribute them to clients, and manage expiration of old versions. This self-service approach reduces the operational complexity burden on administrators while maintaining high data reliability through centralized control.
Data Source
AI summary
In various embodiments a computer-implemented method for managing use of a shared compression dictionary in a distributed database environment. The method includes determining that a given version of the shared compression dictionary should be designated as a current primary version of the shared compression dictionary. The method also includes receiving, from a client device, first write data compressed with a previous primary version of the shared compression dictionary and in response to receiving the first write data, transmitting, to the client device, the current primary version of the shared compression dictionary and an instruction to compress new write data with the current primary version of the shared compression dictionary. Additionally, the method includes receiving, from the client device, a second write data compressed with the current primary version of the shared compression dictionary and storing the second write data in a database.