Shared Compression Dictionary Versioning for Distributed Database Writes

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Managing shared compression dictionaries across database servers and clients is complex, and there is a risk of data loss due to expired or lost dictionaries in client-side compression, leading to inefficient compression ratios and resource strain.

Innovation Solution

A computer-implemented method manages shared compression dictionaries on the database server side, ensuring all clients use the correct version by designating a current primary version and expiring old versions, reducing the risk of data loss and optimizing storage.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of manufacture

If client-side compression is used, then CPU resources are cheaper and easier to scale, but compression ratios are lower and individual clients lack knowledge of remaining database data

Engineering Contradiction:

ImproveCPU resource scalabilityVSAvoidcompression ratio

Core Design Contradiction:

Ease of manufactureVSProductivity

Solution Approach 1:

A dictionary management service is introduced as an intermediary between clients and database servers. This service maintains a centralized dictionary store that clients can access, allowing them to perform client-side compression with improved ratios by using dictionaries trained on global database data patterns, without requiring direct server-side compression

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If server-side compression is used, then compression ratios are better and servers have more physical memory, but CPU resources are consumed and network bandwidth is strained

Engineering Contradiction:

Improvecompression ratioVSAvoidCPU resource consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The dictionary management service acts as an intermediary that prepares compression dictionaries centrally using server resources, then distributes them to clients. This allows clients to perform compression locally with high ratios using pre-trained dictionaries, eliminating the need for servers to perform real-time compression and reducing CPU and network strain

Inventive Principle:

Principle #24Intermediary (Mediator)

3Ease of operation

If compression dictionaries are managed by individual entities, then clients can use dictionaries for compression, but there is risk of data loss when dictionaries are expired or lost

Engineering Contradiction:

Improvedictionary-based compressionVSAvoiddata loss risk

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The dictionary management service serves as a centralized intermediary that maintains authoritative copies of all compression dictionaries. Clients request dictionaries from this service rather than managing them independently, ensuring that the correct dictionary versions are always available for decompression and eliminating data loss risks from local dictionary expiration or loss

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system implements a versioning mechanism where old dictionary versions are not immediately discarded but maintained in the centralized store until superseded by newer versions. This allows safe replacement of dictionaries over time while ensuring that any dictionary needed for decompression remains available, preventing data loss

Inventive Principle:

Principle #34Discarding and recovering

Data Source

PatentUS20250330198A1Cooperative compression in distributed databases

Publication Date: 2025.10.23 NETFLIX INC

AI summary

In various embodiments a computer-implemented method for managing use of a shared compression dictionary in a distributed database environment. The method includes determining that a given version of the shared compression dictionary should be designated as a current primary version of the shared compression dictionary. The method also includes receiving, from a client device, first write data compressed with a previous primary version of the shared compression dictionary and in response to receiving the first write data, transmitting, to the client device, the current primary version of the shared compression dictionary and an instruction to compress new write data with the current primary version of the shared compression dictionary. Additionally, the method includes receiving, from the client device, a second write data compressed with the current primary version of the shared compression dictionary and storing the second write data in a database.