Consistent View Generation for Eventually Consistent Databases

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing techniques for constructing a consistent view of an eventually consistent database often result in increased latency and accuracy issues due to the need for additional replication operations and the risk of providing inaccurate data.

Innovation Solution

A computer-implemented method that determines backup files from multiple datacenters, extracts aggregated data from these files, performs compaction operations to generate consistent data items, and generates a consistent view of the database based on the compacted data without directly accessing the database.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a read-only datacenter is added to store a single replica of each data item, then a consistent view of the database is provided, but the load on existing datacenters increases and latency increases

Engineering Contradiction:

Improveconsistency of database viewVSAvoidlatency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system pre-generates consistent snapshots of the database by collecting data from multiple replicas before they are needed. These snapshots are created in advance using background processes that gather data from all datacenters, perform compaction operations to resolve inconsistencies, and store the resulting consistent views for later retrieval, eliminating the need for real-time consistency checks

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces an intermediary component that acts as a mediator between the eventually consistent database and the read-only datacenter. This intermediary collects data from multiple replicas across different datacenters, performs compaction operations to resolve inconsistencies, and generates consistent snapshots that can be served without increasing load on original datacenters

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If additional replication operations are performed to maintain a read-only datacenter, then a consistent view is achieved, but the load on existing datacenters increases

Engineering Contradiction:

Improveconsistency of database viewVSAvoidload on datacenters

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system performs data collection and compaction operations in advance during background processing cycles, rather than on-demand. This preliminary action creates a pool of pre-processed consistent snapshots that can be served without requiring additional real-time replication operations, thereby reducing the load on existing datacenters

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates copies of data from multiple replicas and processes these copies independently to generate consistent snapshots. By working with copies rather than original data, the system avoids placing additional load on the production datacenters while still achieving consistency through compaction operations on the copied data

Inventive Principle:

Principle #26Copying

3Productivity

If data is replicated across multiple datacenters for availability, then constant availability is ensured, but temporary inconsistencies between replicas occur

Engineering Contradiction:

Improveavailability of databaseVSAvoidconsistency of data

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system performs compaction operations in advance to resolve inconsistencies between replicas before the data is needed. By pre-processing the data from multiple replicas and generating consistent snapshots, the system eliminates the need for real-time consistency checks while maintaining high availability

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces an intermediary layer that collects data from multiple inconsistent replicas, performs compaction operations to resolve conflicts, and generates consistent views. This intermediary acts as a mediator that reconciles the inconsistencies created by distributed replication while preserving the availability benefits of having multiple replicas

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12204521B2Techniques for generating a consistent view of an eventually consistent database

Publication Date: 2025.01.21 NETFLIX INC

AI summary

In various embodiments, a consistency application constructs a consistent view of an eventually consistent database. The consistency application determines multiple backup files that are associated with at least one datacenter included in the eventually consistent database and extracts aggregated data from the backup files. The consistency application performs compaction operation(s) on the aggregated data to generate compacted data. Notably, the aggregated data includes at least two replicas for each data item stored in the eventually consistent database, whereas the compacted data includes a different consistent data item for each data item stored in that eventually consistent database. The consistency application generated the consistent view of the eventually consistent database based on the compacted data. Because the consistency application generates the consistent view based on backup files and does not access the eventually consistent database, generating the consistent view does not adversely impact the performance of the eventually consistent database.