Distributed Bigtable Aggregation Resets Through Changelog Replay

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Multi-master distributed storage systems struggle with inconsistent aggregation value resets due to replication delays, leading to inconsistencies among replicas.

Innovation Solution

An aggregation controller using changelogs and replication watermarks to manage updates and resets in a log-structured merge-tree architecture, ensuring consistent aggregation value resets without significant computational overhead.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If multi-master distributed storage systems allow users to modify stored aggregate values via append-like operations, then aggregation updates can be performed, but inconsistent aggregation value resets occur due to replication delays

Engineering Contradiction:

Improveconsistency of aggregation value resetsVSAvoidreplication delay

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by recording all update operations in a changelog before applying them. When a reset operation is needed, the system uses the changelog to replay only the necessary updates after the reset, ensuring consistency without waiting for all updates to replicate first. This resolves the contradiction by preparing update records in advance and applying them selectively based on timestamps.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements feedback mechanisms through timestamp-based ordering and changelog replay. Each update operation includes a timestamp that serves as feedback about its timing. The system uses this feedback to determine which updates should be applied after a reset, ensuring that resets are applied consistently across all replicas while accounting for replication delays.

Inventive Principle:

Principle #23Feedback

2Reliability

If the system stores all update operations in a changelog, then consistent resets can be achieved, but storage space and computational overhead increase

Engineering Contradiction:

Improveconsistency of aggregation value resetsVSAvoidchangelog management complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system extracts and separates the changelog management function from the main aggregation storage. The changelog is maintained as a separate data structure that records only the necessary update operations. This extraction allows the system to manage complexity by isolating changelog operations from the main aggregation value storage, enabling efficient resets without processing the entire history of operations.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system changes parameters by using timestamp-based filtering and selective replay of updates. Instead of processing all historical updates, the system filters updates based on timestamps relative to the reset operation. This parameter change approach reduces the computational overhead by limiting the scope of updates that need to be replayed after a reset, while maintaining consistency.

Inventive Principle:

Principle #35Parameter changes

3Productivity

If the system compacts and trims unnecessary log entries, then state storage is limited and system efficiency is maintained, but operations to compact and trim increase computational overhead

Engineering Contradiction:

Improvesystem efficiencyVSAvoidcompaction and trimming operations

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system implements periodic compaction and trimming operations that run at scheduled intervals or triggered by specific conditions. Rather than continuously maintaining a compacted structure, the system performs compaction periodically to manage growth of the changelog. This periodic action reduces the computational overhead by concentrating compaction work into discrete operations rather than continuous processing, while maintaining system efficiency.

Inventive Principle:

Principle #19Periodic action

Solution Approach 2:

The system maintains continuity of useful action by keeping the changelog updated with recent operations while periodically removing unnecessary historical entries. The compaction and trimming operations continue the useful action of maintaining a manageable changelog size. This continuous management ensures the system remains efficient without requiring complete reprocessing of historical data, balancing the computational overhead of compaction with the benefits of limited state storage.

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS12461904B2Distributed aggregations in bigtable

Publication Date: 2025.11.04 GOOGLE LLC

AI summary

A method includes receiving, at a first node of a plurality of nodes of a distributed database, an update request for an aggregate value stored at the first node. The update request includes a first sequencer. Based on receiving the update request for the aggregate value, the method includes updating the aggregate value based on the update request, storing a log record of the update to a changelog, and transmitting the update request to each other node. After updating the aggregate value, the method includes receiving, from a second node, a reset request for the aggregate value. The reset request includes a second sequencer. The method includes determining that the reset request predates the update request. Based on determining that the reset request predates the update request, the method includes resetting the aggregate value and updating the aggregate value based on the update request.