Change Log Process Scaling for Distributed Systems

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed storage systems face challenges in providing real-time change notifications due to increased demand, leading to system overload and inconsistent data snapshots for watchers.

Innovation Solution

Implementing a method that dynamically manages change log processes by executing multiple instances when the load exceeds a threshold, merging transaction histories, and using notification locks to ensure transactional consistency and real-time notifications.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If multiple change log process instances are executed to handle high demand, then system scalability and notification throughput are improved, but system complexity and merging overhead increase

Engineering Contradiction:

Improvenotification throughputVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The change log process is segmented into multiple independent instances that can be executed in parallel. Each instance handles a portion of the transaction load independently, allowing the system to scale horizontally by adding more instances when demand increases, thereby improving notification throughput without requiring a complete system redesign

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Multiple change log process instances are merged by consolidating their respective transaction histories into a unified view. The system implements merging logic that combines transactions from different instances while maintaining consistency, allowing parallel processing benefits to be realized without sacrificing data integrity or requiring complex coordination protocols

Inventive Principle:

Principle #5Merging (Combining)

2Reliability

If change log load is distributed across multiple instances, then system overload is prevented, but transaction history merging overhead increases

Engineering Contradiction:

Improvesystem stabilityVSAvoidmerging overhead

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

Transaction histories are prepared and organized in advance within each change log instance before merging is needed. Each instance maintains its transaction history in a standardized format with pre-computed metadata, so that when merging is required, the consolidation process can proceed efficiently without extensive processing overhead

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Instead of physically moving or duplicating entire transaction histories during merging, the system uses lightweight references or pointers to transaction data across instances. This allows multiple instances to share transaction history information without the overhead of actual data copying, reducing merging time while maintaining data consistency

Inventive Principle:

Principle #26Copying

3Reliability

If change logs are truncated onto a single machine, then durability is ensured, but system scalability is reduced due to bottlenecking

Engineering Contradiction:

Improvechange log durabilityVSAvoidtransaction throughput

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The durable change log functionality is segmented across multiple independent process instances rather than being centralized on a single machine. Each instance maintains its own durable transaction history locally, eliminating the single-machine bottleneck while preserving durability guarantees through local persistence mechanisms

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

An intermediary merging mechanism is introduced that coordinates between multiple distributed change log instances. This mediator layer handles the consolidation of transaction histories from multiple instances while ensuring durability, allowing the system to maintain strong consistency guarantees without requiring all transactions to pass through a single centralized log

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS20250190420A1Real-time transactionally consistent change notifications

Publication Date: 2025.06.12 GOOGLE LLC

AI summary

A method includes executing an initial instance of a change log process for a distributed system, each instance of the change log process configured to store a transaction history of transactions executed on the distributed system. The method also includes receiving transaction requests for executing corresponding transactions on the distributed system and determining a change log load based on the received transaction requests. The method includes executing at least one subsequent instance of the change log process when the change log load satisfied a threshold load. When multiple instances of the change log process are executing, the method includes ceasing execution of the at least one subsequent instance of the change log process and merging the transaction history of the initial instance of the change log process and the transaction history of the at least one subsequent instance of the change log process.