Transactional Replicator Group Commit Barrier

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current database transaction systems lack efficient mechanisms for high availability and consistency in distributed environments, particularly in handling failures and data replication across multiple state providers, leading to potential data loss and downtime during backups and restores.

Innovation Solution

A transactional replicator system that employs write-ahead-logging with group commit and barrier processing to ensure atomicity and durability across multiple state providers, allowing for concurrent transactions and minimal resource usage during backups, and enabling consistent state replication and restoration without downtime.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional transactional semantics are used to guarantee ACID properties for a single database, then data consistency is maintained, but system availability and concurrency are reduced

Engineering Contradiction:

Improvedata consistencyVSAvoidsystem availability

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system segments transaction commit operations into individual transaction commits and group commits. Individual commits handle single transactions immediately for high availability, while group commits batch multiple transactions for optimized consistency operations across multiple state providers, resolving the contradiction between immediate availability and systematic consistency

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The barrier mechanism performs preliminary actions by establishing commit boundaries before transactions are fully committed. This preliminary barrier setup allows the system to pre-organize transaction groups and prepare consistency checkpoints, enabling subsequent high-concurrency operations to proceed efficiently without compromising ACID properties

Inventive Principle:

Principle #10Preliminary action

2Reliability

If multiple state providers are used for data replication, then availability is improved, but complexity of managing consistency across providers increases

Engineering Contradiction:

Improvesystem availabilityVSAvoidconsistency management complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The transactional replicator acts as an intermediary between multiple state providers, centralizing consistency management. It receives transactions, determines appropriate commit strategies (individual vs. group), and coordinates across providers, thereby reducing the complexity that would otherwise be distributed and difficult to manage across multiple providers

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

Barriers are established preliminarily to define transaction groups before replication occurs. This preliminary organization allows the system to pre-determine which transactions should be grouped together for atomic commitment across multiple state providers, simplifying the coordination complexity by establishing rules in advance rather than managing them dynamically

Inventive Principle:

Principle #10Preliminary action

3Reliability

If backups and restores are performed, then data safety is improved, but downtime and resource usage increase

Engineering Contradiction:

Improvedata safetyVSAvoiddowntime

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system enables continuous transaction processing during backup and restore operations by using asynchronous replication and WAL. The barrier mechanism allows transactions to continue being accepted and logged while backups occur in the background, and restores can be performed incrementally without requiring complete system downtime, thus maintaining continuity of useful action

Inventive Principle:

Principle #20Continuity of useful action

4Productivity

If group commit is used to commit multiple transactions, then efficiency is improved, but risk of data loss increases if failure occurs

Engineering Contradiction:

Improvecommit efficiencyVSAvoiddata loss risk

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system applies beforehand cushioning through WAL (write-ahead logging) that records all transactions before group commit. This creates a protective cushion: if a group commit fails, the WAL ensures that no committed transactions are lost, as they are already recorded in the write-ahead log. This allows efficient group commits while cushioning against the inherent risk of batch operations

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

Data Source

PatentUS11269927B2Transactional replicator

Publication Date: 2022.03.08 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

A transactional replicator applying group commit and barrier concepts is disclosed. Group commit means that the transactional replicator commits multiple transactions in a group and is not restricted to committing single transactions one-at-a-time and is not limited to operating on a single state provider. Barrier means that the transactional replicator does not move forward to commit additional transactions until the previous group of transactions are completed. All state providers must apply their transactions and update state before additional transactions will be committed. A quorum acknowledgement “unlocks” any locks that were acquired to update the state within a state provider. However, as long as there are no lock conflicts, additional transactions can continue to create new group commits as necessary.