Disjoint Partitioned Log Records for High Throughput Data Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for data replication and transformation in information systems face high implementation costs, performance issues due to latency, and inability to fully parallelize transactions, leading to suboptimal throughput and availability.
Innovation Solution
A method that generates log records with sequence identifiers, target keys, and source property values, allowing for disjoint partition assignment and transmission control, enabling high throughput and reliable replication while maintaining data integrity through optimized processing and transformation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional transaction serialization (two-phase commit) is used to ensure data integrity, then data integrity is improved, but throughput deteriorates due to latency and increased serialization
Solution Approach 1:
The patent divides the replication workload into disjoint partitions that can be processed independently by multiple processing instances. Each partition is assigned to a specific processing instance, allowing parallel execution without interfering with each other. This segmentation enables high throughput while maintaining data integrity through ordered processing within each partition.
Solution Approach 2:
The patent introduces a partitioning dimension to the traditional sequential transaction processing model. By organizing transactions into partitions and assigning them to multiple processing instances, the system transitions from single-threaded serialization to multi-dimensional parallel processing, significantly improving throughput while preserving data integrity through controlled ordering.
2Reliability
If strong data integrity protocols (two-phase commit) are implemented, then data integrity is improved, but system availability deteriorates due to increased dependency on complete system availability
Solution Approach 1:
The patent segments the replication process into independent partitions that can be processed concurrently by multiple instances. This reduces the dependency chain for data integrity verification, allowing the system to maintain integrity even when individual components experience availability issues, thereby improving overall system stability.
3Ease of manufacture
If conventional replication systems are used, then implementation costs are reduced (using existing systems), but programming effort and integration complexity increase due to system-specific target calls
Solution Approach 1:
The patent creates a universal replication framework that can handle multiple target systems through a common interface. The system uses a standardized log record format and processing mechanism that works across different target systems, eliminating the need to program system-specific target calls for each deployment scenario.
Solution Approach 2:
The patent employs a generic log record structure that copies the essential transformation logic across different target systems. Instead of writing separate integration code for each target system, the same log record processing mechanism can be applied universally, significantly reducing programming effort and integration complexity.
4Reliability
If conventional systems process transactions sequentially, then data integrity is maintained, but processing time increases and parallelization is prevented
Solution Approach 1:
The patent segments transactions into disjoint partitions that can be processed in parallel by multiple processing instances. Each partition maintains its own ordering guarantees, allowing concurrent processing that reduces overall processing time while preserving data integrity through controlled partition boundaries.
Solution Approach 2:
The patent adds a partitioning dimension to transaction processing, enabling multiple transactions to be processed simultaneously in different partitions. This dimensional approach to parallelization reduces processing time without compromising data integrity, as each partition maintains ordered processing for its assigned transactions.
Data Source
AI summary
Provided are techniques for, in response to a source object being changed by a local transaction, generating log records, wherein each of the log records includes a sequence identifier, a source system entity modification timestamp, a target key, a target partition identifier, and values of source properties, wherein the source properties are identified using a target key indicator. The log records to process are selected based on a phase value of each log record, wherein the phase value is one of submitted, in-flight, and source system entity state change complete. One or more disjoint partitions are assigned to each of multiple processing instances. Under control of each of the multiple processing instances, it is determined whether to transmit a log record in the one or more disjoint partitions assigned to said processing instance and, in response to determining that the log record is to be transmitted, the log record is transmitted.