Hash-Based Stream Dispatching for Causal Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional transactional replication methods face throughput limitations and causal consistency constraint violations due to applying data changes out of order when using multiple streams for replication.
Innovation Solution
A computer application generates hash values for each data change's constraints and maintains a hash table to ensure conflicting changes are dispatched into the same stream, maintaining their original order and preventing consistency violations, while distributing non-conflicting changes evenly across streams.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data changes are dispatched through multiple streams for parallel replication, then throughput is improved, but causal consistency constraints may be violated due to out-of-order application
Solution Approach 1:
The patent segments data changes into multiple streams for parallel processing, where each stream handles a subset of data changes. By dividing the replication workload into separate streams that can be processed independently and in parallel, the system achieves higher throughput while maintaining consistency within each stream through proper ordering mechanisms.
Solution Approach 2:
The system performs preliminary actions by pre-processing data changes to identify and mark conflicting changes before dispatching them to streams. This includes generating hash values for constraints, identifying conflicts ahead of time, and preparing stream assignment information, which ensures that causal consistency is maintained without requiring complex runtime coordination between streams.
2Productivity
If data changes are applied in parallel through multiple streams, then performance is improved, but the complexity of managing stream coordination increases
Solution Approach 1:
Each stream operates autonomously with self-service characteristics, processing its assigned data changes independently without requiring continuous coordination with other streams. The system provides self-service through automatic conflict detection, stream assignment, and ordering based on pre-computed hash values, eliminating the need for complex inter-stream communication and coordination protocols.
Solution Approach 2:
The system uses parameter changes in the form of hash values derived from constraint information to determine stream assignment and detect conflicts. By transforming constraint data into hash parameters and using these to guide stream selection and ordering, the system simplifies the coordination complexity while maintaining causal consistency across parallel streams.
3Reliability
If a single stream is used for replication, then causal consistency is maintained, but throughput is limited
Solution Approach 1:
The patent transitions from a single-dimensional sequential stream to a multi-dimensional parallel stream architecture. By introducing multiple streams as an additional dimension for data flow, the system maintains consistency guarantees (through proper ordering within streams) while enabling parallel processing across the new dimension, thus breaking the trade-off between consistency and throughput.
Data Source
AI summary
A method for distributing one or more conflicting data changes. The method includes extracting data changes from the source site log, and applying the data changes to the destination site. The method then includes identifying one or more constraints for each data change and generating a hash value for each identified constraint. The method then dispatches the data changes into a plurality of streams based on the hash values. The hash values are used to identify conflicting data changes and determine how the data changes should be sent through the plurality of streams such that data changes are applied in parallel in the plurality of streams without violating any causal consistency constraints. The method then includes committing data changes in the plurality of streams to the destination site in a single transaction, creating a consistent transactional view.