Transactional Grooming in Multi-Master Distributed Data Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data management systems face challenges in handling transactional operations like updates, deletes, and inserts in multi-master distributed environments, particularly due to compatibility issues with append-only storage streams that are mutation unfriendly, affecting efficient processing and space consumption.
Innovation Solution
A method for grooming transactional operations in a multi-master distributed data management system involves maintaining transactional operations in a transaction-local side-log, performing a table log scan, generating tuple sequence numbers, and writing records to groomed transaction blocks, which are then shared across nodes to enhance processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transactional operations are allowed to go to any replica in multi-master distributed systems, then high availability is achieved, but compatibility with append-only storage streams deteriorates
Solution Approach 1:
The patent segments transactional operations into two distinct paths: (1) mutations (updates/deletes) are routed to leader replicas that maintain append-only storage streams, and (2) inserts are routed to follower replicas. This segmentation allows each replica type to specialize in handling specific operation types, resolving the conflict between multi-master availability and append-only storage compatibility.
Solution Approach 2:
The patent introduces leader replicas as intermediaries that receive all mutation operations and transform them into append-only storage format before writing to storage streams. Follower replicas then receive these transformed operations from leaders. This intermediary layer enables compatibility with append-only storage while maintaining multi-master capabilities.
2Adaptability or versatility
If mutation operations are handled in distributed systems, then transactional flexibility is improved, but space consumption and processing efficiency deteriorate
Solution Approach 1:
The patent extracts the mutation transformation logic from follower replicas and concentrates it in leader replicas. Leaders perform the expensive operation of transforming mutations into compact append-only format, while followers only need to apply pre-transformed operations. This extraction reduces redundant processing and space consumption across the distributed system.
3Device complexity
If conventional storage streams are used for transactional operations, then simplicity is maintained, but processing efficiency and scalability deteriorate
Solution Approach 1:
The patent introduces dynamic operation routing based on operation type and replica role. The system dynamically determines which replicas handle which operations (leaders handle mutations, followers handle inserts), and dynamically transforms operations during transit. This dynamic approach enables parallel processing across multiple replicas, dramatically improving throughput while maintaining conceptual simplicity through role-based specialization.
Data Source
AI summary
One embodiment provides for a method for grooming transactional operations in a multi-master distributed data management system including maintaining transactional operations in at least one transaction-local side-log. A table log scan is performed for a table including recording identification for each record in a log. Tuple sequence numbers are generated for the records and the records are written on groomed transaction blocks to the table.