Journal-Based Partitioned Data Materialization in Multi-Store Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing environments face challenges in managing transactions across multiple data stores due to lack of support for ACID properties, varying data models, and network delays, leading to complex and unreliable transaction management mechanisms.
Innovation Solution
A journal-based multi-data-store database system that implements partitioning policies, optimistic concurrency control, and journal forking to manage transactions across diverse data stores, ensuring scalability and robustness by registering data stores as materialization nodes, using partitioning algorithms like horizontal and vertical partitioning, and creating snapshots for efficient synchronization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If ad-hoc transaction management mechanisms are implemented by developers, then transaction support across data store boundaries can be achieved, but the system becomes hard to maintain and lacks robustness
Solution Approach 1:
The patent introduces a journal manager as an intermediary component that coordinates transactions across multiple data stores. The journal manager maintains a journal of committed transactions and uses write appliers to propagate changes to materialization nodes, eliminating the need for developers to implement complex ad-hoc transaction management mechanisms while maintaining ACID properties across data store boundaries
Solution Approach 2:
The system segments transaction management into distinct components: a journal manager for coordination, write appliers for propagation, and materialization nodes for data storage. This segmentation allows each component to have a specific, maintainable function while working together to provide robust transaction support across diverse data stores
2Adaptability or versatility
If data stores are distributed across multiple locations to serve different geographical customers, then service coverage is improved, but network delays and failures increase
Solution Approach 1:
The system performs preliminary actions by maintaining a journal of committed transactions before propagating changes to distributed materialization nodes. This allows the system to record transactions centrally first, ensuring consistency, and then propagate changes to distributed locations, mitigating the impact of network delays and failures while maintaining transaction reliability across geographical locations
3Productivity
If partitioning is applied to data stores to improve scalability, then system capacity increases, but data consistency management becomes more complex
Solution Approach 1:
The system uses feedback mechanisms where materialization nodes report their state and the journal manager tracks which partitions have received which transactions. This feedback allows the system to maintain data consistency across partitioned data stores by knowing the current state of each partition and propagating transactions in the correct order, reducing the complexity of consistency management while improving scalability
Data Source
AI summary
A partitioning policy for materializing writes indicated in journal entries of a multi-data-store database is determined. The policy indicates a partitioning attribute of a data object. A write applier associated with a materialization node of the database processes journal entries according to the partitioning policy. If the value of the partitioning attribute associated with a write of a particular journal entry indicates that the write belongs to a partition materialized at the materialization node, the write applier propagates the write to the materialization node for storage at the node's storage devices.