Distributed Data Grid Resolver for Persistent Store Versioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern computing systems face challenges in supporting persistence and integrity in distributed data grids due to increased user demand and transaction volumes, leading to data loss and performance issues during restarts and failures.
Innovation Solution
A distributed data grid system that uses a persistence layer to store cache content in shared storage or local disks, with a resolver selecting the most recent valid version of partitions and coordinating recovery across members to ensure data integrity and minimize performance impact, employing a recovery quorum and asynchronous messaging for efficient data management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If persistence is implemented in a distributed data grid to prevent data loss, then data reliability is improved, but system complexity increases due to version management and coordination overhead
Solution Approach 1:
A resolver component is introduced as an intermediary that coordinates partition recovery across distributed members. The resolver receives identifiers from multiple members, selects the most recent valid version, and determines which member should recover the partition, thereby managing the complexity of persistence coordination centrally while maintaining distributed reliability
Solution Approach 2:
The system performs preliminary actions by pre-selecting and validating partition identifiers before recovery operations. The resolver identifies and validates the most recent valid version of each partition in advance, ensuring data integrity is established before actual recovery begins, which prevents data loss while maintaining organized complexity
2Manufacturing precision
If the system selects and validates partition versions to ensure data integrity, then data quality is improved, but recovery time increases due to validation overhead
Solution Approach 1:
Members autonomously provide their own partition identifiers and validity information to the resolver without requiring external validation. Each member self-reporting its persisted partitions enables the resolver to quickly aggregate and validate information across the distributed system, improving data quality through collective self-verification while minimizing coordination time
3Reliability
If the system coordinates recovery across multiple members to ensure consistency, then system reliability is improved, but performance decreases due to communication overhead
Solution Approach 1:
The recovery coordination is segmented into distinct phases: members independently identify their persisted partitions, the resolver aggregates identifiers and selects valid versions, and finally recovery is assigned to specific members. This segmentation allows parallel execution of identifier collection and validation, improving system reliability through coordinated recovery while maintaining performance by avoiding sequential bottlenecks
Data Source
AI summary
A system and method can support recovery of persisted data from persistent storage of a distributed data grid, and provide persistent store versioning and integrity. A coordinator in the distributed data grid can receive a plurality of identifiers from one or more members of the distributed data grid, wherein each said identifier is associated with a persisted partition in a persistent storage for the distributed data grid. Then, the coordinator can select an identifier for each partition, wherein each selected identifier is associated with a most recent valid version of a partition. Furthermore, the coordinator can determine an active member in the distributed data grid that is responsible for recovering said partition from a persisted partition associated with the selected identifier.