Garbage Collection for Redundant Change Sets in Document Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems face challenges in managing the consistency of multiple replicas in a document database, particularly in write-dominated workloads, due to the need for total ordering of writes and conflict resolution across geographically distributed data centers.
Innovation Solution
The system implements a geo-replicated, conflict-free replicated document database that operates in both single-master and multi-master modes, using operation-based CRDTs for automatic conflict resolution and garbage collection to eliminate redundant change sets, thereby ensuring deterministic value convergence without global coordination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is replicated to multiple data centers to reduce user perceived latency, then read operations can be serviced by the nearest geographic DC, but managing the consistency of multiple replicas becomes increasingly difficult
Solution Approach 1:
The system employs CRDTs (Conflict-free Replicated Data Types) that automatically resolve conflicts between replicas without requiring centralized coordination. Each replica independently applies changes using deterministic merge functions, allowing the system to self-manage consistency across distributed data centers while maintaining low latency for users.
2Reliability
If total ordering of writes is enforced to ensure consistency across replicas, then all replicas see updates in the same order, but write-dominated workloads fail to benefit from distributed architecture
Solution Approach 1:
The system segments the write operation into two independent parts: (1) local application of the write to the local replica, and (2) propagation of the write to other replicas. This segmentation allows write-dominated workloads to proceed in parallel across multiple data centers without waiting for global coordination, while CRDTs ensure eventual consistency is maintained.
Solution Approach 2:
The system performs preliminary local writes at each replica before propagation to other data centers occurs. This preliminary action enables write operations to be immediately acknowledged to clients, improving write throughput, while consistency is subsequently ensured through CRDT merge operations when replicas synchronize.
3Reliability
If change sets are retained in the document database to ensure consistency, then all changes can be tracked and applied, but storage resources are consumed by redundant and superseded changes
Solution Approach 1:
The system implements garbage collection that identifies and discards superseded change sets. By tracking version vectors and determining when a change set has been superseded by a more recent version, the system safely removes redundant data from storage while maintaining the ability to reconstruct document state, thereby recovering storage resources.
4Speed
If multiple replicas are maintained at every data center, then read operations can be serviced locally, but the quantity of data to be managed and synchronized increases
Solution Approach 1:
The system applies garbage collection to remove superseded change sets from replicas, reducing the total volume of data that needs to be stored and synchronized across data centers. This allows multiple replicas to be maintained for low-latency reads while preventing unbounded growth of stored change history.
Data Source
AI summary
Disclosed herein are exemplary systems and methods for garbage collection and/or deletion in a document database. The methods may include, for each change in a first change set, determining whether a first characteristic of the change is superseded by a second characteristic of a corresponding change in a second change set. The change of the first change set and the change of the second change set can pertain to a document attribute. The method may include determining whether the first change set is redundant with the second change set if each change of the first change set is superseded by a corresponding change of the second change set, and eliminating the first change set from the document database when the first change set is redundant with second change set.