Granular Timestamp Concurrency Control for Key-Value Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale data storage systems face inefficiencies in storage space consumption due to the addition of generational index fields in key-value (KV) stores, which do not contribute to the schema and hinder concurrency control.
Innovation Solution
Implementing snapshot isolation using multiple-tiered sorted data structures and granularly timestamped concurrency control within the KV store engine, eliminating the need for generational index fields by storing multiple snapshot versions and managing timestamps to ensure data consistency across operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If generational index fields are added to all data records in a KV store to implement snapshot isolation, then concurrency control is improved, but storage space consumption increases significantly
Solution Approach 1:
The patent extracts the generational index field from the data record structure itself and relocates it to a separate generation index structure. This separation allows the KV store to implement snapshot isolation without embedding generational metadata in every record, thereby reducing storage space consumption while maintaining concurrency control capabilities.
Solution Approach 2:
The patent introduces a new dimensional organization for tracking data generations by creating a separate generation index structure that maps keys to their generational information. This dimensional change eliminates the need to store generational fields within each record, resolving the contradiction between concurrency control and storage efficiency.
2Reliability
If generational index fields are added to data records for snapshot isolation, then data consistency is improved, but device complexity increases
Solution Approach 1:
By extracting the generational index mechanism from the data record structure and placing it in a separate generation index, the patent reduces the complexity embedded in each record while maintaining the ability to enforce snapshot isolation and data consistency across concurrent operations.
3Quantity of substance
If multiple snapshot versions are stored without generational index fields, then storage space is reduced, but concurrency control efficiency decreases
Solution Approach 1:
The generation index structure serves as an intermediary between the stored snapshot versions and the concurrency control mechanism. It provides the necessary generational information for snapshot isolation without requiring this information to be embedded in each data record, thus maintaining concurrency control efficiency while minimizing storage overhead.
Data Source
AI summary
Systems and methods discussed herein, based on a key-value data store including multiple-tiered sorted data structures in memory and storage, implement granularly timestamped concurrency control. The multiple-tiering of the key-value data store enables resolving the snapshot queries by returning data record(s) according to granularly timestamped snapshot lookup instead of singularly indexed snapshot lookup. Queries return a merged collection of records including updates from data structures in memory and in storage, such that a persistent storage transaction may refer to non-committed updates up to a timeframe defined by the snapshot read timestamp. This way, inconsistency is avoided that would result from merely reading data records committed in storage, without regard as to pending, non-committed updates thereto. The global timestamp further modifies the generation of the local transaction commit timestamp and the local snapshot read timestamp, so as to establish a granularly timestamped concurrency control scheme (over three levels of granularity).