Transactional Key-Value Store with Optimistic Concurrency Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-core computing systems face limitations in data transfer and communication due to hardware bottlenecks, particularly between main memory and secondary storage, and software limitations related to database organization and management, which hinder performance in achieving ACID compliance and scalability.
Innovation Solution
The implementation of a framework for transactional key-value data stores that utilize both volatile and non-volatile random access memory, employing dual pointers and snapshot data structures to optimize data access and management, allowing for concurrent transactions without a central concurrency controller and reducing computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional database management systems are used to ensure ACID compliance, then data consistency and reliability are improved, but computational overhead and processing speed deteriorate
Solution Approach 1:
The database system is segmented into multiple independent processor cores that can operate concurrently on different data partitions. Each core maintains its own transaction log and data structures, eliminating the need for centralized locking and coordination mechanisms. This segmentation allows parallel processing while maintaining ACID compliance through local consistency checks.
Solution Approach 2:
An intermediary logging mechanism is introduced where transaction logs are written to non-volatile storage before committing changes to the main database. This intermediary layer ensures durability and consistency without blocking subsequent read operations, allowing the system to maintain ACID properties while improving throughput by decoupling write confirmation from actual data modification.
2Reliability
If data is stored in secondary non-volatile storage memory, then data durability and capacity are improved, but data access speed deteriorates
Solution Approach 1:
The system introduces a temporal dimension to data storage by maintaining multiple versions of data records across different time points. Recent versions are kept in fast volatile memory for quick access, while historical versions are archived in non-volatile storage. This multi-level temporal storage hierarchy allows the system to provide fast access to current data while preserving durability through persistent storage of critical records.
Solution Approach 2:
Data is pre-loaded into volatile memory from non-volatile storage based on predicted access patterns and frequency. Frequently accessed data is proactively cached in fast memory before actual access occurs, reducing latency for common operations. This preliminary action maintains data durability in non-volatile storage while optimizing access speed for high-demand data.
3Reliability
If centralized concurrency control is implemented, then transaction consistency is improved, but communication overhead and scalability deteriorate
Solution Approach 1:
The centralized concurrency control mechanism is extracted and replaced with distributed optimistic concurrency control at each processor core. Each core independently validates transactions against a versioned copy of the data, eliminating the need for centralized locking and coordination. This extraction removes the communication bottleneck while maintaining consistency through local validation rules and conflict detection mechanisms.
Solution Approach 2:
Each processor core is empowered to independently manage its own transaction validation and conflict resolution without requiring centralized coordination. The system provides each core with the necessary tools and data structures to perform self-validation, allowing transactions to be processed autonomously. This self-service approach dramatically reduces inter-processor communication overhead while maintaining transaction consistency through distributed validation.
4Quantity of substance
If massive amounts of data are kept in secondary storage, then data capacity and persistence are improved, but data transfer bandwidth requirements increase
Solution Approach 1:
The system applies local quality optimization by storing data in a distributed manner across multiple processor cores and storage nodes, with each location holding only the data it needs or frequently accesses. This localized data distribution eliminates the need for centralized data gathering and reduces overall data transfer bandwidth requirements. Each processor works with locally cached data, minimizing inter-node communication and energy consumption.
Data Source
AI summary
Example implementations disclosed herein can be used to build, maintain, and access databases built database in multi-core computing systems with large VRAM and huge NVRAM. The database with optimistic concurrency control can be built on a transactional key-value data store that includes logically equivalent data pages stored in both VRAM and VRAM. Data records in volatile data pages in the VRAM represent the most recent version of the data. Data records in the NVRAM immutable and are organized in a stratified composite snapshot. A distributed log gleaner process is used to process log entries corresponding to transactions on the volatile data pages and construct the snapshot. The log gleaner sorts the log entries by epoch, key range, and most recent use to partition the snapshot across multiple nodes.