Read Only Bufferpool for Multi-Node Database Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multi-core computing systems face limitations in data transfer between main memory and secondary storage, and inter-processor communication, with database management systems being computationally costly due to software-based limitations, particularly in maintaining ACID properties for transactions in databases with massive data sets.
Innovation Solution
The implementation of a framework for transactional key-value data stores using dual pointers and snapshot data structures across volatile and non-volatile random access memory, allowing for ACID compliance and reducing computational overhead by distributing data across multiple nodes with hybrid memory architectures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If database management systems are implemented to maintain ACID properties for transactions, then data consistency and reliability are improved, but computational overhead and processing speed deteriorate
Solution Approach 1:
The system segments the database into fixed-size data pages that can be independently managed and cached. Each page can be handled separately by different processor cores, enabling parallel processing while maintaining ACID properties through page-level locking and versioning rather than row-level or transaction-level coordination.
Solution Approach 2:
The system pre-allocates fixed-size data pages and maintains versioned copies in cache memory before transactions occur. This preliminary structuring allows transactions to operate on pre-prepared data structures with predictable memory addresses, eliminating the need for complex runtime allocation and reduction of computational overhead.
2Quantity of substance
If data is stored in secondary non-volatile storage memory, then storage capacity is improved, but data access speed deteriorates
Solution Approach 1:
The system extracts frequently accessed data pages from secondary non-volatile storage into volatile cache memory, creating a two-tier storage architecture. The fixed-size page structure enables efficient batching and caching strategies where hot data resides in fast volatile memory while cold data remains in high-capacity non-volatile storage.
Solution Approach 2:
The system introduces a page cache layer as an intermediary between the application and secondary storage. This cache uses the fixed-size page structure to hold copies of frequently accessed pages in volatile memory, acting as a buffer that provides fast access to commonly used data while maintaining the ability to retrieve additional data from secondary storage when needed.
3Power
If data is distributed across multiple processor cores, then computing power is improved, but inter-processor communication overhead increases
Solution Approach 1:
The system segments the database into fixed-size pages with unique identifiers, allowing each processor core to independently manage and cache pages locally. This segmentation enables cores to operate on different pages simultaneously without requiring complex inter-processor coordination, reducing communication overhead while maintaining data consistency through page-level versioning.
Solution Approach 2:
The system creates copies of data pages in the cache memory of multiple processor cores. When a core needs data, it accesses the local copy rather than requesting it from remote storage or other cores, eliminating the need for inter-processor communication for read operations and significantly reducing communication overhead in multi-core systems.
4Speed
If database management techniques are optimized for speed, then operational speed is improved, but computational cost increases
Solution Approach 1:
The system changes the fundamental parameter of data organization from variable-size records to fixed-size pages. This parameter change enables efficient memory addressing, predictable cache behavior, and simplified page management operations that execute faster with lower computational cost. The fixed size allows for direct calculation of page addresses and boundaries without complex pointer chasing or memory allocation overhead.
Data Source
AI summary
Example implementations disclosed herein include techniques for a ready only bufferpool for use in local nodes of a multi-node computing system. Read only transactions executed by a processor can reference a ready only bufferpool resident in a VRAM on the same node. If the desired data page is in the bufferpool the transaction can immediately read data records from the cached data pages. If the desired data page is not in the bufferpool, then the transaction can cause a copy of a corresponding data page in a secondary memory to be installed in the bufferpool. The bufferpool can include more than one copy of a data page simultaneously to handle and prevent cache line misses. Data page are dropped from the bufferpool based on an incrementing per data page counter.