Cloud Native Database Tree Data Structure Reading via Near Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional cloud-based database systems face performance and scalability issues due to redundant data storage, leading to resource waste and high storage costs, as they do not effectively manage data retrieval in a scalable and efficient manner.
Innovation Solution
The method and apparatus for reading data using near data processing (NDP) in a cloud native database, which involves scanning a buffer pool to identify required pages, applying shared locks, and utilizing a private buffer pool to copy and process pages, allowing for concurrent modifications and efficient data retrieval without blocking other transactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional database architecture with multiple SQL nodes and storage nodes is used, then data redundancy and reliability are improved, but storage cost and resource waste increase significantly
Solution Approach 1:
The patent extracts the SQL processing functionality from the traditional distributed database architecture, allowing a single SQL node to handle all queries directly against storage nodes without requiring multiple SQL replicas. This eliminates the need for redundant data storage across multiple SQL nodes while maintaining full data accessibility and reliability.
Solution Approach 2:
The SQL node is designed to universally handle all types of database operations (reads, writes, queries) directly, eliminating the need for specialized replica nodes. The storage nodes provide universal data storage services to all SQL nodes, reducing the need for multiple copies of data across the system.
2Productivity
If data is scanned from storage nodes over the network, then data retrieval is achieved, but network traffic and I/O operations increase
Solution Approach 1:
The patent implements a buffer pool that pre-loads and caches data pages from storage nodes before they are actually needed for query processing. This preliminary action reduces the frequency of network I/O operations during query execution, as frequently accessed data is already available in the buffer pool memory.
Solution Approach 2:
The buffer pool acts as an intermediary layer between the SQL node and storage nodes, caching data pages in memory to reduce direct network traffic. This intermediary layer absorbs read requests, serving data from cache when possible and minimizing the number of network round trips required.
3Reliability
If shared locks are applied to ensure data consistency, then read consistency is improved, but write concurrency is reduced
Solution Approach 1:
The patent segments the locking mechanism into page-level shared locks rather than row-level or table-level locks. This segmentation allows multiple readers to access different pages simultaneously without blocking each other, while writers can still modify specific pages with exclusive locks. The buffer pool manages these granular locks to balance consistency and concurrency.
Solution Approach 2:
The locking system dynamically adjusts lock granularity and duration based on the operation type. Readers acquire short-lived shared locks only for the duration of page access, while writers acquire exclusive locks only when modifying data. This dynamic approach minimizes lock contention and maximizes concurrent access efficiency.
Data Source
AI summary
The present disclosure provides a method of reading data maintained in a tree data structure, such as B+ tree, using near data processing (NDP) in a cloud native database. According to embodiments, a desired LSN will be used in NDP page reads on the master computing node (e.g. master SQL node). When the master computing node (e.g. master SQL node) reads the regular page, the maximum desired LSN (e.g. the latest page version number) for that regular page will be used. Embodiments use features of the desired LSN and page locking, wherein correct versions of pages can be obtained by using the desired LSN associated with a page, in combination with page locking, and can enable the reading of a consistent tree structure and achieve good read/write concurrency.