Page Digest Filtering in OLTP Cluster Database Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In OLTP cluster database systems, existing methods face inefficiencies due to increased processing overheads when a page is not in the database buffer, leading to re-reads from the network or disk, which increases processing time and overheads.
Innovation Solution
A page query method is introduced where a data processing node searches shared memory for a page digest to determine if the target data is on the page, allowing it to skip or load the page as necessary, and manages physical locks in a passive releasing manner to reduce unnecessary disk I/O and interaction with the central node.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a page is not in the database buffer, then the page must be re-read from network or disk, but this increases processing overheads and processing time
Solution Approach 1:
The patent pre-loads page digests into shared memory before they are needed for query processing. When a data processing node needs to query a page, the page digest is already available in shared memory, allowing immediate determination of whether the page needs to be loaded into the database buffer, thus avoiding unnecessary disk I/O and network communication delays
Solution Approach 2:
The patent introduces page digests as an intermediary data structure between the database buffer and the query processing system. These digests contain metadata about pages (such as page IDs, data ranges, or other indexing information) that allow data processing nodes to quickly determine page relevance without accessing the actual page data, thereby reducing unnecessary disk I/O operations
2Quantity of substance
If a page is re-read from network or disk when buffer is insufficient, then data can be retrieved, but processing overheads are increased
Solution Approach 1:
The system pre-generates and stores page digests in shared memory before query operations begin. This preliminary preparation allows data processing nodes to quickly assess page relevance using simple memory lookups rather than complex disk I/O operations, significantly reducing processing overhead while ensuring data can be retrieved when needed
Solution Approach 2:
The patent extracts essential metadata from full page data to create compact page digests. These digests contain only the critical information needed for query filtering (such as page identifiers, data ranges, or index information), separating the selection function from the data retrieval function, thereby reducing processing overhead while maintaining complete data retrieval capability
3Productivity
If page digests are used to filter pages, then processing efficiency is improved, but shared memory usage increases
Solution Approach 1:
The patent optimizes the structure and size of page digests by carefully selecting which parameters to include in the digest data structure. By including only the essential metadata needed for query filtering (such as compact page identifiers and minimal indexing information), the system achieves high processing efficiency while keeping the memory footprint of page digests manageable
Solution Approach 2:
The system segments the memory management of page digests by organizing them in shared memory with efficient data structures that allow rapid access and minimal memory overhead. The segmentation approach allows the system to handle large numbers of page digests without proportionally increasing total memory usage, as digests for inactive or expired pages can be efficiently removed or archived
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A page query method and a data processing node in an OLTP cluster database are disclosed. The OLTP cluster database includes a central node and at least one data processing node, the central node manages a PLock in a passive PLock releasing processing manner, and the method includes: searching a shared memory of the data processing node for a page digest of a first page when the data processing node in the OLTP cluster database needs to query the first page during a data query and the first page is not in storage space of a database buffer of the data processing node (101); and if it is determined, according to the page digest, that target data of the data query cannot be on the first page, skipping performing a query operation on the first page; or if it is determined, according to the page digest, that target data of the data query may be on the first page, loading the first page into the storage space of the database buffer, and traversing the first page to perform the data query (102).