Page Split Detection in Query Processing Pushdowns
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale query processing systems face high network traffic and processing burdens due to the need for compute nodes to handle and process data from page servers, leading to inefficiencies in handling large databases and complex queries.
Innovation Solution
Implementing page split detection and affinity in query processing pushdowns, where page servers perform pushdown operations and detect page splits, allowing them to handle processing tasks typically done by compute nodes, reducing network traffic and memory pressure by offloading operations and maintaining off-row data locally.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If compute nodes perform all processing operations on data retrieved from page servers, then query processing can be completed, but network traffic volume and processing burden on compute nodes become excessively high
Solution Approach 1:
The patent extracts processing operations from compute nodes and relocates them to page servers. Specifically, pushdown operations such as filtering, aggregation, and projection are moved from compute nodes to page servers, allowing data processing to occur at the storage layer rather than being centralized at the compute layer, thereby reducing network traffic and compute node burden
Solution Approach 2:
The patent introduces a new dimension of processing by enabling page servers to perform computational operations traditionally reserved for compute nodes. This transforms the architecture from a two-tier model (storage vs. compute) to a more distributed model where processing capabilities exist at multiple levels, effectively adding a processing dimension at the storage layer
2Loss of energy
If page servers perform pushdown operations and detect page splits, then network traffic and memory pressure are reduced, but system complexity increases
Solution Approach 1:
The patent implements self-service mechanisms at the page server level, where page servers autonomously detect page splits, manage off-row data, and execute pushdown operations without requiring constant coordination with compute nodes. This self-service capability reduces the need for complex inter-node communication and coordination protocols
Solution Approach 2:
The patent employs preliminary action by having page servers proactively detect page splits before they cause processing failures and by pre-managing off-row data localization. This advance detection and management prevents operational failures and reduces the need for complex error handling and recovery mechanisms
3Speed
If off-row data is stored at the same page server as the data, then data access efficiency improves, but storage management complexity increases
Solution Approach 1:
The patent merges the storage of off-row data with the primary data at the same page server. This collocation ensures that related data elements are stored together, enabling efficient access during pushdown operations without requiring additional data retrieval from other page servers, thus improving access efficiency while maintaining manageable storage organization
Data Source
AI summary
Methods for page split detection and affinity in query processing pushdowns are performed by systems and devices. Page servers perform pushdown operations based on specific, and specifically formatted or generated, information, instructions, and data provided thereto from a compute node. Page servers also determine that page splits have occurred during reading of data pages maintained by page servers during pushdown operations, and also during fulfillment of compute node data requests. To detect a data page has split, page servers utilize information from a compute node of an expected next data page which is compared to a next data page in the page server page index. A mismatch in the comparison by page servers indicates data page was split. Compute nodes and page servers store and maintain off-row data generated during data operations via page affinity considerations where the off-row data is stored at the same page server as the data.