Operation Fragmentation with Metadata Serialization in Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale query processing systems face high network traffic and processing burdens due to the need for compute nodes to handle all query operations, leading to inefficiencies and performance issues, especially when dealing with large databases.
Innovation Solution
Implementing operation fragmentation with metadata serialization in query processing pushdowns, where compute nodes generate query text fragments and serialize metadata to enable page servers to perform processing operations, reducing the load on compute nodes and optimizing network traffic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If compute nodes perform all query processing operations, then query processing capability is maintained, but network traffic and processing burden increase significantly
Solution Approach 1:
The patent segments query processing operations into two parts: pushdown operations (filtering, projection, aggregation) executed at page servers, and non-pushdown operations executed at compute nodes. This segmentation allows processing to be distributed closer to data storage, reducing network traffic while maintaining query processing capability.
Solution Approach 2:
The patent introduces a query processing pushdown engine as an intermediary component that analyzes incoming queries, identifies pushdown-eligible operations, and generates pushdown requests. This intermediary enables automatic offloading of processing tasks from compute nodes to page servers, reducing processing burden on compute nodes.
2Quantity of substance
If databases are scaled out to larger sizes, then data capacity increases, but processing burden and network traffic are compounded
Solution Approach 1:
The system segments large-scale database processing across multiple page servers, with each server handling a portion of the data. Pushdown operations are executed locally at each page server, avoiding the need to concentrate all processing at compute nodes, thus maintaining processing efficiency as data capacity scales.
Solution Approach 2:
The patent adds a new dimension to query processing by enabling execution at the storage layer (page servers) in addition to the compute layer. This dimensional expansion allows processing to occur closer to data, improving scalability without compromising processing efficiency.
3Ease of operation
If compute nodes retrieve data from page servers for processing, then data access is enabled, but network traffic volume increases
Solution Approach 1:
The patent applies preliminary filtering and projection operations at page servers before data is transmitted to compute nodes. By performing these operations in advance (pushdown operations), only the necessary data is transferred over the network, significantly reducing network traffic volume while maintaining data access capability.
Solution Approach 2:
The patent extracts and executes pushdown-eligible operations (filtering, projection, aggregation) at page servers, removing these processing steps from the compute node workflow. This extraction reduces the amount of data that needs to be transferred to compute nodes, thereby reducing network traffic.
4Productivity
If pushdown operations are implemented, then processing load on compute nodes is reduced, but system complexity increases
Solution Approach 1:
The patent introduces a query processing pushdown engine as an intermediary that automates the analysis and transformation of queries into pushdown requests. This intermediary manages the complexity of determining which operations can be pushed down, shielding users from system complexity while enabling compute node efficiency improvements.
Solution Approach 2:
The patent creates a copy of the query processing logic at page servers in the form of pushdown operations. Instead of requiring complex coordination between compute nodes and page servers, the system copies essential processing capabilities to the storage layer, simplifying the overall system architecture.
Data Source
AI summary
Methods for operation fragmentation with metadata serialization in query processing pushdowns are performed by systems and devices. A compute node receives a query directed to database data, and generates query text fragments. Portions of metadata of the database are read from different page servers, and are serialized by the compute node. Page identities of data pages in a page server that stores the data are determined from a page index at the compute node, and the compute node provides the text fragments, the serialized metadata, and the page identities to the page server storing the data. The page server compiles the text fragments based on the serialized metadata to generate an executable query plan for the query. The page server initializes and performs execution of the executable query plan against the data as a pushdown query operation, and a result of pushdown query operation is provided to the compute node.