Query Fragmentation with Metadata Serialization for Database Pushdown
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale query processing systems face high network traffic and processing burdens due to the need for compute nodes to handle all query processing operations, leading to inefficiencies as databases scale out.
Innovation Solution
Implementing operation fragmentation with metadata serialization in query processing pushdowns, where compute nodes generate query text fragments and serialize metadata to enable page servers to perform pushdown operations, thereby offloading processing from compute nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If compute nodes perform all query processing operations, then query processing capability is maintained, but network traffic and processing burdens increase significantly
Solution Approach 1:
The patent segments query processing operations into two parts: metadata serialization is performed at the compute node and pushed down to page servers, while data processing operations are executed at the page servers. This segmentation allows the compute node to send only serialized metadata and operation instructions, while page servers handle the actual data processing, thereby reducing network traffic and compute node burden.
Solution Approach 2:
The patent introduces serialized metadata as an intermediary that carries operation instructions from compute nodes to page servers. This intermediary enables the page servers to understand and execute complex query operations without requiring the compute node to process all data locally, thus reducing network traffic while maintaining processing capability.
2Volume of stationary object
If databases are scaled out to larger sizes, then storage capacity increases, but network traffic and processing burdens on compute nodes compound
Solution Approach 1:
The patent segments the database system into compute nodes that handle metadata serialization and page servers that handle data storage and processing. When databases scale out, new page servers can be added to the distributed system, and the compute nodes push down metadata and operations to the appropriate page servers, allowing the system to handle larger storage capacities without proportionally increasing network traffic or compute node burden.
3Productivity
If page servers perform pushdown operations, then processing burden on compute nodes decreases, but metadata serialization complexity increases
Solution Approach 1:
The patent applies preliminary action by serializing metadata and operation instructions at the compute node before pushing them down to page servers. This preliminary serialization prepares the metadata in a format that page servers can directly execute, reducing the need for complex runtime processing at the page server level and simplifying the overall system architecture.
Data Source
AI summary
Methods for operation fragmentation with metadata serialization in query processing pushdowns are performed by systems and devices. A compute node receives a query directed to database data, and generates query text fragments. Portions of metadata of the database are read from different page servers, and are serialized by the compute node. Page identities of data pages in a page server that stores the data are determined from a page index at the compute node, and the compute node provides the text fragments, the serialized metadata, and the page identities to the page server storing the data. The page server compiles the text fragments based on the serialized metadata to generate an executable query plan for the query. The page server initializes and performs execution of the executable query plan against the data as a pushdown query operation, and a result of pushdown query operation is provided to the compute node.