Query Processing Pushdown to Page Servers

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed database systems face inefficiencies in handling analytical workloads that are scan-intensive over large data sets, as all query processing is performed at compute nodes, requiring the transfer of large data volumes between compute nodes and page servers.

Innovation Solution

The system enables compute nodes to pushdown certain query processing tasks to page servers, reducing data transfer and leveraging idle compute resources at page servers. This involves maintaining multiple versions of data objects at page servers, determining which versions satisfy queries, and modifying table metadata for concurrent queries across different table schema versions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If all query processing is performed at compute nodes, then query processing capability is maintained, but data transfer volume between compute nodes and page servers increases significantly

Engineering Contradiction:

Improvequery processing capabilityVSAvoiddata transfer volume

Core Design Contradiction:

ProductivityVSLoss of substance

Solution Approach 1:

The patent segments query processing tasks into two categories: those that can be executed at page servers (filtering, projection, aggregation) and those that must remain at compute nodes (complex joins, transactions). This segmentation allows pushdown of appropriate operations to page servers, reducing data transfer while maintaining query processing capability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a new dimension of query processing by enabling execution at the storage layer (page servers) in addition to the compute layer. This dimensional expansion allows filtering and processing to occur closer to the data, reducing the amount of data that needs to be transferred across the network.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Device complexity

If compute resources are centralized at compute nodes, then query processing is simplified, but idle compute resources at page servers are underutilized

Engineering Contradiction:

Improvequery processing architectureVSAvoidresource utilization flexibility

Core Design Contradiction:

Device complexityVSAdaptability or versatility

Solution Approach 1:

The patent makes page servers multi-functional by enabling them to perform both storage functions and query processing functions. Page servers can serve as storage nodes and simultaneously execute pushed-down query operations, transforming them from single-function storage devices to universal nodes that can adapt to different workload requirements.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent introduces dynamic task allocation where query processing operations are dynamically pushed down to page servers based on workload characteristics, data locality, and resource availability. This dynamic approach allows the system to adapt resource utilization in real-time rather than maintaining a static architecture.

Inventive Principle:

Principle #15Dynamics

3Reliability

If page servers maintain multiple versions of data objects, then concurrency control is improved, but version determination complexity increases

Engineering Contradiction:

Improveconcurrency controlVSAvoidversion determination logic

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent uses copying of version metadata (transaction IDs, timestamps) alongside data versions at page servers. When queries are pushed down, the compute node copies the relevant transaction context and sends it with the query, allowing the page server to determine visible versions through simple comparison rather than complex distributed locking or consensus protocols.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS12210514B2Schema and data modification concurrency in query processing pushdown

Publication Date: 2025.01.28 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Distributed database systems including compute nodes and page servers are described herein that enable compute nodes to pushdown certain query processing compute tasks to the page servers to take advantage of otherwise idle compute resources at the page servers, and to reduce the quantity of data that moves between compute nodes and page servers. A distributed database system includes a page server and a compute node, wherein the page server is configured to maintain multiple versions of stored data objects. The compute node is configured to receive a query and generate a transaction context (TC) and modified table schemas (MTS) scoped to the query, and pushdown the query, TC and MTS to the page server that is configured to determine which data objects at the page server satisfy the query, and for each such object, which version of the object should be returned based on the TC.