Query Processing Pushdown in Distributed Database Page Servers

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed database systems face inefficiencies in handling analytical workloads that are scan-intensive over large data sets, as all query processing is performed at compute nodes, necessitating the transfer of large data volumes, which consumes significant system resources and reduces throughput.

Innovation Solution

Implementing query processing pushdowns that allow compute nodes to offload certain tasks to page servers, enabling these servers to apply filters and return only relevant data, thereby reducing data transfer and leveraging idle compute resources at page servers.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If all query processing is performed at compute nodes, then query processing capability is concentrated and controlled, but large amounts of data must be transferred from page servers to compute nodes, consuming significant system resources and reducing throughput

Engineering Contradiction:

Improvequery processing throughputVSAvoidsystem resource consumption

Core Design Contradiction:

ProductivityVSLoss of energy

Solution Approach 1:

The patent segments query processing tasks between compute nodes and page servers. Compute nodes handle query routing and coordination, while page servers execute filter operations and data processing locally. This segmentation allows processing to occur closer to the data, reducing the volume of data that must be transferred across the network and improving overall system throughput for analytical workloads.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a new dimension of query processing capability at the page server level, transforming the traditional single-location processing model. By enabling page servers to perform filter operations and data processing locally, the system adds a distributed processing dimension that reduces network traffic and improves scalability for scan-intensive analytical queries.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Productivity

If compute nodes pushdown query processing tasks to page servers, then data transfer volume is reduced and idle compute resources at page servers are utilized, but system complexity increases due to coordination and version management requirements

Engineering Contradiction:

Improveanalytical workload performanceVSAvoidsystem architecture complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent makes page servers multi-functional by enabling them to perform both their traditional storage and data serving functions plus query processing operations. Page servers can now execute filter operations, process analytical queries locally, and manage multiple versions of data objects, transforming them from simple storage nodes into capable processing units that reduce the burden on compute nodes.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent introduces compute nodes as intermediaries that coordinate between the query source and page servers. Compute nodes manage the pushdown logic, handle query routing, and coordinate version management, allowing page servers to focus on executing processing tasks locally while maintaining system-wide consistency through the intermediary coordination layer.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If page servers maintain multiple versions of data objects for transaction support, then transactional consistency is ensured, but determining which version to return for analytical queries becomes more complex

Engineering Contradiction:

Improvetransactional consistencyVSAvoidversion management complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent uses version copying to maintain multiple snapshots of data objects at page servers. Each transaction creates a copy of the data state at that point in time, allowing analytical queries to access historical versions while transactions maintain consistency. This copying approach enables version management without requiring complex in-place modification tracking.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent performs preliminary versioning actions by maintaining multiple pre-computed versions of data objects at page servers before analytical queries arrive. This preliminary preparation allows the system to quickly determine which version to return based on query timing and transaction context, rather than computing versions on-demand, thereby reducing query processing complexity.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11797523B2Schema and data modification concurrency in query processing pushdown

Publication Date: 2023.10.24 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Distributed database systems including compute nodes and page servers are described herein that enable compute nodes to pushdown certain query processing compute tasks to the page servers to take advantage of otherwise idle compute resources at the page servers, and to reduce the quantity of data that moves between compute nodes and page servers. A distributed database system includes a page server and a compute node, wherein the page server is configured to maintain multiple versions of stored data objects. The compute node is configured to receive a query and generate a transaction context (TC) and modified table schemas (MTS) scoped to the query, and pushdown the query, TC and MTS to the page server that is configured to determine which data objects at the page server satisfy the query, and for each such object, which version of the object should be returned based on the TC.