Scan Server Parallel Query Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multi-node, multi-instance database systems, especially dual-format databases, optimizing scan operations is challenging due to inefficient resource utilization and difficulty in load balancing data scan operations across multiple instances, leading to slower performance compared to accessing data from volatile memory.
Innovation Solution
A connectionless, stateless parallel query server, known as the scan server, is introduced to divide scan tasks into independently executable fragments and distribute them across multiple nodes for parallel execution, utilizing a client/server model with a load balancer and inter-process communication manager to optimize resource allocation and inter-process communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If dedicated parallel query slaves are allocated at the database instance where the query is initiated, then scan operations can be executed in parallel, but processing resources are inefficiently utilized and load balancing across multiple instances is difficult to achieve
Solution Approach 1:
The patent segments the dedicated query slave functionality into a separate scan server component that is independent of database instances. Scan tasks are divided into fragments and distributed to multiple scan servers across different instances, allowing parallel execution while enabling centralized resource management and load balancing across the entire system.
Solution Approach 2:
The patent introduces a scan server as an intermediary component between database instances and storage systems. The scan server receives scan tasks from database instances, manages the parallel execution fragments, and coordinates resource allocation across multiple instances, thereby decoupling the query initiation instance from the actual scan execution resources.
2Speed
If dedicated query slaves reserve processing resources for scan operations, then scan performance is improved, but resources are blocked and cannot be directed to higher priority database queries
Solution Approach 1:
The patent implements dynamic resource allocation where scan servers can be dynamically created, scaled, and terminated based on system workload conditions. The scan server pool can be expanded during periods requiring intensive scan operations and reduced when resources are needed for other high-priority queries, providing flexibility while maintaining scan performance when needed.
Solution Approach 2:
The patent changes the parameter of resource allocation from static (dedicated slaves permanently reserved at each instance) to dynamic (scan servers that can be created, scaled, and terminated based on system conditions). This allows the system to adjust the number and capacity of scan servers according to workload demands, prioritizing scan operations when appropriate while freeing resources for other queries when needed.
3Speed
If in-memory columnar stores are used for frequently accessed data, then retrieval speed is dramatically improved, but the system becomes a multi-instance dual-format system where data may reside on different nodes, complicating scan optimization
Solution Approach 1:
The patent creates a universal scan server architecture that can handle scan operations for data stored in multiple formats (row-major and column-major) and located in multiple places (buffer cache, IMC stores, and disk). The scan server is designed to work with any data format and location, providing a unified interface that simplifies access to dual-format data across multi-instance systems.
Solution Approach 2:
The scan server acts as an intermediary that abstracts the complexity of data distribution across different nodes and formats. It receives scan requests, determines the optimal location and format of the required data (whether in buffer cache, IMC stores, or on disk across different instances), and coordinates the retrieval and assembly of results, thereby hiding the distribution complexity from the querying database instances.
Data Source
AI summary
A method, apparatus, and system for a multi-instance database scan server is provided to process data scan operations for segments within in-memory stores, such as in-memory columnar (IMC) stores. The scan server uses a client/server based model, wherein each database instance in the database management system (“DBMS”) is associated with a client-side scan server and a server-side scan server. The client-side scan server can reside in the data and/or index layer of the DBMS, receiving requests for scan operations on in-memory segments. After receiving a data scan request from an IMC client, the client-side scan server can create several independently executable scanning tasks to process the request. The scanning tasks are each assigned to a particular server-side scan server for processing based on one or more factors, which may include a load balancing component. Additional factors to be considered may include instance affinity and/or NUMA (Non-Uniform Memory Access) affinity.