In-Memory Data Chunk Distribution for Cluster Query Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In a multi-node database cluster, loading the same data into the volatile memory of every node limits the cluster's caching capacity, as each node can only cache approximately the same amount of data as a single node, despite having N times the volatile memory of a single node.
Innovation Solution
Distributing distinct portions of database objects across multiple nodes, allowing each node to independently determine which chunks are assigned to it, and maintaining a sub-chunk-to-node mapping for efficient query execution, enabling parallel processing and redundancy to optimize volatile memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the same data is loaded into the volatile memory of every node in the cluster, then each node can quickly access the data without disk I/O, but the cluster's total caching capacity is limited to the memory size of a single node
Solution Approach 1:
The patent divides the database objects into distinct chunks and distributes these chunks across different nodes in the cluster. Each node stores only the chunks assigned to it, rather than duplicating all data. This segmentation allows the cluster to utilize the combined memory capacity of all nodes, achieving linear scaling of caching capacity with the number of nodes while maintaining fast in-memory access for the distributed data portions.
2Quantity of substance
If data is distributed across multiple nodes, then the cluster can utilize N times the volatile memory capacity, but query processing requires coordination and communication between nodes
Solution Approach 1:
The patent pre-establishes sub-chunk-to-node mappings that define which node hosts which data chunks before query execution. These mappings are maintained in each node's memory, allowing nodes to independently determine where data resides without real-time coordination. When a query is executed, the system can quickly route requests to the appropriate nodes using these pre-computed mappings, minimizing communication overhead and avoiding complex runtime coordination.
3Productivity
If distinct portions of database objects are distributed across nodes, then volatile memory usage is optimized and parallel processing is enabled, but maintaining consistency and transparency of data distribution becomes complex
Solution Approach 1:
The patent implements a universal sub-chunk-to-node mapping mechanism that serves multiple functions: it defines data location for query routing, enables parallel processing by identifying which nodes hold which data portions, and provides transparency by allowing any node to determine data locations without centralized coordination. This single mapping structure handles all distribution management needs, simplifying the system despite the distributed architecture.
Data Source
AI summary
Techniques are described herein for executing queries on distinct portions of a database object that has been separate into chunks and distributed across the volatile memories of a plurality of nodes in a clustered database system. The techniques involve receiving a query that requires work to be performed on data that resides in a plurality of on disk extents. A parallel query coordinator that is aware of the in-memory distribution divides the work into granules that align with the in-memory separation. The parallel query coordinator then sends each granule to the database server instance with local in memory access to the data required by the granule and aggregates the results to respond to the query.