Distributed In-Memory Database Query Execution Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed in-memory database systems face significant challenges due to the duplication and storage of large result sets during query execution, leading to inefficiencies and increased memory usage.
Innovation Solution
The proposed solution involves a method where queries are distributed to multiple shards, and each shard generates array provider data structures that identify responsive identifiers. These data structures are then materialized, with a portion of the responsive data being persisted to disk while materializing, allowing for efficient merging and reduction of in-memory data footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If each machine generates and stores complete result sets during query execution, then query processing can be performed in parallel across distributed machines, but significant data storage and memory usage is required
Solution Approach 1:
The patent segments the result set into two distinct components: (1) a consolidated list of unique identifiers (IDs) representing all responsive records across shards, and (2) the actual data records themselves. This segmentation allows the system to first aggregate and deduplicate IDs in memory efficiently, then selectively materialize only the necessary data records to disk, thereby reducing peak memory consumption while maintaining parallel query processing capability.
Solution Approach 2:
The patent performs preliminary aggregation of identifier lists from all shards before materializing the actual data records. By first collecting and consolidating the ID lists in memory (which are more compact than full result sets), the system prepares the groundwork for efficient data retrieval later. This preliminary action reduces the memory footprint during the intermediate processing stage while preserving the ability to generate complete result sets when needed.
2Speed
If complete result sets are materialized in memory for each shard, then fast in-memory access is achieved, but data duplication occurs across multiple machines
Solution Approach 1:
Instead of copying and materializing complete data result sets across all machines, the patent copies only the compact identifier lists from each shard to the central coordinating machine. These ID lists serve as references that can be quickly processed and deduplicated. The actual data records are then retrieved on-demand from the originating shards only when needed, eliminating redundant data duplication while maintaining fast access through efficient ID-based lookup.
3Productivity
If all responsive data is kept in memory during query execution, then immediate access and processing is possible, but memory resources are exhausted for large result sets
Solution Approach 1:
The patent implements a dynamic memory management strategy where the system adapts between in-memory and disk-based storage based on the size of the result set. For small to medium result sets, complete data can be kept in memory for fast access. For large result sets, the system automatically spills excess data to disk while maintaining the compact ID list in memory, thus dynamically adjusting resource usage to prevent memory exhaustion while preserving query execution efficiency.
Solution Approach 2:
The patent extracts and separates the identifier information from the actual data records. By taking out just the ID lists (which are significantly smaller than full result sets) and keeping them in memory, the system maintains fast processing capability for the critical identification and filtering operations. The bulkier data records are extracted from memory and stored on disk, reducing memory consumption while allowing on-demand retrieval when needed.
Data Source
AI summary
In some implementations, the techniques described herein relate to a method including: receiving, by a processor, a query from a client device; distributing, by the processor, the query to a plurality of shards; receiving, by the processor, a plurality of array provider data structures from the plurality of shards, a given array provider data structure identifying responsive identifiers from a corresponding shard; materializing, by the processor, the plurality of array provider data structures; persisting, by the processor, a portion of responsive data on disk while materializing the plurality of array provider data structures; merging, by the processor, data stored on the disk; and returning, by the processor, a result set based on the data to the client device.