Fragmented Query Execution Plans for In-Memory Database Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In-memory database systems face challenges in achieving sub-second response times due to high volumes of enterprise data, with traditional query execution plans (QEPs) leading to excessive main memory accesses and write latency, particularly when using non-volatile memory (NVM), which deteriorates with each write operation.
Innovation Solution
Implementing fragmented query execution plans (fQEPs) that partition time-consuming operators into multiple fragments, optimizing data to fit within on-chip caches, reducing main memory accesses, and using profiling tools to determine execution times and energy costs, thereby minimizing writes to NVM and improving response times.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional query execution plans are used to process high volumes of enterprise data, then query processing can be performed, but main memory accesses become excessive and write latency increases
Solution Approach 1:
The patent applies segmentation by dividing the query execution plan into multiple fragments, where each fragment processes a portion of the data. This is achieved by partitioning the input data into multiple segments and executing corresponding operations on each segment in parallel, thereby reducing the sequential main memory access time while maintaining overall query processing capability.
2Reliability
If non-volatile memory is used for data storage, then persistence is achieved, but write operations cause latency and deteriorate memory lifetime
Solution Approach 1:
The patent extracts critical data that requires persistence and moves it to dedicated non-volatile memory storage locations, separating this from the main processing memory. By taking out only the necessary persistent data rather than all data, the system reduces write operations to NVM, thereby extending its lifetime while maintaining required persistence for critical information.
3Productivity
If data is kept in main memory for in-memory computing, then throughput is improved, but memory resources become constrained in cloud environments
Solution Approach 1:
The patent implements dynamic memory management where the system adjusts the amount of data kept in main memory based on availability and access patterns. By dynamically loading only frequently accessed data into main memory and using fragmentation to process data in smaller chunks, the system maintains high throughput while adapting to constrained memory resources in cloud environments.
4Loss of time
If query execution plans are optimized for speed, then response time decreases, but energy consumption increases due to frequent main memory accesses
Solution Approach 1:
The patent applies preliminary action by pre-loading frequently accessed data into main memory before queries are executed. By using fragmentation to identify and pre-load only the necessary data portions, the system reduces subsequent main memory accesses during query processing, thereby lowering energy consumption while maintaining fast response times through cached data availability.
Data Source
AI summary
Implementations of the present disclosure include methods, systems, and computer-readable storage mediums for receiving a query from an application, determining that the query has been received before, and in response, retrieving statistics for the query from a metadata table, providing a query execution plan (QEP) associated with the query, the QEP including a plurality of operators executed to provide a query result, one or more operators of the plurality of operators requiring input from computer-readable memory, fragmenting at least one operator of the plurality of operators into multiple fragment operators to provide a fragmented QEP (fQEP), the multiple operators being executed to each provide a portion of a result of the at least one operator, and storing the fQEP in computer-readable memory.