Many-Core In-Memory Column Store Query Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems face inefficiencies in query execution on many-core processor systems due to lengthy recursive processes and lack of explicit memory access patterns, leading to suboptimal performance and increased latency.
Innovation Solution
The method involves pattern analysis of query definitions to generate a query plan using pre-compiled query execution sub-components, optimizing the sequence of table processing and memory access, and employing pseudo code representation to reduce the number of passes through tables, thereby improving query execution speed and resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional recursive query execution processes are used on many-core systems, then query processing can be performed, but execution speed is slow and latency is high
Solution Approach 1:
The patent pre-compiles query execution sub-components and stores them in a library before query execution. When a query is received, the system identifies patterns and assembles pre-compiled sub-components to form the execution plan, avoiding recursive compilation during query execution. This preliminary preparation significantly reduces query latency and execution time on many-core systems.
Solution Approach 2:
The patent divides query execution into independent, pre-compileable sub-components (such as scan, filter, join, aggregation operations) that can be separately optimized and assembled. This segmentation allows the system to reuse compiled sub-components across multiple queries and efficiently parallelize execution across many cores without the overhead of full recursive compilation.
2Productivity
If column-based storage is used, then memory bandwidth is improved and access efficiency increases, but memory access patterns become more fine-grained requiring better optimization
Solution Approach 1:
The patent changes the memory access parameters by organizing data in column-based storage and using explicit memory access patterns in the compiled query sub-components. This allows the system to optimize for sequential access within columns while managing the fine-grained access requirements through structured access patterns defined in the pre-compiled code.
3Speed
If in-memory storage is used, then bandwidth availability improves and latency reduces, but hardware and energy costs increase
Solution Approach 1:
The patent creates a copy of frequently accessed query execution logic in the form of pre-compiled sub-components stored in memory. This allows rapid execution without repeated compilation overhead, reducing the energy cost per query while maintaining high-speed in-memory access. The system copies only the essential execution logic rather than duplicating entire data sets.
Data Source
AI summary
A pattern can be identified in at least part of a query whose definition is received in a query request. The identified pattern can be matched with a set of pre-defined patterns, each of which has associated therewith at least one pre-compiled query execution sub-component of a plurality of pre-compiled query execution sub-components retained in a library. A plan for executing the query can be generated, for example by incorporating the pre-compiled query execution sub-component associated with the matched pattern into the plan based on a pseudo code representation of the plan derived from the definition.