Scan Sharing for Column-Based In-Memory Database Query Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In-memory column-oriented database systems face high CPU cycles and memory bandwidth limitations during query processing due to inefficient predicate evaluation, particularly in data warehouse environments, where cache thrashing and high latency occur due to the lack of scan sharing and dynamic scheduling.
Innovation Solution
Implement a scan sharing method that groups queries based on column predicates and schedules shared scans dynamically, using real-time information to optimize memory usage and reduce cache thrashing, thereby improving query performance and throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If column-based in-memory database systems process queries independently without scan sharing, then each query can be evaluated individually, but memory bandwidth consumption increases and query latency increases due to redundant scans
Solution Approach 1:
The patent merges multiple queries into scan groups based on predicate compatibility, allowing shared scans to serve multiple queries simultaneously. This combining approach reduces redundant memory accesses and optimizes memory bandwidth utilization while maintaining query processing throughput.
Solution Approach 2:
The shared scan mechanism serves multiple queries with compatible predicates through a single scan operation. This multi-functional approach allows the same scan to fulfill multiple query requirements, reducing overall memory bandwidth consumption while improving system productivity.
2Loss of energy
If queries are processed with dynamic scan sharing and scheduling, then memory bandwidth efficiency improves, but system complexity increases due to grouping and scheduling mechanisms
Solution Approach 1:
The patent segments queries into scan groups based on predicate compatibility and organizes them hierarchically. This segmentation approach manages complexity by dividing the query processing workload into manageable groups while optimizing memory bandwidth efficiency through shared scans within each group.
Solution Approach 2:
The system dynamically adjusts query grouping and scan scheduling based on real-time conditions. This dynamic approach optimizes memory bandwidth efficiency by adapting to varying query workloads while managing system complexity through flexible, condition-based grouping and scheduling mechanisms.
3Loss of energy
If scan operations are shared among multiple queries, then memory bandwidth requirements are reduced, but query evaluation time may increase due to scheduling overhead
Solution Approach 1:
The patent performs preliminary query analysis to identify predicate compatibility and pre-group queries into scan groups before execution. This preliminary action reduces scheduling overhead during query evaluation by establishing groups in advance, thereby reducing memory bandwidth requirements without significantly increasing query evaluation time.
Solution Approach 2:
The system maintains continuous scan operations that serve multiple queries simultaneously through shared scans. This continuous approach reduces memory bandwidth requirements by eliminating redundant scans while minimizing query evaluation time through efficient, overlapping execution of compatible queries.
Data Source
AI summary
A plurality of queries of an in-memory column-oriented database are obtained. Each of the queries has at least one column predicate. For each of the queries, it is determined which of a plurality of columns in the in-memory column-oriented database need to be scanned. The plurality of queries are grouped into a plurality of scan groups, based at least in part on the determining step. The column predicates are evaluated for those of the queries in a given one of the scan groups in accordance with a scheduling scheme. The grouping step and the scheduling scheme take into account real time scan information.