In-Memory Store Optimization via Heatmap Knapsack Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in automatically and efficiently maintaining an in-memory area, such as an in-memory columnar store (IMCS), due to changing data access patterns, which requires manual intervention and significant computational resources for periodic optimization.
Innovation Solution
A periodic performance optimizer (PPO) uses a heatmap to track database access statistics and selects the most appropriate knapsack algorithm based on available resources, optimizing the configuration of the in-memory store by automatically selecting and dynamically changing the to-be-mirrored elements, thereby simplifying maintenance and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If test queries are executed periodically to determine optimal elements for in-memory loading, then the in-memory store configuration can be optimized for current access patterns, but significant computational resources are consumed and manual intervention is required
Solution Approach 1:
The system performs preliminary analysis by executing test queries before determining the optimal in-memory configuration. This preliminary action identifies which database elements should be loaded into memory in advance, avoiding the need for repeated computational analysis during production operations. The test query results are stored and reused to guide configuration changes.
Solution Approach 2:
The system automatically generates and applies configuration changes based on test query results without requiring manual database administrator intervention. The automated process selects optimal elements, generates configuration files, and applies changes to the in-memory store, enabling the system to self-optimize based on observed access patterns.
2Adaptability or versatility
If the in-memory store configuration is updated periodically to adapt to changing access patterns, then performance is maintained, but manual intervention and computational resources are required
Solution Approach 1:
The system automatically monitors access patterns, executes test queries, and generates configuration changes without manual intervention. The database administrator simply initiates the process, and the system autonomously determines optimal elements, creates configuration files, and applies changes, eliminating the need for manual schema conversion operations.
Solution Approach 2:
The system continuously monitors database access patterns and uses this feedback to dynamically adjust the in-memory configuration. Test queries analyze current access patterns, and the results feed into the configuration generation process, creating a closed-loop system that automatically adapts to changing workloads.
3Measurement precision
If computational resources are reserved to run test queries for each refresh, then optimal configuration can be determined, but resource availability is reduced during production operations
Solution Approach 1:
The system schedules test query execution periodically during maintenance windows or low-activity periods rather than continuously during production operations. This periodic action allows comprehensive configuration analysis to be performed with high accuracy while minimizing interference with production computational resources.
Solution Approach 2:
The system executes a limited set of representative test queries that provide sufficient information for configuration optimization without exhaustively testing all possible scenarios. This partial action approach achieves adequate optimization accuracy while consuming fewer computational resources than exhaustive analysis.
Data Source
AI summary
A method, apparatus, and system for periodic performance optimization through heatmap based management of an in-memory area are provided. A heatmap is maintained to track database accesses, and a sliding most recent time window of the heatmap is externalized to a desired granularity level to provide access statistics regarding candidate elements to be possibly placed in the in-memory area. Initially and on a periodic basis, an appropriate knapsack algorithm is chosen based on an analysis on the computational costs versus the benefits of applying various knapsack algorithms for the candidate elements. Using the chosen algorithm in conjunction with a selected performance model, an optimized configuration of the in-memory area is determined. The optimized configuration indicates a set of elements chosen from the candidate elements, optionally specified with compression levels. A task scheduler then schedules the appropriate tasks, working in a coordinated fashion, to reconfigure the in-memory area according to the optimized configuration.