Adaptive Sampling for Autonomous Database Services
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems face inefficiencies in collecting data statistics, leading to bottlenecks in database tuning and machine learning model inference due to high resource consumption and inaccuracies caused by inadequate sampling methods.
Innovation Solution
Adaptive sampling techniques that establish a feedback loop with machine learning models to iteratively collect statistics until convergence criteria are met, allowing for efficient and accurate estimation of database content statistics, and enabling rapid recalibration of statistics types and quality as ML models evolve.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dedicated statistics gathering modules are used to continuously maintain data statistics, then statistics availability is improved, but system throughput deteriorates due to resource consumption
Solution Approach 1:
The patent implements periodic sampling at configurable intervals (e.g., every N rows or M seconds) rather than continuous monitoring, allowing the statistics gathering module to take measurements at regular intervals while remaining inactive between intervals, thus reducing resource consumption and preserving system throughput while still maintaining statistics availability
Solution Approach 2:
The patent uses sampling techniques that examine only a partial portion of data (e.g., 1% sample rate) rather than processing all data continuously, providing sufficient statistics for ML models while consuming minimal system resources and avoiding throughput degradation
2Reliability
If on-demand statistics collection is performed during histogram creation, then statistics availability is improved, but collection time and analysis time increase significantly
Solution Approach 1:
The patent performs statistics collection in advance during low-utilization periods or background operations, preparing statistics before they are needed for ML model inference, thus ensuring availability without causing time delays when the statistics are actually required for decision-making
Solution Approach 2:
The patent creates sampled copies of data (e.g., 1% sample) to generate statistics, rather than processing the entire dataset, dramatically reducing collection time while producing sufficient statistics for accurate ML model predictions
3Ease of manufacture
If hard-coded statistics collection is used for query optimization, then collection simplicity is improved, but adaptability to evolving tuning automation requirements deteriorates
Solution Approach 1:
The patent implements dynamic, configurable statistics collection where parameters such as sample rate, collection interval, and data sources can be adjusted based on workload characteristics and ML model requirements, allowing the system to adapt to evolving tuning automation needs while maintaining operational simplicity through centralized configuration
4Productivity
If manual sampling is used to collect statistics before database tuning, then tuning analysis speed is improved, but performance bottlenecks occur due to large table sizes
Solution Approach 1:
The patent dynamically adjusts sampling parameters (sample rate, sample size, interval) based on table size, data distribution, and required precision, automatically optimizing the balance between collection speed and accuracy without requiring manual intervention or complex user configuration
5Ease of manufacture
If fixed sample size sampling is used, then collection simplicity is improved, but measurement precision deteriorates due to insufficient sample quality
Solution Approach 1:
The patent implements feedback mechanisms where ML model performance on sampled data is monitored and used to adjust future sampling strategies, increasing sample size or refining sampling intervals when precision requirements are not met, while maintaining simplicity through automated closed-loop control
Data Source
AI summary
Herein are approaches for self-optimization of a database management system (DBMS) such as in real time. Adaptive just-in-time sampling techniques herein estimate database content statistics that a machine learning (ML) model may use to predict configuration settings that conserve computer resources such as execution time and storage space. In an embodiment, a computer repeatedly samples database content until a dynamic convergence criterion is satisfied. In each iteration of a series of sampling iterations, a subset of rows of a database table are sampled, and estimates of content statistics of the database table are adjusted based on the sampled subset of rows. Immediately or eventually after detecting dynamic convergence, a machine learning (ML) model predicts, based on the content statistic estimates, an optimal value for a configuration setting of the DBMS.