Database I/O Cost Estimation via Queue Depth Calibration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face inaccuracies in estimating the cost of input/output (I/O) operations on stable storage, which affects the selection of execution plans for queries, as they do not adequately consider factors like band size and queue depth.
Innovation Solution
The Queue Depth Aware Disk Transfer Time (QDADTT) model is introduced, which estimates I/O costs based on band size and queue depth, providing a more accurate calculation of I/O times by calibrating these parameters and using them to determine the amortized costs of random I/O within a given band size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional query optimizers use conventional I/O cost estimation methods, then the system is simpler to implement, but the accuracy of execution plan selection deteriorates due to insufficient consideration of band size and queue depth
Solution Approach 1:
The patent changes the parameters used in I/O cost estimation from traditional simple metrics to include band size (number of pages accessed) and queue depth (number of concurrent I/O operations). This parameter expansion allows the cost model to accurately reflect real storage system behavior, resolving the contradiction between estimation accuracy and model complexity by making the complexity worthwhile through significantly improved precision.
Solution Approach 2:
The patent implements a calibration process that uses actual measured I/O performance data to adjust and refine the cost model parameters. By feeding back real-world measurements of storage access times and comparing them against model predictions, the system continuously improves estimation accuracy while keeping the model structure manageable through data-driven refinement rather than purely theoretical complexity.
2Measurement precision
If the database system performs detailed calibration of QDADTT model parameters, then I/O cost estimation accuracy improves, but system setup time and operational complexity increase
Solution Approach 1:
The patent performs calibration of the QDADTT model parameters during system initialization or setup phases, before the actual query processing begins. This preliminary calibration action allows the system to learn and store optimal cost model parameters in advance, so that during normal operation the system can quickly use these pre-computed values without repeating time-consuming calibration measurements, thus resolving the contradiction between calibration thoroughness and time consumption.
Solution Approach 2:
The patent creates a calibrated cost model that captures the essential I/O characteristics of the storage system in a simplified representation. Instead of performing complex calibration continuously, the system creates a copy of the calibrated parameters that can be quickly applied to multiple queries, effectively copying the results of expensive calibration work into a reusable model structure that speeds up subsequent operations.
3Reliability
If the query optimizer considers multiple factors (band size, queue depth, storage technology) in cost estimation, then execution plan selection accuracy improves, but the computational overhead increases
Solution Approach 1:
The patent introduces specific parameters (band size in pages, queue depth as integer) that capture the essential variability in I/O patterns without requiring complex continuous measurements. These discrete, easily computable parameters allow the optimizer to consider multiple factors while maintaining low computational overhead, as they can be quickly calculated from query execution plans without requiring heavy analysis of actual I/O traces during query processing.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for calibrating and using a stable storage model. An embodiment operates by generating, by a central computer, an access request for a stable storage, wherein the access request comprises a plurality of page accesses; measuring a cost to execute the access request on the stable storage; amortizing the cost over the plurality of page accesses; and calibrating, by the central computer, a stable storage model based on the amortized cost.