Adaptive Batch Sizing for Memory-Bounded Record Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, particularly when handling large volumes of data.
Innovation Solution
A database system architecture that employs parallelized data input, storage, retrieval, and query processing across multiple computing devices, utilizing predictive models to optimize batch sizes and buffer intervals for efficient data management and query execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If parallelized data processing is implemented across multiple computing devices, then processing speed and productivity are improved, but device complexity increases
Solution Approach 1:
The system divides data processing into separate phases (bulk load phase and query execution phase) and segments operations across multiple computing devices. During bulk load, data is loaded in parallel across devices; during query execution, only necessary devices are activated. This segmentation enables high productivity while managing complexity through phased operations.
Solution Approach 2:
The system dynamically adjusts the number of active computing devices based on workload requirements. A predictive model determines the optimal batch size and which devices should be active, transitioning between different processing modes. This dynamic adaptation allows the system to achieve high processing speeds when needed while reducing complexity during idle periods.
2Loss of time
If batch sizes are optimized using predictive models, then query execution time is reduced, but measurement precision requirements increase
Solution Approach 1:
The system uses a predictive model that learns from historical data access patterns and feedback to optimize batch sizes. The model continuously refines its predictions based on actual usage patterns, reducing the need for highly precise measurements while achieving optimal query execution times through iterative improvement.
Solution Approach 2:
The system changes the parameter of batch size based on predicted data access patterns. By adjusting batch size parameters dynamically rather than using fixed precise measurements, the system reduces measurement precision requirements while optimizing query execution time through statistical predictions and adaptive parameter tuning.
3Productivity
If real-time data storage and concurrent query execution are enabled, then productivity improves, but reliability challenges increase
Solution Approach 1:
The system segments data storage and query execution into distinct phases with dedicated buffers. During bulk load, data is written to a first buffer; during query execution, a second buffer is used for reading. This segmentation prevents interference between writing and reading operations, maintaining reliability while enabling high throughput through parallel processing.
Solution Approach 2:
The system introduces intermediary buffers between data input and query execution processes. These buffers act as mediators that decouple the bulk load phase from the query execution phase, allowing real-time processing while maintaining data consistency through proper data flow management and synchronization mechanisms.
Data Source
AI summary
A record processing and storage system operates by: automatically determining a buffer interval for a mapping function of batch size to estimated memory utilization; automatically selecting a first batch size for performance of a first data format conversion based a first intersection between an upper bound of the buffer interval and a current memory availability; performing the first data format conversion upon a first batch of data having the first batch size; automatically updating the buffer interval based on a difference between an actual amount of memory utilized in performing the first data format conversion and an estimated memory utilization mapped to the first batch size via the mapping function; automatically selecting a second batch size for performance of a second data format conversion based a second intersection between an upper bound of the updated buffer interval and the current memory availability; and performing the second data format conversion upon a second batch of data having the second batch size.