Bulk Query Load Balancing via Keyset Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Bulk processing of database records often results in unevenly distributed workload and memory usage due to unpredictable processing loads across batches, making it difficult to manage efficiently.
Innovation Solution
The approach involves determining bulk sets with multiple keysets, each containing unique dimension attribute values, and calculating load scores to transfer keysets between bulk sets to minimize the total difference in load scores, thereby achieving uniform load impacts and reduced memory footprints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If bulk processing is used to query database records in batches, then productivity is improved, but the workload and memory usage become unevenly distributed
Solution Approach 1:
The patent segments the bulk query workload into multiple individual query operations, each targeting a specific subset of records. By breaking down the bulk processing into smaller, manageable query units, the system achieves more uniform distribution of workload and memory usage across processing batches, while still maintaining the productivity benefits of bulk data retrieval
Solution Approach 2:
The patent dynamically adjusts query parameters such as batch size, record selection criteria, and processing intervals based on system load and performance metrics. This parameter optimization enables the system to balance productivity with uniform workload distribution, preventing memory overload while maintaining efficient data retrieval rates
2Productivity
If bulk processing is used to query database records, then productivity is improved, but memory usage increases and becomes unevenly distributed
Solution Approach 1:
The patent divides the bulk data retrieval operation into segmented query batches, where each batch processes a controlled subset of records. This segmentation limits the amount of data held in memory at any given time, reducing the overall memory footprint while maintaining productive data retrieval through systematic batch processing
Solution Approach 2:
The patent implements a mechanism where processed data is discarded from memory after use, and memory resources are recovered for subsequent batches. This approach prevents memory accumulation and ensures consistent, manageable memory usage levels throughout the bulk processing operation, while productivity is maintained through continuous batch retrieval
Data Source
AI summary
A computer-implemented method includes determining a plurality of bulk sets for querying database records. The method also includes assigning a plurality of keysets to the plurality of bulk sets, with each keyset comprising a unique set of dimension attribute values from the database records. The method also includes calculating a predicted load score of each bulk set. The method also includes performing a transfer of a keyset from a first bulk set to a second bulk set when the transfer reduces a difference between predicted load scores of the first bulk set and the second bulk set. The method also includes, after the transfer, executing bulk queries using the plurality of bulk sets.