Resource Queues for Database Memory Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional databases face challenges in managing large amounts of data, leading to 'Out of memory' errors due to insufficient system resources, and existing methods to prevent this are either expensive or not scalable.
Innovation Solution
Implementing a resource queue system that differentiates queries based on priority and allocates memory and concurrent query limits, allowing for dynamic memory allocation to operators within queries, and using a binary search algorithm to optimize memory allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If more physical memory is added to the database system, then the system can handle larger queries without 'Out of memory' errors, but the cost increases significantly
Solution Approach 1:
The patent segments the database system into multiple resource queues (e.g., low priority, normal priority, high priority queues) that divide and manage memory resources separately. Each queue has its own memory allocation limits, allowing the system to handle multiple queries concurrently with limited total memory by processing them in priority order rather than requiring all memory to be available simultaneously.
Solution Approach 2:
The patent implements dynamic memory allocation where the system adjusts memory allocation to queries based on their priority level and current system state. High priority queries can allocate more memory dynamically while low priority queries receive less, allowing the same physical memory to serve different query needs flexibly rather than requiring fixed over-provisioning.
2Reliability
If the amount of concurrent queries is reduced, then memory pressure is decreased and Out of memory errors are prevented, but the productivity and responsiveness of the database system decreases
Solution Approach 1:
The patent segments concurrent queries into different priority queues, allowing the system to maintain higher overall concurrency by managing queries in tiers. Multiple low-priority queries can run concurrently within their allocated memory limits, while high-priority queries receive dedicated resources, thereby increasing total productive capacity without causing memory exhaustion.
Solution Approach 2:
The patent changes the parameter of query prioritization by introducing priority levels and associated memory allocation rules. This allows the system to process more queries concurrently by adjusting memory allocation parameters dynamically based on query priority, rather than uniformly limiting all queries to the same conservative memory threshold.
3Device complexity
If uniform memory allocation is used for all queries, then the system is simple to implement, but efficient queries may starve while inefficient queries consume excessive resources
Solution Approach 1:
The patent applies local quality by assigning different memory allocation characteristics to different query priority levels. High-priority queries receive more generous memory allocation with higher limits, while low-priority queries receive more restrictive allocation. This localized differentiation optimizes resource distribution to match query importance without requiring complex per-query analysis.
Data Source
AI summary
A method, article of manufacture, and apparatus for processing queries, comprising receiving a query, determining a query classification for the query, assigning the query to a resource queue based on the determined query classification, and placing the query in the assigned resource queue. In some embodiments, the resource queue is divided into a plurality of slots, and the query is placed in a slot. The resource queue may be associated with a resource queue memory allocation, and each of the plurality of slots is associated with a slot memory allocation.