Database Table Chunking for Memory-Constrained Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large datasets exceeding available memory limits pose challenges for real-time data analysis, as existing database systems struggle to efficiently process data without loading entire tables into memory, leading to increased processing times and resource utilization.
Innovation Solution
The method involves dividing database tables into data chunks, with each chunk processed independently, allowing for parallel or serial processing based on query requirements, and ensuring unique row identifiers are persistent across chunks to facilitate efficient query execution and reduce memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the entire table is loaded into memory for processing, then query execution speed is improved, but memory usage exceeds available memory limits
Solution Approach 1:
The patent divides the table into multiple data chunks that can be processed independently. Each chunk is loaded into memory separately, allowing the system to process large tables that exceed available memory by working with smaller manageable segments rather than loading the entire table at once.
2Quantity of substance
If data is processed in chunks, then memory usage is reduced, but processing time increases due to multiple loading operations
Solution Approach 1:
The patent performs preliminary actions by pre-processing metadata and maintaining persistent row identifiers across chunks before actual query execution. This preparation work enables more efficient processing during the actual query execution phase, reducing the overall time penalty of chunked processing.
3Quantity of substance
If data chunks are processed independently, then memory resources are optimized, but query complexity increases due to chunk management overhead
Solution Approach 1:
The patent introduces an intermediary layer of chunk metadata and persistent row identifiers that mediates between the independent data chunks and the query processing logic. This intermediary structure simplifies the complexity by providing a unified interface for querying across chunks while maintaining the memory efficiency of independent processing.
4Productivity
If row identifiers are not persistent across chunks, then chunk independence is maintained, but query accuracy decreases due to duplicate or missing rows
Solution Approach 1:
The patent implements universal persistent row identifiers that function across all data chunks simultaneously. Each row identifier serves multiple purposes: maintaining chunk independence for efficient processing while also ensuring query accuracy by providing a consistent reference that works universally across the entire table regardless of chunk boundaries.
Data Source
AI summary
A database server receives a request to perform a primary query on a table of a database. A first table query can be generated and can include a starting row identifier, ROW A, and a number of rows, n, for generating a data chunk from the table of the database. Multiple table queries can be performed each having a different starting row identifier and each defining the number of rows forming a data chunk. The primary query can be extended with the first table query in preparation for performing the primary query on the first data chunk.