Database Query Scheduling for Parallel Segment Chunking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, leading to inefficiencies in handling large volumes of data.
Innovation Solution
A database system architecture that employs parallelized data input, storage, and query processing across multiple computing devices and nodes, utilizing segment-based data partitioning and optimized query planning to enhance processing efficiency and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored and processed using traditional sequential methods, then hardware simplicity is maintained, but processing speed and efficiency deteriorate when handling large volumes of data
Solution Approach 1:
The patent segments data into fixed-size page units (e.g., 4KB pages) and divides the database into multiple segments that can be independently processed. This segmentation enables parallel processing across multiple computing nodes while maintaining manageable unit sizes, thus improving processing speed without requiring overly complex system architecture.
Solution Approach 2:
The patent introduces a new dimension of parallel processing by distributing data segments across multiple computing nodes simultaneously. Instead of sequential single-node processing, the system processes multiple segments in parallel across different nodes, dramatically improving processing speed for large data volumes while keeping individual node complexity manageable.
2Productivity
If traditional data storage and processing methods are used, then system simplicity is maintained, but productivity deteriorates when handling massive data volumes
Solution Approach 1:
By dividing the database into multiple segments of fixed-size pages, the system enables parallel processing operations. Each segment can be processed independently by different computing nodes, significantly increasing data processing throughput while maintaining relatively simple processing logic at each node level.
Solution Approach 2:
The patent introduces a query processing system that acts as an intermediary between the user query and the distributed data segments. This intermediary manages the complexity of coordinating parallel processing across multiple nodes, handling query optimization, segment routing, and result aggregation, thereby improving productivity while shielding users from architectural complexity.
3Reliability
If data is processed in large batches without segmentation, then processing steps are reduced, but processing precision and data integrity deteriorate due to hardware limitations
Solution Approach 1:
The patent divides data into fixed-size page segments that can be processed independently and verified individually. This segmentation allows for better error detection, consistency checks, and data integrity verification at each segment level, improving overall reliability while distributing processing management across multiple manageable units rather than one large complex operation.
Data Source
AI summary
A processing module of a database system is operable to determining a query for execution. A plurality of row sets are processed over a temporal period in conjunction with execution of the query based on automatically selecting a requested number of rows to be included in each row set based on row scheduling parameters, generating a scheduling request for the requested number of rows, and receiving each row set for processing in response to the scheduling request via access to one segment of a set of segments required for execution of the query. A query resultant for the query is generated based on processing of the plurality of row sets, and further based on processing of an additional plurality of rows included in the set of segments via a set of other processing modules implemented in parallel with the processing module.