LTS Segment Retrieval Scheduling for Variable Query Execution Rates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, leading to inefficiencies in handling large volumes of data.
Innovation Solution
A database system architecture that employs parallelized data input, storage, and processing subsystems, along with a query and response system, to optimize data partitioning and query planning, allowing for independent execution of operations across multiple computing devices and nodes, thereby enhancing processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional sequential processing is used, then system complexity is low, but processing speed is limited by hardware constraints and data storage methods
Solution Approach 1:
The database system segments data into segments and divides query processing into multiple independent tasks that can be executed in parallel across different computing devices. Each segment can be processed independently, enabling concurrent operations that significantly increase processing speed while managing complexity through structured division of work.
Solution Approach 2:
The system transitions from single-threaded sequential processing to multi-dimensional parallel processing by introducing additional processing dimensions through multiple computing devices and nodes. This dimensional expansion enables simultaneous execution of processing operations, overcoming hardware speed limitations through concurrent rather than sequential operations.
2Speed
If parallel processing is implemented, then processing speed increases, but idle times between operations increase
Solution Approach 1:
The scheduling system ensures continuous useful action by dynamically allocating segments to available computing devices and maintaining a pipeline where segments are constantly being processed. The scheduler prevents idle times by ensuring that when one segment finishes, another segment is immediately assigned to an available device, creating continuous productive operation throughout the parallel processing pipeline.
Solution Approach 2:
The system employs dynamic scheduling that adapts to real-time processing conditions. The scheduler continuously monitors segment completion status and dynamically reassigns segments to available computing devices, optimizing the flow of work to minimize idle times. This dynamic adjustment allows the system to respond to varying processing speeds and availability conditions, maintaining high utilization across all processing nodes.
3Speed
If data is stored in traditional formats, then storage simplicity is maintained, but retrieval speed is limited
Solution Approach 1:
The database segments data into manageable units that can be independently retrieved and processed. This segmentation allows the system to retrieve only the necessary segments for a given query rather than scanning entire tables, significantly improving retrieval speed. The segmented structure enables parallel retrieval operations across multiple computing devices while maintaining manageable complexity through standardized segment formats.
Data Source
AI summary
A store and compute sub-system of a database system includes pluralities of computing nodes of a plurality of computing devices of a computing device cluster are operable to obtain a plurality of input/output expressions of a query regarding a dataset, wherein the query includes a plurality of expressions. The pluralities of computing nodes are further operable to, schedule, at an initial row retrieval rate, retrieval of a first set of LTS segments of the plurality of LTS segments from memory as at least some of operand data for a first set of IO expressions and reduce, by the first plurality of computing nodes, the initial row retrieval rate for scheduling retrieval of another LTS segment when the first plurality of computing nodes determines it is executing one or more IO expressions of the plurality of IO expressions on one or more previously retrieved segments at an unfavorable rate.