Segment Handle Architecture for Parallel Database Query Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, leading to inefficiencies in query execution.
Innovation Solution
A database system architecture that utilizes segment handles to optimize data storage and processing through parallelized sub-systems, including a parallelized data input, store, retrieve, and process system, query and response system, and administrative and configuration systems, which enable efficient query execution and metadata management across multiple computing devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in traditional sequential formats, then storage simplicity is maintained, but query processing speed deteriorates
Solution Approach 1:
The patent divides data storage into segments with segment handles that reference specific data portions. This segmentation allows parallel query processing across multiple computing devices while maintaining organized storage structure, resolving the contradiction between processing speed and storage complexity.
Solution Approach 2:
The patent introduces a new dimension of segment handles as intermediate references between storage and processing. This handle-based approach adds a logical layer that enables parallel access without complicating physical storage, improving query speed while managing complexity through abstraction.
2Productivity
If parallel processing is implemented across multiple computing devices, then query execution efficiency is improved, but system complexity increases
Solution Approach 1:
Segment handles serve multiple functions: they identify data locations, enable parallel processing coordination, and facilitate metadata management. This multi-functionality allows a single mechanism to support parallel processing efficiency while avoiding the need for separate complex coordination systems.
Solution Approach 2:
Segment handles act as intermediaries between the storage system and parallel processing operations. This intermediary layer simplifies the interface between computing devices and stored data, enabling efficient parallel processing without direct complex device-to-device communication protocols.
3Speed
If segment handles are used for data reference, then query execution speed is enhanced, but metadata management complexity increases
Solution Approach 1:
Segment handles are pre-generated and stored with metadata during data ingestion, before query execution occurs. This preliminary organization of handle-reference pairs enables fast query execution without requiring complex real-time metadata management during processing operations.
Data Source
AI summary
A database system operates by: determining a set of assigned segments based on data ownership information; generating a plurality of segment handles and a corresponding plurality of segment metadata for the set of assigned segments based on performing a segment activation step for the set of assigned segments; determining a first query for execution requiring access to a first segment of the set of assigned segments; and executing an IO operator of the first query based on loading the first segment by utilizing a corresponding one of the plurality of segment handles.