Parallelized Database Page Generation for Faster Query Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, leading to inefficiencies in query execution.
Innovation Solution
A parallelized database system architecture that includes a parallelized data input, store, retrieve, and process sub-system, along with a query and response sub-system, which optimizes query plans using cost analysis and distributes query execution across multiple computing devices and nodes, enabling efficient storage and processing of large datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in traditional sequential storage methods, then storage simplicity is maintained, but query processing speed is limited by hardware constraints
Solution Approach 1:
The patent divides data into columnar segments organized in immutable data files, where each segment contains data for a specific column. This segmentation enables parallel processing of different columns across multiple computing devices, significantly improving query processing speed while maintaining manageable system complexity through structured organization.
Solution Approach 2:
The patent transitions from traditional row-based sequential storage to a columnar storage architecture, adding a dimensional transformation to the data organization. This allows queries to access specific columns across multiple data files in parallel, effectively utilizing additional processing dimensions to overcome hardware speed limitations.
2Productivity
If co-processing options are restricted, then system simplicity is maintained, but execution efficiency is limited
Solution Approach 1:
The patent creates a universal query execution framework that can handle multiple types of queries (filtering, aggregation, joins) across different data formats and storage configurations. The system provides adaptable co-processing capabilities through standardized interfaces that work across various computing devices, improving execution efficiency without sacrificing versatility.
Solution Approach 2:
The patent implements dynamic query execution plans that adapt to available computing resources and data characteristics. The system can dynamically allocate processing tasks across multiple devices, adjust parallelization strategies, and optimize execution paths based on real-time conditions, thereby improving productivity while maintaining adaptability.
3Loss of time
If data is processed sequentially, then resource allocation is simple, but processing time increases for large datasets
Solution Approach 1:
The patent segments large datasets into smaller columnar chunks that can be processed independently in parallel across multiple computing devices. Each device processes its assigned segment simultaneously, dramatically reducing overall processing time while the segmentation provides a clear framework that manages the complexity of parallel coordination.
Solution Approach 2:
The patent merges results from multiple parallel processing operations into a unified output. The system combines partial results from different computing devices, integrates aggregated data, and produces final query results by merging intermediate outcomes, thereby reducing total processing time while using structured merging operations to control architectural complexity.
Data Source
AI summary
A parallelized data input sub-system of a database system includes pluralities of processing core resources of pluralities of loader nodes of pluralities of computing devices of a plurality of computing device clusters. A first set of loader nodes of the pluralities of loader nodes is operable to ingest a plurality of datasets, determine a first set of processing core resources of the pluralities of processing core resources for ingesting the first dataset, determine a second set of processing core resources of the pluralities of processing core resources for ingesting the second dataset, direct, via first routing data, the first dataset to the first set of processing core resources, for temporary storage as a first set of pages, and direct, via second routing data, the second dataset to the second set of processing core resources for temporary storage as a second set of pages.