Parallelized Database Query Execution with Columnar Segments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, which hinder efficient execution of functions, especially when handling large volumes of real-time and non-real-time data.
Innovation Solution
A parallelized database system architecture that divides data into segments, optimizes query plans, and distributes processing across multiple nodes and storage clusters, enabling concurrent execution of queries and efficient data storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in traditional database systems, then data can be persisted and queried, but processing speed is limited by hardware constraints and storage methods
Solution Approach 1:
The patent segments data into multiple partitions and distributes them across different storage nodes. Each partition can be processed independently by different query processing nodes, enabling parallel execution of queries. This segmentation eliminates the single-point bottleneck of traditional databases and allows the system to scale processing speed by adding more nodes.
Solution Approach 2:
The patent introduces a new architectural dimension by separating storage and query processing into independent, distributed components. Instead of a monolithic database system, it creates a multi-dimensional architecture where storage nodes and query processing nodes operate independently but coordinate through a distributed file system, enabling massive parallel processing.
2Quantity of substance
If data volume increases, then more data can be stored, but processing time increases and scalability is hindered
Solution Approach 1:
By partitioning data across multiple storage nodes and enabling independent query processing on each partition, the system can handle increasing data volumes without proportionally increasing processing time. Parallel query execution on segmented data allows the system to maintain consistent performance as data scale grows.
Solution Approach 2:
The patent changes the fundamental parameters of data storage and processing by using a distributed file system with columnar data format and compression. These parameter changes enable efficient storage of massive data volumes while maintaining fast query performance through vectorized processing and selective data retrieval.
3Speed
If traditional storage methods are used, then data can be stored, but execution speed is limited
Solution Approach 1:
The patent replaces traditional row-based storage mechanics with columnar storage and compression techniques. This substitution enables more efficient data retrieval by storing only the necessary columns for each query and using compression to reduce I/O operations, significantly improving execution speed.
Solution Approach 2:
The patent fundamentally changes storage parameters by implementing columnar data organization, compression algorithms, and vectorized processing. These parameter changes transform the storage system from a general-purpose sequential access system to a specialized high-speed query processing system.
Data Source
AI summary
A database system is operable to generate and a first plurality of column-formatted segments from a first plurality of rows. A query indicates parameters for generating a result set, and further indicating an instruction to store the result set. A query operator execution flow that includes a loading operator is generated. The query is executed based on accessing at least one of the first plurality of rows, processing the at least one of the first plurality of rows to generate a second plurality of rows as the result set, and executing the loading operator. At least one new column-formatted segment is from the second plurality of rows based on execution of the loading operator. The at least one new column-formatted segment is stored access in future query executions.