Columnar Data Sorting Across Distributed Nodes for Faster Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems face limitations in processing speed due to hardware constraints, data storage methods, and restricted co-processing options, which hinder efficient data processing and query execution.
Innovation Solution
A parallelized database system architecture that divides data into partitions, segments, and slabs, using key columns for sorting and redundancy encoding, and employs a distributed processing approach across multiple computing nodes for parallel data input, storage, retrieval, and query processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in traditional database systems with centralized processing, then data consistency is maintained, but processing speed and query execution time are limited by hardware constraints
Solution Approach 1:
The patent divides data into partitions, segments, and slabs, and distributes processing across multiple computing nodes. Each node independently processes portions of data in parallel, eliminating the centralized processing bottleneck while maintaining data consistency through coordinated segment identifiers and version vectors.
2Loss of time
If multiple computing nodes are used for parallel processing, then query execution time is reduced, but data consistency and coordination become more difficult to maintain
Solution Approach 1:
The patent implements version vectors and segment identifiers that provide continuous feedback about data state across distributed nodes. Each node tracks versions of segments it has processed, enabling automatic detection and resolution of consistency conflicts without centralized coordination, thus maintaining reliability while enabling parallel execution.
3Speed
If data is sorted and organized for efficient querying, then retrieval speed improves, but storage space and processing overhead increase
Solution Approach 1:
The patent organizes data into hierarchical segments and slabs with segment identifiers that enable efficient indexing and retrieval. By segmenting data at multiple levels and using compact segment ID structures, the system achieves fast retrieval through targeted segment access while minimizing redundant storage overhead compared to fully sorted structures.
Data Source
AI summary
A method includes receiving, by a first computing entity of a database system, data that is organized in rows and columns. The method further includes determining, by the first computing entity, one or more key columns from the columns based on a desired sort criteria for the data. The method further includes sorting, by the first computing entity, other columns of the columns based on the one or more key columns to produce sorted other columns. The method further includes sending, by the first computing entity, the one or more key columns to a second computing entity of the database system for storage in a first storage location associated with the second computing entity. The method further includes sending, by the first computing entity, the sorted other columns to the second computing entity for storage in a second storage location associated with the second computing entity.