Z-Curve Database Indexing for Multi-Column Query Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large distributed databases face inefficiencies in data retrieval and indexing due to the need for global indexing across multiple computer systems, leading to bottlenecks and slow performance, especially with large data stores having many columns and rows.
Innovation Solution
The use of a multidimensional data structure with a z-curve and z-tree to efficiently manage data, where a query box specifies data ranges, and a state engine traverses the z-tree using de-interleaved addresses to track states and eliminate unnecessary nodes, and a distance engine uses a heap-based approach for nearest neighbor analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If global indexing is used across multiple computer systems to enable data retrieval, then data accessibility is improved, but system performance deteriorates due to bottlenecks and slow syncing
Solution Approach 1:
The patent divides the global index into local indexes stored on individual computer nodes. Each node maintains its own local index for the data it stores, eliminating the need for a centralized global index. This segmentation allows each node to operate independently for query processing, removing the bottleneck caused by centralized index management and syncing operations.
2Productivity
If files are stored in sorted order to eliminate syncing needs, then system scalability is improved, but query performance deteriorates when multiple columns are involved
Solution Approach 1:
The patent introduces a multi-dimensional indexing structure where data can be indexed along multiple column dimensions simultaneously. Instead of sorting files sequentially by a single column, the system creates index structures that allow traversal along any column dimension, enabling efficient queries on multiple columns without requiring multiple sequential sorts.
3Loss of information
If multiple sequential sorts are performed on data columns to handle multi-column queries, then query completeness is improved, but processing time deteriorates significantly
Solution Approach 1:
The patent performs preliminary indexing operations during data insertion, organizing data into a multi-dimensional index structure in advance. When a multi-column query arrives, the system can directly traverse the pre-built index structure along the relevant dimensions without performing sequential sorts at query time. This preliminary organization of data eliminates the need for time-consuming sequential sorting operations.
Data Source
AI summary
Improved access of a multidimensional database can be implemented using a space-filling curve, such as a z-order curve, and a tree structure. A state engine is configured to track the state of the dimensions of the database as the tree structure is traversed. Different states allow optimizations such as avoidance of analyzing a dimension for further nodes of the tree and removal of multiple nodes in a sub-tree.