Dataset Sub-dataset Segmentation for Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analytical database systems face inefficiencies in handling large datasets due to the lack of indexing for online transaction processing and the need for improved sort technologies to manage data expansion, particularly in optimizing scan performance and query processing.
Innovation Solution
A method of sorting and storing datasets by dividing them into non-overlapping or overlapping sub-datasets based on attribute values, assigning these to data blocks with predefined storage limits, and creating attribute value information for selective processing, which optimizes data block usage and reduces false positives during query processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If the dataset is stored without indexing in analytical database systems, then storage simplicity is maintained, but query processing speed deteriorates
Solution Approach 1:
The dataset is divided into multiple sub-datasets based on attribute value ranges, with each sub-dataset stored in separate data blocks. This segmentation allows the system to quickly locate and process only relevant portions of data during queries, improving query processing speed without requiring traditional indexing structures.
Solution Approach 2:
The data is pre-divided into sub-datasets and organized in data blocks with predefined storage limits before queries are executed. This preliminary organization enables faster query processing by eliminating the need to scan entire datasets, while maintaining storage simplicity through straightforward block-based storage.
2Measurement precision
If the dataset is divided into many small data blocks, then query selectivity is improved, but the number of I/O operations increases
Solution Approach 1:
Multiple sub-datasets are merged into single data blocks where they fit within predefined storage limits. This merging reduces the total number of data blocks that need to be read during queries, decreasing I/O operations while maintaining the selectivity benefits of fine-grained data organization through attribute-based division.
3Productivity
If data is stored in sorted order to improve scan performance, then scanning efficiency is improved, but data insertion and update operations become more complex
Solution Approach 1:
The dataset is segmented into sub-datasets based on attribute value ranges and stored in separate data blocks. This segmentation provides natural sorting within blocks, improving scan performance for range queries while keeping insertion and update operations simpler compared to maintaining sorted order across the entire dataset.
Solution Approach 2:
Each data block maintains local sorting properties for its contained sub-datasets, which is sufficient for improving scan performance. This local quality approach avoids the complexity of global sorting while still achieving efficient data retrieval through attribute-based organization.
4Productivity
If traditional indexing is implemented to speed up point queries, then query speed is improved, but storage space consumption increases
Solution Approach 1:
The attribute-based data block organization serves multiple functions simultaneously: it enables fast point queries through direct attribute value range matching, supports range queries efficiently, and provides structured storage without requiring separate indexing structures. This multi-functionality achieves query speed improvement without additional storage space consumption for indexes.
Data Source
AI summary
Sorting and storing a dataset, the dataset comprising at least one attribute. The method includes defining a set of data blocks and assigning to each data block a predefined maximum number of entries or a predefined maximum amount of storage, dividing the dataset into a sequence of multiple sub-datasets each having one value or a range of values of the attribute, wherein each pair of successive sub-datasets of the sequence are non-overlapping or overlapping at their respective extremum value of the attribute, for each sub-dataset of the multiple sub-datasets: in case the sub-dataset fully or partially fits into a data block of the defined data blocks storing the sub-dataset into at least the data block, the sub-dataset that partially fits into the data block comprising a number of entries that is smaller than a predefined maximum threshold.