Partition-Based Progressive Top-K Subarray Query Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for processing top-k subarray queries in array databases are inefficient, as they require computing scores of all subarrays, leading to high computational costs and decreased usability, and do not support distributed processing or prevent memory overflow in large datasets.
Innovation Solution
The proposed method, PPTS (Partition-based Progressive Top-k Subarray Query Processing), partitions the array to prune the search space, allows progressive query processing, and supports distributed processing by dividing the array into chunks, enabling efficient top-k subarray query processing without computing all subarrays and optimizing memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the scores of all subarrays are computed using the naive method, then the top-k subarrays can be selected, but the computation is very time consuming and memory intensive
Solution Approach 1:
The array is divided into multiple partitions, and the query processing is segmented into iterative steps. Each iteration processes a subset of partitions and returns partial results, avoiding the need to compute all subarray scores at once. This segmentation enables progressive query processing that reduces time consumption while maintaining accuracy.
Solution Approach 2:
Instead of computing scores for all subarrays (excessive action), the method computes scores for only a subset of subarrays in each iteration (partial action). By strategically selecting which partitions to process next based on current top-k results, the method achieves acceptable accuracy with significantly reduced computation time.
2Measurement precision
If the scores of all subarrays are computed, then the top-k subarrays can be identified, but usability is severely decreased because users cannot know the answers until all computations are complete
Solution Approach 1:
The query processing is organized as periodic action with iterative iterations. Each iteration periodically returns intermediate top-k results to the user, allowing them to review partial answers and potentially adjust query parameters or stop processing early. This periodic feedback mechanism dramatically improves usability while maintaining the option to achieve full accuracy if needed.
3Productivity
If distributed processing is implemented to handle large array data, then processing capacity is increased, but coordination and data distribution add system complexity
Solution Approach 1:
The array data and query processing are segmented across multiple distributed nodes. Each node handles specific partitions independently, processing local subarrays and returning results. This segmentation enables scalable processing capacity while minimizing coordination complexity, as nodes operate largely autonomously with simple result aggregation at the coordinator.
Data Source
AI summary
A method of processing top-k (k is a natural number) query in array data is described. The method of processing the top-k query is performed in a computing device including at least a processor, and includes dividing an array including a plurality of cells into a plurality of partitions, and finding an ith answer in the array.