Query Processing via Data Subset Decomposition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data analysis systems face challenges in providing prompt and accurate query results due to the high dimensionality and large size of data sets, leading to slow processing times and inefficient use of resources, especially in interactive data exploration scenarios where quick decision-making is required.
Innovation Solution
The system decomposes queries into subqueries, allowing each subquery to be analyzed using pre-stored data subsets that cover specific dimensions, reducing the need to access the entire data set and enabling faster query processing without requiring prior knowledge of data distribution or historical workloads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the entire data set is accessed to cover all target dimensions, then query accuracy is improved, but query processing time increases significantly
Solution Approach 1:
The patent divides the entire data set into multiple data subsets, where each subset covers specific target dimensions. When a query is received, only the relevant subsets needed to cover the query's target dimensions are accessed and analyzed, rather than processing the complete data set. This segmentation enables the system to maintain query accuracy by covering all necessary dimensions while significantly reducing processing time by excluding irrelevant data portions.
2Speed
If data subsets are pre-stored for fast access, then query processing speed is improved, but storage requirements increase
Solution Approach 1:
The patent implements local quality by creating data subsets with different characteristics optimized for specific query types and dimension combinations. Each subset is tailored to cover particular target dimensions efficiently, allowing the system to store multiple specialized subsets rather than one large generic data set. This approach improves query processing speed for specific dimension combinations while managing storage requirements through targeted, purpose-built subsets.
3Adaptability or versatility
If multiple data subsets are used to cover target dimensions, then query coverage is improved, but system complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-dividing the data set into multiple subsets before query processing, with each subset pre-configured to cover specific target dimensions. This pre-organization eliminates the need for complex runtime decisions about data partitioning. When a query arrives, the system simply identifies which pre-existing subsets are needed to cover the query's target dimensions and processes them, thereby improving query coverage while keeping the system relatively simple through advance preparation.
Data Source
AI summary
In implementations of the subject matter described herein, a solution for query processing is provided. In this solution, data subsets are pre-stored for example in a fast access storage device for data analysis, each including data entries corresponding to one or more dimensions. If two or more data subsets are needed to cover target dimensions corresponding to query items in a received query, instead of turning to analyze a source data set that is not stored, the query is decomposed into subqueries. By means of the decomposing, the target dimension(s) corresponding to the query item(s) in each subquery can be covered by a single data subset. The data subset is analyzed for each subquery and a query result for the query is determined based on analysis results of the subqueries. In such way, the query result for the query can obtained in a fast manner from the available data subsets.