Microcube Query Clustering for Enterprise Data Redundancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing enterprise software systems face significant processing redundancy when executing multiple queries that specify similar dimensions, measures, and filters, leading to inefficient data retrieval.
Innovation Solution
The implementation of a microcube system, which clusters and modifies queries to use an intermediate result set, optimizing query execution by grouping queries based on common dimensions, measures, and filters, using a clustering algorithm and scoring function to determine query grouping for efficient execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple queries are executed separately on a remote data source, then each query can be processed independently, but significant processing redundancy occurs leading to inefficient data retrieval
Solution Approach 1:
The patent combines multiple similar queries into a single aggregated query by identifying common dimensions, measures, and filters. The system merges queries that share overlapping data requirements, executes the combined query once, and then distributes results to satisfy individual query requirements, thereby eliminating redundant processing operations.
Solution Approach 2:
The patent segments the query processing into distinct phases: query analysis and grouping, aggregated query generation, execution of aggregated query, and result distribution. This segmentation allows the system to handle multiple queries systematically by first identifying similarities, then processing them collectively, and finally delivering individualized results.
2Loss of time
If queries are batched together, then processing overhead is reduced, but the remote data source still processes each query separately generating separate result sets
Solution Approach 1:
The system performs preliminary analysis of queries to identify commonalities in dimensions, measures, and filters before execution. By pre-grouping queries with similar requirements and pre-generating the aggregated query structure, the system prepares the processing framework in advance, allowing efficient execution without redundant operations during the actual query processing phase.
Solution Approach 2:
The patent introduces an intermediary layer between the query interface and the data source that acts as a query optimization engine. This intermediary analyzes incoming queries, groups them by similarity, generates optimized aggregated queries, and manages result distribution, thereby simplifying the overall processing complexity while reducing execution time.
3Reliability
If each query is processed independently, then query results are accurate and complete, but processing redundancy leads to significant performance degradation
Solution Approach 1:
The system changes the parameters of query processing by transforming individual query parameters (dimensions, measures, filters) into grouped parameter sets. Queries with identical or overlapping parameter specifications are identified and processed together with modified query structures that preserve the accuracy requirements of individual queries while enabling efficient bulk processing.
Data Source
AI summary
A system includes determination of a plurality of queries, each of the plurality of queries indicating a first object as a data source, generation of a first query on the first object based on a first two or more of the plurality of queries, modification of each of the first two or more queries to indicate the first query as the data source, requesting of execution of the first query and the modified first two or more queries, and reception of a result set associated with each of the first two or more queries.