Dynamic Aggregate Generation for High Performance Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Legacy approaches for querying large datasets are resource-intensive, time-consuming, and limited in achieving fast and high-quality query results across various data storage environments, failing to provide the desired performance and accuracy.
Innovation Solution
The implementation of dynamic aggregate generation and updating techniques that create aggregate views, tables, and metadata from a virtual multidimensional data model, reducing query response time and improving data quality by accessing smaller aggregate views and continually maintaining them.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional database structures are used to store large volumes of data, then data storage capacity is improved, but query execution time and resource consumption increase
Solution Approach 1:
The patent segments the large dataset into multiple partitions organized in a columnar format within a distributed file system. Each partition contains data for a specific time period or category, allowing queries to access only relevant partitions rather than scanning the entire dataset. This segmentation dramatically reduces query execution time while maintaining large storage capacity.
Solution Approach 2:
The patent introduces a new dimensional organization by creating aggregate tables with pre-computed metrics at multiple levels of granularity. These aggregate tables add a temporal and hierarchical dimension to the data structure, enabling fast queries through dimension-based filtering without scanning all raw data.
2Productivity
If legacy query approaches are used on large datasets, then comprehensive data analysis is achieved, but processing resources and time consumption increase
Solution Approach 1:
The patent performs preliminary actions by pre-computing aggregate metrics and storing them in aggregate tables during data loading or periodic update cycles. When queries are executed, the system leverages these pre-computed aggregates to quickly retrieve results without performing expensive computations at query time, thereby reducing processing resource consumption while maintaining comprehensive analysis capability.
Solution Approach 2:
The patent introduces aggregate tables as intermediary structures between the raw data in the distributed file system and the query processing engine. These intermediaries contain pre-aggregated data that mediates between storage efficiency and query performance, allowing the system to answer complex analytical queries with reduced computational overhead.
3Loss of time
If query caching is used to improve query performance, then response time is reduced for cached queries, but data freshness and query result quality deteriorate
Solution Approach 1:
The patent implements dynamic aggregate tables that are automatically updated when new data is appended to the distributed file system. The system monitors data changes and triggers incremental updates to the aggregate tables, ensuring that cached results remain fresh and accurate. This dynamic update mechanism maintains both fast query response times and data reliability simultaneously.
Solution Approach 2:
The patent incorporates feedback mechanisms where the query engine monitors data change patterns and aggregate table usage. When significant data changes are detected or when aggregate tables become stale, the system automatically triggers regeneration or incremental updates of the aggregate structures, ensuring query results remain accurate and fresh without sacrificing response time performance.
Data Source
AI summary
The present disclosure provides a detailed description of techniques used in systems, methods, and in computer program products for dynamic aggregate generation and updating for high performance querying of large datasets. Certain embodiments are directed to technological solutions for determining at least one aggregate of selected virtual cube attributes (e.g., measures, dimensions, etc.) describing a virtual multidimensional data model of a subject database, and generating an aggregate table and a set of aggregate metadata for the aggregate. In some embodiments, an aggregate database statement configured to operate on the subject database can be issued to generate the aggregate table and/or aggregate metadata. Further, the aggregate can be dynamically determined responsive to receiving a database statement configured to operate on the virtual multidimensional data model representing the subject database. Also, the aggregate table can comprise one or more partitions in an aggregate view to facilitate aggregate management and/or quality.