Analytical View Machine Merging Partial Aggregates in Distributed Databases
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed database systems face challenges in computing and caching complex aggregates, such as average, standard deviation, and variance, which are not easily decomposable or combinable, leading to inefficiencies in query processing.
Innovation Solution
Implementing a system with worker machines that store partial function modules to initialize, update, and serialize aggregates, and an analytical view machine that merges partial results from these workers to produce final results without accessing database partitions, using the initialize-update-serialize-merge infrastructure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If complex aggregates (average, standard deviation, variance) are computed in a distributed database, then query processing capability is improved, but decomposability and ease of combination across nodes deteriorates
Solution Approach 1:
The patent segments complex aggregate computations into partial aggregate functions that can be independently computed on distributed worker nodes. Each worker node maintains partial state for complex aggregates, allowing the computation to be divided across multiple nodes while preserving the ability to combine results centrally at the analytical view machine.
Solution Approach 2:
The patent introduces an intermediary analytical view machine that acts as a mediator between worker nodes and query processors. This intermediary receives partial results from distributed workers, performs the final combination operations for complex aggregates, and returns complete results, thereby resolving the decomposability issue while maintaining query processing capability.
2Speed
If analytical views materialize pre-aggregated data, then query processing speed is improved, but system complexity increases
Solution Approach 1:
The patent implements preliminary action by pre-computing and materializing analytical views with complex aggregates before queries are executed. Worker nodes pre-calculate partial aggregates and store them in analytical views, so that when queries arrive, the system can quickly retrieve and combine pre-computed results rather than performing complex computations in real-time.
Solution Approach 2:
The patent introduces dynamics by making the analytical view system adaptable to different query patterns and aggregate types. The system dynamically determines which analytical views to materialize based on query workloads, and can refresh or update materialized views as underlying data changes, balancing the complexity overhead with actual query performance benefits.
3Productivity
If worker nodes maintain partial aggregate state, then parallel processing efficiency is improved, but memory usage and node complexity increase
Solution Approach 1:
The patent applies local quality by allowing each worker node to maintain only the partial state necessary for its specific data partition and assigned aggregate computations. Each node optimizes its memory usage by storing only relevant partial aggregates for its local data, rather than maintaining global state, thereby improving parallel processing efficiency while controlling memory consumption at each node.
Data Source
AI summary
A system has worker machines connected by a network. Each worker machine stores at least one partition of a distributed database and a partial function module to initialize a database dimension, increment an aggregate of the database dimension in response to a data change and to serialize a last aggregate of the database dimension in response to a refresh operation to form partial results of an analytical view. An analytical view machine connected to the network responds to a query without accessing a partition of the distributed database. The analytical view machine merges partial results from the worker machines to produce a final result to the query.