Federated Data Visualization Query Compilation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Integrating data from multiple sources into a single visualization is complex, requiring pre-built static databases or federated queries, which can be cumbersome and inefficient, especially when dealing with large datasets and diverse data sources.
Innovation Solution
The solution dynamically derives integration points and links between data sets based on the required data for visualization, using automatic schema matching and manual interfaces, and leverages fast databases by compiling visual specifications into SQL or MDX queries, federating them to reduce data transfer and enable efficient processing of large datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is pre-loaded and integrated into a middle tier database, then query performance is improved, but system complexity and data collocation requirements increase
Solution Approach 1:
The system performs preliminary actions by pre-compiling visual specifications into executable query plans and pre-establishing data relationships. The query compilation step transforms visual specifications into optimized SQL or MDX queries before execution, and the system pre-loads only the specific data subsets required for each visualization task rather than entire datasets, thereby improving query performance while avoiding full data collocation.
Solution Approach 2:
The system segments data processing by dividing large datasets into smaller, visualization-specific subsets. Instead of loading and integrating entire databases into the middle tier, the system compiles queries to retrieve only the necessary data portions for each specific visual task, reducing data transfer and memory requirements while maintaining query performance.
2Ease of operation
If data from multiple sources is integrated into a single database, then data access simplicity is improved, but data transfer overhead and processing time increase
Solution Approach 1:
The system introduces a query compilation intermediary layer that translates visual specifications into optimized federated queries. This intermediary compiles the visual specification into SQL or MDX queries that can directly access multiple data sources without physically integrating the data, thereby maintaining data access simplicity while eliminating the need to transfer and collocate entire datasets into a single database.
Solution Approach 2:
The system extracts only the necessary data subsets from multiple sources based on compiled query requirements. Instead of integrating all data from multiple sources into a single database, the compilation step identifies and retrieves only the specific data needed for each visualization, reducing data transfer overhead and processing time while maintaining ease of data access through the unified interface.
3Productivity
If visual specifications are compiled into SQL or MDX queries, then data processing efficiency is improved, but query complexity increases
Solution Approach 1:
The system performs preliminary compilation of visual specifications into optimized SQL or MDX queries before data retrieval. This compilation step transforms high-level visual specifications into efficient executable queries that leverage the capabilities of fast databases, improving data processing efficiency by preparing optimized query plans in advance rather than generating ad-hoc queries during execution.
Solution Approach 2:
The compilation system automatically generates optimized queries from visual specifications without requiring manual intervention. The system self-services by translating visual specifications into appropriate SQL or MDX queries based on the target data source type, handling query optimization and transformation automatically, thereby improving processing efficiency while managing query complexity through automation rather than manual complexity.
Data Source
AI summary
A user selects a first group of one or more dimension fields that specifies a primary aggregation for a data visualization, where the one or more dimension fields are in a first data source. When one or more first dimension fields in the first group are not dimension fields in a second data source, a client device joins the first data source with the second data source at a first level of detail that is more granular than the primary aggregation to form a single combined data set that includes the one or more dimension fields specified in the first group and one or more measure data fields aggregated according to the first group. The client device rolls up the combined data set, including the one or more measure data fields, to form a final data set and displays the data visualization using the data from the final data.