Viz Data Service Query Abstraction for Federated Analytics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data analytics systems face challenges in federating data queries across different databases while managing security, and there is no unified endpoint for data consumers to query for semantically correct and analytically useful insights.
Innovation Solution
The implementation of a Viz Data Service (VDS) that introduces a higher-level query interface on top of published data sources, allowing clients to express intent through their queries and enabling the compilation of SQL-like queries that satisfy this intent, along with a smart switching process for selecting data servers or services.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a SQL-like query interface is provided on top of published data sources, then clients can query data from multiple databases, but clients must incorporate all desired semantics into the queries they send to the data server, increasing query complexity
Solution Approach 1:
The VizQL Data Service acts as an intermediary layer between clients and published data sources. It receives high-level VizQL queries from clients, translates them into appropriate SQL-like queries for the underlying data sources, and handles the complexity of data federation and security policies. This mediator approach allows clients to use simple VizQL without needing to understand the complex SQL-like query structure required by the data sources.
Solution Approach 2:
The system segments the query processing into distinct layers: the VizQL layer for high-level query expression, the translation layer for converting VizQL to SQL-like queries, and the execution layer for actually querying the data sources. This segmentation allows each layer to be optimized independently and simplifies the interface for clients while maintaining versatility across different data sources.
2Adaptability or versatility
If clients send SQL-like queries to the data server, then data can be queried across multiple databases, but clients must know how to express complex semantics like FIXED LoD calculations, increasing difficulty of operation
Solution Approach 1:
The VizQL Data Service serves as an intermediary that translates high-level VizQL expressions (including complex semantics like FIXED LoD calculations) into SQL-like queries. Clients only need to express their desired semantics in VizQL, which is simpler and more intuitive, while the service handles the complex translation to SQL-like syntax automatically.
Solution Approach 2:
The system changes the parameter space for query expression by introducing VizQL as a higher-level query language with different semantic primitives. Instead of requiring clients to use SQL-like syntax with its specific syntax rules, clients can use VizQL's simplified syntax, and the system translates between the two parameter spaces.
3Quantity of substance
If data is scattered across many silos in different databases, then data diversity and volume increase, but there is no easy way of federating data queries while managing security, increasing system complexity
Solution Approach 1:
The VizQL Data Service acts as a central intermediary that manages data federation across multiple databases. It receives queries, determines the appropriate data sources, translates queries for each source, applies security policies (such as row-level security), and aggregates results. This mediator approach consolidates the complexity of data federation into a single service layer.
Solution Approach 2:
The VizQL Data Service provides universal access to multiple data sources through a single interface. It can query different types of databases, apply various security policies, and handle diverse query types all through the same VizQL interface, making the system multi-functional and simplifying client interactions.
4Adaptability or versatility
If there is no unified endpoint for data consumers, then data consumers can connect to multiple data sources, but they cannot reliably query for semantically correct and analytically useful insights, reducing reliability
Solution Approach 1:
The VizQL Data Service provides a unified intermediary endpoint that ensures reliable and semantically correct query results. It translates client queries into appropriate SQL-like queries for the underlying data sources, handles data type conversions, applies security policies, and aggregates results consistently. This mediator ensures that regardless of which data source is queried, the results are semantically correct and analytically useful.
Data Source
AI summary
A server system is communicatively connected to a plurality of computing devices and one or more databases. The server system receives one or more queries from a computing device. The one or more queries specify a data source. The server system determines a level of security applicable to a user of the computing device. The server system translates the one or more queries into one or more logical queries according to semantics of the data source, and transmits the logical queries to a query pipeline of the server system. The server system executes the one or more queries against a first database of the one or more databases to retrieve query results from the data source. The server system applies the determined level of security to the query results to obtain one or more data sets, and returns the one or more data sets to the computing device.