Federated Query Schema Rewriting for Secure Cross-Dataset Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques are inadequate for managing integrated access to both public and privately-accessible datasets, requiring expert knowledge of programming languages and complex database systems, and lack intrinsic functionality to locate, manage, and execute queries across disparate data resources.
Innovation Solution
A platform utilizing federated query generation and schema rewriting optimization, which includes an inference engine to parse and convert queries, rewrite them into a standardized format (RDF triples), and optimize them for access to multiple data sources, including public and private databases, while ensuring access control.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is siloed within individual applications or systems, then data security and access control are maintained, but data accessibility and sharing capability deteriorate
Solution Approach 1:
The system segments data access into two distinct layers: the underlying data sources remain segmented and protected within their respective systems, while a virtualized layer above enables unified access. The federated query engine operates at this virtualization layer, allowing data to be accessed across systems without physical consolidation, thus maintaining security boundaries while enabling sharing.
Solution Approach 2:
The patent introduces a platform as an intermediary between data sources and data consumers. This platform includes a federated query engine and schema rewriting engine that mediate access requests, translating queries into system-specific formats and managing authentication/authorization without requiring data to leave its secure environment. The intermediary enables cross-system access while preserving data security boundaries.
2Productivity
If data is consolidated into a centralized data warehouse, then data accessibility and analysis capability are improved, but system complexity and infrastructure requirements worsen
Solution Approach 1:
Instead of physically consolidating data into a centralized warehouse, the system creates virtual copies of data schemas and metadata at the platform layer. The schema rewriting engine generates virtual schema representations that enable unified querying without duplicating actual data. This copying approach provides centralized access capabilities while avoiding the storage and infrastructure complexity of a physical data warehouse.
Solution Approach 2:
The platform acts as an intermediary that provides centralized data analysis capabilities without requiring centralized data storage. The federated query engine and schema rewriting engine work together to translate analytical queries into system-specific operations, enabling complex data analysis across distributed sources while keeping the infrastructure distributed and manageable.
3Adaptability or versatility
If federated queries are executed across multiple data sources, then comprehensive data access is achieved, but query execution time and processing overhead increase
Solution Approach 1:
The system performs preliminary actions by pre-defining and caching schema mappings, data type relationships, and query templates before actual data requests occur. The schema rewriting engine pre-processes and validates query structures against known data source schemas, preparing execution plans in advance. This preliminary preparation reduces the processing time required during actual query execution across distributed sources.
Solution Approach 2:
The patent employs parameter changes by dynamically adjusting query execution parameters based on data source characteristics, query complexity, and system load. The federated query engine optimizes execution by transforming queries into different formats or parameter sets suitable for specific data sources, and by adjusting batch sizes, timeout values, and parallelism levels to minimize overall execution time while maintaining comprehensive data access.
4Adaptability or versatility
If data schemas vary across different sources, then data source flexibility is maintained, but query compatibility and integration difficulty worsen
Solution Approach 1:
The schema rewriting engine creates virtual schema copies that represent different data source schemas in a unified format. Instead of requiring physical schema standardization across sources, the system copies and transforms schema definitions at the virtualization layer, enabling queries to be written against a consistent logical schema while underlying sources maintain their native schema structures and flexibility.
Solution Approach 2:
The platform serves as an intermediary that manages schema variations between diverse data sources. The schema rewriting engine acts as a mediator that automatically translates between different schema representations, handling data type mappings, null value representations, and structural differences. This intermediary layer maintains data source flexibility while reducing integration complexity by abstracting away schema differences from query writers.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
Various techniques are described for platform management of integrated access of public and privately-accessible datasets utilizing federated query generation and query schema rewriting optimization, including receiving at a dataset access platform a query formatted according to a first data schema, generating a copy of the query, saving the query and the copy to a datastore, parsing the copy of the query in the first schema using an inference engine, determining whether the query comprises data associated with an access control condition associated with accessing the dataset, the access control condition being configured to indicate whether the query is permitted to access the dataset, and rewriting, using a proxy server, the copy of the query in a second schema by converting the copy of the query into a triple associated with the query and another triple associated with the access control condition.