Federated Query Engine for Privacy-Preserving Fuzzy Table Joins
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Federated query systems lack a means to control privacy independently across multiple datasets, which may have different privacy requirements due to organizational, data nature, or jurisdictional constraints.
Innovation Solution
A federated query engine system that incorporates privacy-preserving features, such as cryptographic long-term keys and bloom-filter-based cryptographic approaches, to enable fuzzy table joins and leverage storage system infrastructure for secure data sharing, while generating an optimal privacy-compliant execution plan.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If federated query systems extract and join data from multiple remote sources, then query capability and data accessibility are improved, but privacy control and data security deteriorate
Solution Approach 1:
The patent introduces privacy-preserving table join techniques as an intermediary mechanism between federated query systems and sensitive data. This allows queries to execute across distributed data sources while the join operation itself remains protected through cryptographic methods, preventing direct exposure of plaintext sensitive data during the federation process.
Solution Approach 2:
The patent replaces traditional plaintext-based table join operations with cryptographic methods. Instead of mechanically comparing plaintext values across distributed systems, the system uses encrypted representations and cryptographic protocols to perform joins, substituting the mechanical data comparison process with a secure computational approach that preserves privacy.
2Measurement precision
If plaintext column values are compared for table joins, then join accuracy and query correctness are improved, but data exposure and privacy leakage increase
Solution Approach 1:
The patent changes the parameter state of data from plaintext to encrypted form during table join operations. By transforming data into cryptographic representations, the system maintains the ability to perform accurate joins (through cryptographic comparison protocols) while fundamentally altering the data state to prevent privacy leakage that would occur with plaintext comparison.
Solution Approach 2:
The patent substitutes traditional plaintext comparison mechanics with cryptographic comparison protocols. Instead of directly comparing plaintext column values to achieve join accuracy, the system uses cryptographic methods that provide equivalent join functionality while preventing the privacy leakage inherent in plaintext exposure.
3Ease of operation
If data is moved across networks for federated queries, then data accessibility and query flexibility are improved, but network transmission risks and security vulnerabilities increase
Solution Approach 1:
The patent introduces cryptographic protocols as an intermediary layer between data movement operations and network transmission. This allows federated queries to access distributed data sources while the cryptographic layer protects the data during transmission, mitigating network security risks without sacrificing data accessibility.
4Reliability
If different privacy requirements are enforced across multiple datasets, then compliance with jurisdictional and organizational constraints is improved, but system complexity and query planning difficulty increase
Solution Approach 1:
The patent applies local quality by allowing different privacy-preserving techniques to be applied to different datasets based on their specific privacy requirements. Each dataset can have its own privacy constraints and corresponding cryptographic methods applied locally, enabling compliance with diverse jurisdictional and organizational constraints without requiring a uniform complex system-wide approach.
Data Source
AI summary
A federated query engine system and method for multiple datasets is enhanced with privacy preserving features. It may, for example, limit the movement of data from one or more of the datasets being accessed. It may use cryptographic long-term keys, enabling fuzzy table joins that do not require a comparison of the plaintext column values. The query plan may leverage the particular infrastructure of the storage system that houses each of the datasets. The query engine receives a standard SQL query, translates the query into a logical plan for performing the query across the multiple datasets, converts the logical plan into physical plans that are specific to the implementational architecture of the multiple datasets, and sends these physical plans to SQL workers located near the data warehouses housing each dataset.