Data Conflation Platform with Compliance-Aware Join Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managing large volumes of user data across multiple organizations is complex due to compliance, privacy, and access control requirements, especially when datasets are shared and combined, leading to security and isolation challenges.
Innovation Solution
A platform that conflates data from multiple sources by matching entities based on similarity, applying compliance and access control policies through join queries, and enforcing policies on a per-record, per-field basis, using encryption and logical/physical isolation to ensure secure and compliant data access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data from multiple organizations is combined and shared, then data utility and analytical value are improved, but security risks and compliance complexity increase
Solution Approach 1:
The patent introduces a data conflation platform as an intermediary between multiple organizations. This platform conflates data from different sources by matching entities across datasets while enforcing compliance policies and access controls, enabling secure data sharing without direct organization-to-organization access. The intermediary architecture allows data utility improvement while maintaining security boundaries.
2Reliability
If compliance policies are enforced on all data access operations, then privacy and regulatory compliance are improved, but system overhead and processing complexity increase
Solution Approach 1:
The patent implements compliance policies and access control rules in advance during data conflation operations. By pre-configuring compliance requirements and enforcing them at the point of data joining and merging, the system avoids the need for continuous compliance checking on subsequent access operations, reducing overall system overhead while maintaining compliance assurance.
3Reliability
If fine-grained access control is applied to individual fields and records, then data privacy protection is improved, but query processing time and computational overhead increase
Solution Approach 1:
The patent applies fine-grained access control selectively to specific fields and records based on their sensitivity and compliance requirements. Not all data fields undergo the same level of scrutiny - the system identifies which fields require enhanced privacy protection and applies corresponding access control measures only to those specific locations, rather than uniformly across the entire dataset, thus balancing privacy protection with query performance.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
The disclosed embodiments provide a system for managing data conflation. During operation, the system generates matches between a first set of entities in a first dataset from a first data provider and a second set of entities in a second dataset from a second data provider based on comparisons of fields in the first and second datasets. Next, the system modifies a join query for joining the first and second datasets to include operators representing compliance rules for the first or second datasets. The system executes the modified join query to produce a joined dataset that adheres to the compliance rules and stores data related to the joined dataset within a platform that logically isolates the data from additional datasets. During processing of queries of the data, the system modifies the queries to include additional operators that enforce access control policies for the data.