Secure Entity Resolution via Probabilistic Scoring and Segmented Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data integration and visualization systems struggle to identify and reconcile records related to a common entity across large datasets with varying access restrictions, leading to incomplete, duplicate, or conflicting information, especially when records lack identical fields and access is restricted for security or privacy reasons.
Innovation Solution
A secure entity resolution system that automatically analyzes large datasets, identifies records related to a common entity, and securely presents them to users with varying access levels, using machine learning models and user interfaces to facilitate reconciliation and collaboration while maintaining access controls.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If filtering functions are used to identify records with identical fields, then records can be isolated and aggregated, but the system fails when records lack identical fields or have identical fields in only a small number of fields
Solution Approach 1:
The system changes the matching parameters from requiring identical fields to using similarity scoring based on multiple factors including field value similarity, record structure similarity, and contextual relevance. This allows records with partially matching fields to be identified as potential duplicates through probabilistic scoring rather than strict equality checks.
Solution Approach 2:
The system introduces an intermediary similarity scoring mechanism that bridges records with non-identical fields. This intermediary layer computes a similarity score that mediates between disparate record representations, enabling the system to identify related records even when they lack identical fields by translating different data representations into a common similarity metric.
2Reliability
If access restrictions are imposed for security or privacy reasons, then security is improved, but the ability to identify and reconcile related records across multiple sources deteriorates
Solution Approach 1:
The system segments the record matching process into multiple stages: initial identification using accessible fields, similarity scoring using available data, and hierarchical review where restricted fields are progressively accessed only when needed. This segmentation allows security restrictions to be maintained while still enabling record reconciliation through staged information access.
Solution Approach 2:
The system performs preliminary actions by identifying potential duplicate records using only unrestricted fields before attempting to access restricted information. This preliminary identification phase allows the system to flag candidate records for reconciliation without violating security constraints, and only accesses restricted fields when necessary for final verification.
3Measurement precision
If large datasets with billions of records are analyzed, then comprehensive entity resolution is achieved, but the computational time and resources required increase significantly
Solution Approach 1:
The system segments the large dataset into manageable chunks or batches, processing records in distributed parallel streams. This segmentation allows the computational task to be divided across multiple processing units, reducing the time required to analyze billions of records while maintaining comprehensive entity resolution through coordinated processing of all segments.
Solution Approach 2:
The system implements a two-phase approach where it first performs partial action by identifying high-probability duplicate candidates using a subset of fields and quick similarity checks, then applies excessive action by performing more thorough analysis only on the identified candidates rather than all record pairs. This dramatically reduces processing time while maintaining resolution completeness.
Data Source
AI summary
Computer implemented systems and methods are disclosed for automatically and canonically identifying related data in various data structures while maintaining secure access to individual data objects and their properties. Data structures may include a plurality of records, wherein each record is associated with a respective entity. Access to individual records, or in some embodiments access to individual attributes of a record, may be restricted to particular users or groups on a per-item basis. In accordance with some embodiments, systems and methods are disclosed for identifying pairs of records, determining a probability that both records relate to a common entity, and securely notifying and presenting users with information regarding the probability while preserving the access restrictions for the individual records and attributes. Record pairs that potentially refer to the same entity may be linked, reconciled, or ignored in response to feedback from users who have access to one or both records.