Encrypted Match Indexes for Secure Bulk Duplicate Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems face challenges in detecting and eliminating duplicate records, especially when these records contain encrypted fields, as encryption complicates the matching process and existing methods often require decrypting data, which compromises security.
Innovation Solution
The system generates encrypted match indexes for each record based on defined match rules, allowing for bulk operations to identify duplicates without exposing plaintext data, using deterministic encryption and intercepting database access operations to create and manage these indexes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data fields are encrypted with customer's encryption key for data security, then data security is improved, but duplicate detection capability deteriorates
Solution Approach 1:
The patent introduces encrypted match indexes as an intermediary mechanism between encrypted data fields and duplicate detection operations. These indexes store deterministic encrypted values of the actual data fields, allowing the duplicate detection system to compare encrypted values without accessing or decrypting the underlying sensitive data. This mediator layer enables duplicate detection functionality while preserving data encryption and security.
Solution Approach 2:
The system performs preliminary action by pre-computing and storing encrypted match indexes for encrypted data fields before duplicate detection is needed. These indexes are generated using deterministic encryption, ensuring that identical plaintext values always produce identical encrypted index values. This preliminary preparation enables efficient duplicate detection without requiring real-time decryption or computation during the detection process.
2Difficulty of detecting and measuring
If existing duplicate detection methods are used on encrypted data, then duplicate detection is achieved, but data security is compromised due to decryption requirements
Solution Approach 1:
The encrypted match indexes serve as an intermediary that enables duplicate detection on encrypted data without requiring decryption. The detection system operates entirely on the encrypted index values, comparing them to identify duplicates, while the original encrypted data fields remain protected and never exposed in plaintext form during the detection process.
Solution Approach 2:
The system creates encrypted copies of the data fields in the form of match indexes. These indexes are deterministic encrypted representations that can be used for comparison and duplicate detection. The original encrypted data remains unchanged and secure, while the index copies enable detection operations without compromising the security of the source data.
3Productivity
If bulk operations are implemented for duplicate detection, then processing efficiency is improved, but complexity of handling encrypted data increases
Solution Approach 1:
The system performs preliminary action by pre-computing encrypted match indexes for all relevant encrypted data fields before bulk duplicate detection operations. This preparation step transforms the complex task of comparing encrypted data into a simpler comparison of pre-computed index values, enabling efficient bulk processing without requiring complex encrypted data manipulation during the actual detection phase.
Solution Approach 2:
The patent replaces the mechanical process of directly comparing encrypted data fields with a substitution approach using encrypted match indexes. Instead of performing complex encrypted data operations during bulk processing, the system substitutes these with simpler comparisons of deterministic encrypted index values, thereby improving processing efficiency while reducing the complexity of handling encrypted data during operations.
Data Source
AI summary
Provided herein are system, methods and computer program products for identifying duplicate records stored in a database system, comprising: generating a plurality of encrypted match indexes for each of a plurality of records stored in the database system, each of the plurality of encrypted match indexes encrypts a value of each encryption enabled field of a respective one of the plurality of records defined by at least one match rule, creating a cluster of records comprising at least one set containing at least two records of the plurality of records, the at least two records having respective encrypted match indexes corresponding to the at least one match rule, causing identification of duplicate records in the at least one set according to detection of records of the at least one set having respective match indexes matching the at least one match rule, and outputting an indication of the identified duplicate records.