Encrypted Match Index for Duplicate Detection in Cloud Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud computing platforms face challenges in detecting duplicates in encrypted data fields, as conventional duplicate detection methods fail when dealing with fields encrypted using deterministic or probabilistic encryption schemes, which result in varying and unpredictable ciphertexts.
Innovation Solution
The implementation of encrypted match indexes allows for duplicate detection in cloud computing platforms by using a match index that is encrypted using a unique identifier as the key or initialization vector, enabling comparison and identification of duplicates within encrypted data fields.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data fields are encrypted using deterministic or probabilistic encryption schemes to ensure data security, then data security is improved, but duplicate detection capability deteriorates because encrypted ciphertexts vary and become unpredictable
Solution Approach 1:
The match index is segmented into multiple components including a hash of the encrypted field value and metadata about the encryption scheme. This segmentation allows the system to compare encrypted values without decrypting them, maintaining security while enabling duplicate detection through the hash component which remains consistent for identical plaintext values.
Solution Approach 2:
A match index structure is introduced as an intermediary between the encrypted data fields and the duplicate detection mechanism. The match index contains hashed representations of encrypted values and encryption scheme metadata, serving as a mediator that enables comparison without exposing the actual encrypted data or requiring decryption operations.
2Ease of manufacture
If conventional duplicate detection methods are used on encrypted data, then processing simplicity is maintained, but detection accuracy deteriorates because encrypted ciphertexts cannot be directly compared
Solution Approach 1:
The system transforms the encrypted field values into hash representations and extracts encryption scheme metadata as new parameters for comparison. By changing from direct ciphertext comparison to hash-based comparison with scheme verification, the system maintains processing simplicity while achieving accurate duplicate detection even with varying encrypted representations.
3Difficulty of detecting and measuring
If encrypted match indexes are implemented to enable duplicate detection, then duplicate detection capability is improved, but system complexity increases due to additional encryption and index management requirements
Solution Approach 1:
The match index is pre-computed and stored alongside the encrypted data fields during data ingestion. By performing the encryption-based index creation in advance rather than during query operations, the system reduces real-time computational complexity while maintaining enhanced duplicate detection capability during search operations.
4Reliability
If data is stored in encrypted form to maintain security, then data security is improved, but data usability deteriorates because encrypted data cannot be directly analyzed or processed
Solution Approach 1:
Instead of decrypting the actual data for analysis, the system creates and operates on copies in the form of match indexes containing hash representations and metadata. These index copies enable duplicate detection and data quality operations on encrypted data without requiring access to the plaintext, thus maintaining security while improving usability.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed herein are system, method, and computer program product embodiments for detecting duplicates with exact and fuzzy matching on encrypted match indexes using an encryption key in a cloud computing platform. An embodiment operates by determining a match rule index value upon reception of a new record. The embodiment encrypts the match index rule value using the customer's encryption key and a deterministic encryption method and stores the encrypted match rule index value. Duplicate detection may be later performed by using the same deterministic encryption method to determine a cypher text for a candidate entry and comparing the ciphertext to the stored encrypted match indexes.