Encrypted Data Substring Token Indexing for Search Sargability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for searching encrypted data records require accessing and decrypting every record, making them inefficient compared to searching unencrypted records, as they cannot utilize sorted indexes for filtering without decryption.
Innovation Solution
Generating and storing substring tokens of different lengths for encrypted data values, allowing for efficient searching by encrypting the search string and matching it against encrypted tokens in a separate table using binary search algorithms or trees, thereby avoiding the need to decrypt all records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional search methods are used on encrypted data records, then data security is maintained, but search performance deteriorates because every record must be accessed and decrypted
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing encrypted substring tokens for all possible substrings of encrypted data values before search operations. This allows the search system to efficiently query encrypted data without decrypting entire records, as the tokens are prepared in advance and can be directly compared with encrypted search strings.
Solution Approach 2:
The patent creates copies of the encrypted data in the form of substring tokens stored in a separate search index table. These token copies enable efficient searching operations on encrypted data without requiring access to or decryption of the original encrypted records, thus maintaining security while improving search performance.
2Measurement precision
If all encrypted records are decrypted for searching, then complete search accuracy is achieved, but CPU time and I/O operations increase significantly
Solution Approach 1:
The patent extracts only the necessary portion of encrypted data for searching by computing substring tokens from encrypted values. Instead of decrypting entire records, the system extracts and stores encrypted substrings in a dedicated index table, allowing accurate pattern matching to be performed on these extracted tokens without processing the full encrypted records.
Solution Approach 2:
The patent segments encrypted data values into multiple substrings of different lengths and stores each as a separate encrypted token. This segmentation allows the search system to efficiently match patterns by comparing encrypted search strings against the segmented tokens, achieving accurate search results without decrypting complete records.
3Productivity
If a sorted index is used for unencrypted data, then search efficiency is improved, but this approach cannot be directly applied to encrypted data without decryption
Solution Approach 1:
The patent changes the parameter of the index structure by storing encrypted substring tokens instead of unencrypted values. This allows the index to maintain sorted order on encrypted data using deterministic encryption, enabling efficient binary search operations on encrypted values without requiring decryption, thus preserving search efficiency while working with encrypted data.
Data Source
AI summary
A method for increasing sargability of encrypted records to allow for searching of a first column of a first data table for encrypted values containing a search string without having to decrypt all encrypted values involves, for each respective data record in the first data table, accessing an encrypted data value from the first column, decrypting the accessed encrypted data value, generating from the decrypted data value a respective plurality of substrings of various lengths, encrypting each substring of the respective plurality of substrings as an encrypted substring token, and storing each encrypted substring token in association with a reference value for lookup of a corresponding record in the first data table. Subsequently, the first column of the first data table can be searched for encrypted values containing a first search string by encrypting the first search string and searching for encrypted substring tokens matching the encrypted first search string.