Encrypted Similarity Search Management for Confidential RAG Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing RAG systems face challenges in performing secure and efficient confidential similarity searches due to the difficulty in searching encrypted storage and embedding data, particularly when a reliable private key holder is required for decrypting search results.
Innovation Solution
A search management device and method that encrypts storage and embedding data using searchable encryption keys, performs random shuffle on encrypted data, and decrypts only the relevant data within the system, ensuring secure and efficient confidential similarity searches without relying on a third-party decryption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If storage data and embedding data are encrypted for security, then security is improved, but search capability deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-processing the encrypted embedding data into an encrypted index structure before search operations. The search management device creates encrypted indexes that enable efficient similarity search without requiring decryption of the actual embedding data, thus maintaining security while enabling search functionality.
Solution Approach 2:
The patent introduces an intermediary mechanism through the use of encrypted indexes and search management devices that mediate between the encrypted storage system and search operations. The search management device acts as an intermediary that can perform similarity searches on encrypted data without exposing the underlying plaintext, resolving the contradiction between security and search capability.
2Reliability
If a third party decrypts search results to ensure reliability, then reliability is improved, but operational efficiency deteriorates
Solution Approach 1:
The patent applies self-service by enabling the search management device to perform decryption operations autonomously using pre-configured encryption keys, without requiring external third-party intervention. The system is designed to automatically manage key distribution and decryption processes, improving operational efficiency while maintaining reliability through cryptographic guarantees.
Solution Approach 2:
The patent segments the decryption function from the search function, allowing different components of the system to handle different aspects of data processing. The search management device holds specific encryption keys that enable it to decrypt search results independently, while other components remain focused on their specialized functions, improving overall system efficiency.
3Reliability
If searchable encryption is used to maintain security during search, then security is improved, but search speed deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing encrypted indexes of embedding data before actual search operations. These pre-prepared encrypted indexes allow the search management device to perform similarity searches by comparing encrypted vectors directly, avoiding the need to decrypt data during search operations and thus maintaining both security and speed.
Solution Approach 2:
The patent changes the parameter space by operating entirely within the encrypted domain, transforming the search problem from one requiring plaintext comparison to one that can be solved through encrypted vector operations. By changing the operational parameters to work with encrypted representations, the system achieves both confidentiality and acceptable search performance.
Data Source
AI summary
A prompt generation server stores encrypted storage data obtained by encrypting storage data with a common key and encrypted storage embedding data obtained by encrypting storage embedding data indicating a feature of the storage data with a searchable encryption key for each piece of the storage data in a search DB. The prompt generation server generates encrypted prompt embedding data obtained by encrypting prompt embedding data indicating a feature of a user prompt with a searchable encryption key, and transmits the encrypted prompt embedding data to a data management server. The prompt generation server receives encrypted related data which is encrypted storage data corresponding to the encrypted storage embedding data similar to encrypted target embedding data, decrypts the encrypted related data with the common key, and acquires related data.