In-Memory Similarity Search Using Encoded Vectors for Sensitive Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data transfer methods for sensitive information are vulnerable to interception and decryption by sniffers, compromising data security, especially when using encryption and decryption for every memory operation.
Innovation Solution
Implementing a secure in-memory unit on an associative processing unit (APU) with a decryptor, neural proxy hash encoder, and encoded vector data store for secure similarity search, utilizing encoded feature sets to perform similarity searches in parallel, reducing the need for full decryption and enhancing data security.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If encryption and decryption are performed for every memory operation, then data security is improved, but system performance and processing speed deteriorate
Solution Approach 1:
The patent segments the data processing pipeline into distinct stages: encoding stage (where data is transformed into encoded feature sets), storage stage (where encoded vectors are stored in compressed form), and decoding stage (where only necessary data is decrypted). This segmentation allows encryption to be applied selectively rather than to every memory operation, improving performance while maintaining security.
Solution Approach 2:
The patent applies preliminary encoding action by transforming data into encoded feature sets before storage and transmission. The encoded vectors are stored in compressed form in the encoded vector data store, and only when needed are decoded to the original data. This preliminary encoding reduces the frequency and intensity of decryption operations required during memory access.
2Ease of operation
If full decryption is performed for data processing, then data accessibility is improved, but security vulnerability to sniffers increases
Solution Approach 1:
The patent introduces encoded feature sets as an intermediary representation between the original data and the processing system. Instead of storing and processing raw encrypted data, the system stores and processes encoded vectors that can be securely transmitted and stored. The encoded vectors serve as a mediator that maintains security while enabling efficient similarity search operations.
Solution Approach 2:
The patent creates a compressed copy of the data in encoded vector form for storage and processing. The encoded vector data store stores compressed representations of the data, allowing the system to work with copies of the data in an encoded form rather than the original encrypted form, reducing the security risk of full decryption while maintaining processing capability.
3Reliability
If encoded vectors are transmitted and processed, then data security is improved, but bandwidth and memory requirements increase
Solution Approach 1:
The patent changes the parameter representation of data by transforming it into encoded feature sets with reduced dimensionality. The encoded vectors are compressed representations that capture the essential characteristics of the original data while requiring less storage space and bandwidth for transmission. This parameter transformation maintains security through encoding while reducing the quantity of data that must be handled.
Data Source
AI summary
A system includes a secure, in-memory unit implemented on an associative processing unit (APU) for performing a secure similarity search. The unit implements a decryptor, a neural proxy hash encoder, an encoded vector store and a similarity searcher. The decryptor decrypts an encrypted data vector into a data vector. The neural proxy hash encoder encodes the data vector into an encoded search data vector. The encoded vector data store stores a plurality of encoded search candidate vectors and the similarity searcher performs a similarity search between an encoded search query vector and the plurality of encoded search candidate vectors.