Vector Embeddings for Private Distributed Document Discovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing security measures for safeguarding sensitive data create barriers to efficient information discovery across organizational silos, preventing employees from accessing relevant documents maintained by other teams or organizations.
Innovation Solution
Utilizing vector embedding models to represent documents as high-dimensional numeric vectors, enabling secure and efficient document discovery by calculating similarity scores without exposing raw data, and allowing access to document IDs for follow-up requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If security measures are implemented to safeguard sensitive data, then data protection is improved, but information discoverability deteriorates
Solution Approach 1:
The patent introduces vector embeddings as an intermediary representation that bridges the gap between security requirements and information discoverability. Instead of directly searching or exposing raw sensitive data, the system converts documents into vector embeddings that preserve semantic meaning while obscuring the actual content. Users can perform similarity searches on these embeddings to discover relevant information without accessing the underlying sensitive data, thus maintaining both security and discoverability.
Solution Approach 2:
The system transforms the parameter representation of data from raw text format to vector embedding format. This parameter change allows the same information to be represented in a form that is both secure (does not reveal actual content) and searchable (preserves semantic relationships). The vector embeddings maintain the essential characteristics needed for similarity matching while removing the sensitive information that would be exposed in traditional search systems.
2Loss of information
If vector embedding models are used to represent documents, then document discoverability is improved, but computational complexity increases
Solution Approach 1:
The system performs the computationally intensive task of creating vector embeddings in advance, during an offline preprocessing stage. By pre-computing and storing the vector embeddings of documents, the system avoids the need to perform complex transformations during actual search operations. This preliminary action shifts the computational burden from the online search phase to the offline indexing phase, making real-time document discovery much faster and more efficient.
3Loss of information
If distributed repositories are searched for relevant documents, then information completeness is improved, but access control complexity increases
Solution Approach 1:
The vector embedding system serves as a mediator that enables cross-organizational document discovery without requiring direct access to the underlying distributed repositories. Users can perform similarity searches across multiple organizations' document collections by querying the vector embeddings, and the system returns results in a controlled manner. This intermediary layer simplifies access control by operating at the embedding level rather than requiring complex permission management at the document level across distributed systems.
Data Source
AI summary
Secure distributed document discovery via vector similarity is performed by a data discovery service. The service receives a request to perform a search using input data. The service sends the input data and an embedding protocol to different data repositories that comprise vector databases. The service receives a results from the different data repositories; each result identifies documents and corresponding document similarity scores. The service generates an overall ranking of the documents according to the document similarity scores. The service returns a query result that identifies the at least a portion of the documents and indicates the ranking of the identified documents.