Semantic Search Workflow for Accurate Regulatory Response Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to efficiently and accurately search large datasets for semantically similar information, particularly in response to regulatory inquiries, due to the loss of institutional knowledge and the limitations of keyword searches, leading to incomplete or inaccurate responses.
Innovation Solution
A computer-implemented system using embedding models like S-BERT to map natural language queries into an embedding space, enabling semantic searches across document sets, with indexing models like FAISS for efficient retrieval, and reranking models to improve accuracy based on user feedback.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If keyword search is used to search company records, then search speed is improved, but search accuracy deteriorates because semantically similar queries are not identified
Solution Approach 1:
The patent transforms the search parameter from exact keyword matching to semantic similarity measurement using embedding vectors. Each query and document is converted to a vector representation in a high-dimensional space, and similarity is measured by the angular distance between vectors. This parameter change enables the system to identify semantically equivalent queries even when using different keywords, resolving the contradiction between speed and accuracy.
Solution Approach 2:
The patent replaces the mechanical keyword-matching system with a machine learning-based semantic embedding system. Instead of comparing individual words, the system uses pre-trained language models (like Sentence-BERT) to generate vector representations that capture the semantic meaning of entire queries. This substitution allows the system to maintain fast search performance while dramatically improving accuracy by understanding the intent behind queries rather than just matching words.
2Adaptability or versatility
If institutional knowledge is stored in individual employees, then adaptability to new situations is improved, but loss of information occurs when employees leave the organization
Solution Approach 1:
The patent creates a digital copy of institutional knowledge by training machine learning models on historical query-response pairs and company documents. The trained models embed organizational expertise into vector representations that can be queried independently of any specific employee. When a new query arrives, the system retrieves relevant information from this captured institutional knowledge, ensuring continuity even when employees leave. This copying principle transforms tacit knowledge held by individuals into explicit, retrievable organizational assets.
3Measurement precision
If more comprehensive search methods are used to improve accuracy, then search precision is improved, but processing time increases
Solution Approach 1:
The patent performs preliminary action by pre-computing embedding vectors for all company documents and organizing them into an efficient retrieval structure (such as FAISS index) before actual search queries arrive. The semantic embedding models are trained in advance on historical data, and the document database is pre-indexed based on their vector representations. When a query comes in, the system only needs to compute the query's embedding and perform fast vector similarity search, avoiding the need for complex real-time analysis. This preliminary preparation enables high-precision semantic search with minimal processing time.
Data Source
AI summary
Exemplary embodiments provide methods, mediums, and systems for performing a reusable, intelligent semantic search across a potentially large number of records. Embodiments may be particularly useful for responding to requests for information from regulatory agencies. In one embodiment, an embedding model is trained to embed queries in an embedding space. When a new query is received, the new query is embedded with the embedding model. A set of documents (e.g., previous responses to regulatory inquiries) may be searched using the embedded query and an indexing model that allows for efficient searches of embedding spaces. A number of results may be returned from the document store, and the results may be ranked by a ranking model. User feedback about the quality of the results may be received, and the ranking model may be retrained based on the feedback.