Document Graph Retrieval With Structural Context for Complex Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional document retrieval systems fail to utilize the structural context of documents, leading to inadequate search results in structurally complex documents such as source code, programming documentation, and API specifications, as they either rely solely on keyword matching (lexical search) or disregard document structure in semantic searches.
Innovation Solution
An integrated search method that combines lexical and semantic relevance, reconstructing the structural context by modeling documents as document graphs, tracing hierarchical structures, and utilizing large language models (LLMs) to generate comprehensive and coherent search results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If lexical search is used to match exact terms, then search speed is improved, but search accuracy deteriorates due to inability to understand context and homographs
Solution Approach 1:
The patent combines lexical search and semantic search into an integrated search system that processes both exact term matching and contextual understanding simultaneously, resolving the contradiction between speed and accuracy by leveraging the strengths of both approaches
Solution Approach 2:
The search process is segmented into multiple stages: lexical search for quick term matching, semantic search for contextual understanding, and relevance scoring for result ranking. This segmentation allows the system to maintain speed while improving accuracy through progressive refinement
2Measurement precision
If semantic search is used to understand user intent and context, then search accuracy is improved, but system complexity and processing time increase due to NLP requirements
Solution Approach 1:
The system applies semantic search selectively based on query characteristics, using lexical search for simple queries and escalating to full semantic analysis only when needed, thereby reducing overall system complexity while maintaining accuracy for complex queries
Solution Approach 2:
The patent introduces an intermediary relevance scoring mechanism that bridges lexical and semantic search results, filtering and ranking results based on multiple factors including term frequency, semantic relevance, and structural context, thereby managing complexity through layered processing
3Loss of information
If semantic search with NLP models is deployed, then contextual understanding is improved, but processing speed decreases compared to lexical searching
Solution Approach 1:
The system performs preliminary lexical search to identify candidate documents and terms before applying more computationally intensive semantic analysis, thereby reducing the volume of data requiring NLP processing and maintaining overall speed while improving contextual understanding
4Ease of operation
If document structure is ignored in semantic search, then processing simplicity is maintained, but search relevance deteriorates in structurally complex documents
Solution Approach 1:
The patent applies different processing strategies to different parts of the document structure: simple lexical matching for content text, and structure-aware semantic analysis for hierarchical elements like headings, sections, and code blocks, thereby maintaining simplicity where possible while improving relevance where needed
Data Source
AI summary
Certain aspects of the present disclosure describe a method of information retrieval. In certain aspects, the method includes identifying a set of relevant nodes of a document graph embedding semantic units associated with the document based on a document search query. The method further includes reconstructing a structural context for each relevant node in the set of relevant nodes. The method further includes processing the set of relevant nodes and the structural context of each relevant node with a large language model to generate a contextual response to the document search query.