Neural Database Training With Metadata-Augmented Document Chunks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for retrieving information from documents, such as those related to tax laws and regulations, are inefficient and resource-intensive, often requiring significant time and computational resources due to lexical keyword searches and vector comparisons, leading to non-targeted search results.
Innovation Solution
A neural database is trained using structured training data generated by segmenting documents into text chunks and extracting contextual and index metadata, which are then mapped to these chunks to create augmented text chunks, enabling a semantic search guided by a structured index.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional lexical keyword search and vector comparison methods are used to retrieve information from documents, then comprehensive search coverage is achieved, but search time and computational resources increase significantly
Solution Approach 1:
The patent segments documents into smaller text chunks and creates hierarchical indexes (document-level, section-level, and paragraph-level indexes) to enable progressive filtering. This segmentation allows the system to first identify relevant documents using coarse-grained indexes, then progressively narrow down to specific sections and paragraphs using finer-grained indexes, significantly reducing search time while maintaining comprehensive coverage.
Solution Approach 2:
The patent performs preliminary actions by pre-processing documents into structured chunks and generating multiple hierarchical indexes before actual search operations. The training phase pre-computes embeddings and organizes data into document indexes, section indexes, and paragraph indexes, so that during search operations, the system can directly query pre-processed structures rather than analyzing raw documents in real-time.
2Reliability
If traditional lexical keyword search and vector comparison methods are used to retrieve information from documents, then all documents can be searched, but computational resources are consumed excessively
Solution Approach 1:
The patent divides the computational workload by segmenting the search process into multiple stages, each handling a specific level of granularity. Document-level indexes handle coarse filtering with minimal computational resources, section-level indexes handle intermediate filtering, and paragraph-level indexes handle fine-grained matching. This segmentation distributes computational resources efficiently across different search stages.
Solution Approach 2:
The patent applies partial action by not processing all documents at full detail during initial search phases. Instead, it performs partial processing at document and section levels first, then only processes relevant portions at the paragraph level. This avoids the excessive computational resources that would be required to analyze every document in full detail simultaneously.
3Measurement precision
If metadata extraction and augmentation processes are implemented to train the neural database, then search precision and targeting are improved, but data processing complexity increases
Solution Approach 1:
The patent segments metadata extraction into distinct components: document-level metadata (title, author, date), section-level metadata (headers, summaries), and paragraph-level metadata (keywords, entities). This segmentation allows each type of metadata to be extracted and processed independently, managing complexity while improving search precision through multi-level metadata filtering.
Solution Approach 2:
The patent creates a universal training data structure that serves multiple functions: the same augmented text chunks with embedded metadata are used for both training the neural database and for actual search operations. This multi-functional approach reduces overall system complexity by reusing the same processed data structures rather than maintaining separate structures for training and operation.
Data Source
AI summary
Aspects of the disclosure provide a method, including: generating a plurality of text chunks by processing a document, wherein each text chunk includes: a configured portion of the document, and location metadata associated with the document; processing, with a machine learning model, a first subset of the text chunks to extract contextual metadata; processing, with the machine learning model, a second subset of the text chunks to extract index metadata; generating a first structured data file including a mapping between the contextual metadata and the location metadata; generating a second structured data file including a mapping between the index metadata and the location metadata; associating each contextual metadatum and each index metadatum with at least one text chunk based on the first structured data file and the second structured data file to generate a plurality of augmented text chunks; and training a neural database based on the augmented text chunks.