Unstructured Document Structuring via Vector Space Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for generating structured documents from unstructured ones are inefficient, as they often provide irrelevant results due to overlapping data points in tree structures and pruning, leading to decreased accuracy in finding nearest data elements during keyword searches.
Innovation Solution
A processor-implemented method that identifies breakpoints in unstructured documents using split points like newline symbols or punctuation marks, segments the document into fragments, classifies them as title or non-title fragments, constructs a data tree, and assigns vectors to each node based on importance, allowing for a queryable structured document format using a machine learning model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If keyword search is used to access information in large documents, then search speed is improved, but search accuracy deteriorates due to irrelevant results
Solution Approach 1:
The patent transforms the search approach by changing from keyword matching to vector space similarity search. Documents and queries are converted to vector representations, and similarity is measured using cosine similarity or other distance metrics in the vector space, fundamentally changing the search parameters from discrete keyword matching to continuous similarity measurement
Solution Approach 2:
The patent replaces the mechanical keyword-matching system with a machine learning-based vector representation system. Instead of mechanically comparing keywords, the system uses trained models to generate vector embeddings that capture semantic meaning, allowing for more accurate similarity-based search
2Productivity
If metric space search with tree data structure is used, then search efficiency is improved, but search accuracy deteriorates due to overlapping data points and pruning
Solution Approach 1:
The patent extracts the problematic tree data structure and pruning mechanism from the search system. By removing the tree structure entirely and using flat vector space search with similarity metrics, it eliminates the source of accuracy degradation while maintaining efficiency through optimized vector search algorithms
Solution Approach 2:
The patent changes the search parameter space from hierarchical tree structures with pruning thresholds to continuous vector space with similarity distances. This allows for more precise control over search results through adjustable similarity thresholds without the structural constraints of tree-based approaches
3Loss of time
If tree data structure with pruning is used to locate nearest data elements, then processing time is reduced, but accuracy in finding nearest data elements deteriorates
Solution Approach 1:
The patent replaces the mechanical tree traversal and pruning process with a direct vector similarity computation approach. By using pre-computed vector representations and similarity metrics, the system directly identifies nearest neighbors without the intermediate steps of tree traversal, eliminating accuracy loss from pruning while maintaining efficiency
Data Source
AI summary
A system for generating a queryable structured document from an unstructured document using a machine learning model is provided. The system (i) identifies breakpoints in the unstructured document, (ii) segments the unstructured document into one or more fragments based on identified breakpoints, (iii) classifies the one or more fragments as one or more title fragments or one or more non-title fragments based on a sequence of a position of words used in each fragment of the one or more fragments, (iv) constructs a data tree using the one or more title fragments and the one or more non-title fragments as a node of the data tree; (v) assigns one or more vectors to each node of the data tree, and (vi) generates a structured document by providing matrix representation for each node of the data tree.