Hierarchical AI Platform for Vector Embedding Precision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI-based querying systems struggle to efficiently process and query a relatively small and specific set of data objects, such as medical records or engineering documents, due to limitations in data processing and retrieval techniques.
Innovation Solution
An AI platform that utilizes hierarchical parsing, metadata processing, and vectorization to process and query a specified set of data objects. This platform includes a system with a processor configured to hierarchically parse files into connected data chunks, generate metadata, process chunks and metadata with an embedding model to generate vector embeddings, and store these embeddings along with textual summaries for querying.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional flat parsing is used to process files, then the processing is simpler, but the retrieval precision and contextual understanding deteriorate
Solution Approach 1:
The patent segments files into hierarchically connected data chunks that preserve the original document structure (sections, subsections, paragraphs). This segmentation maintains contextual relationships while enabling precise retrieval by allowing queries to target specific hierarchical levels without processing the entire file flatly.
Solution Approach 2:
The patent introduces a hierarchical dimension to the data structure, transforming flat file parsing into multi-level hierarchical parsing. This adds organizational depth (parent-child relationships, hierarchical levels) that improves querying precision by enabling context-aware retrieval while managing complexity through structured organization.
2Measurement precision
If hierarchical parsing is implemented to improve contextual understanding, then the querying precision improves, but the processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary hierarchical parsing and metadata generation during the indexing phase, organizing data into hierarchically connected chunks before queries are submitted. This preliminary action stores the hierarchical structure and relationships in advance, so that during querying, the system can quickly retrieve relevant chunks without performing complex hierarchical analysis in real-time, thus reducing query processing time.
3Productivity
If metadata filtering is applied to reduce search space, then the retrieval efficiency improves, but the system complexity increases
Solution Approach 1:
The patent segments metadata into hierarchical levels and types, allowing filtering operations to work on specific segments rather than the entire metadata set. This segmentation enables efficient retrieval by letting users filter at different hierarchical levels (e.g., by section type, by hierarchy level) without requiring complex full-system metadata processing.
Data Source
AI summary
An AI based system and method for processing and querying files. A method of processing files for an artificial intelligence (AI) querying service includes: hierarchically parsing the files into a set of hierarchically connected data chunks; generating metadata for each of the hierarchically connected data chunks, wherein the metadata includes hierarchical information; processing the hierarchically connected data chunks and metadata with an embedding model to generate vector embeddings that include the hierarchical information; generating textual summaries from the hierarchically connected data chunks; and storing the vector embeddings, textual summaries, and hierarchically connected data chunks for the AI querying service.