Pre-ingested Enrichment Indexing for Search Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search utilities do not include pre-ingested enrichment of data files, leading to latency issues during search operations as enrichment is typically implemented at runtime.
Innovation Solution
Implementing content enrichment of document data and data source connector content through a computing system that ingests and indexes enrichment data using a large language model, allowing for pre-ingestion enrichment, which minimizes latency by making enrichment data searchable and retrievable prior to search queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If enrichment is implemented at runtime during search, then search utility can process data files, but search latency increases
Solution Approach 1:
The system performs enrichment action beforehand by extracting and indexing enrichment data when data files are initially ingested into the data store, rather than performing enrichment at runtime during search queries. This preliminary action eliminates the need for costly enrichment operations during search, thereby reducing search latency while maintaining full search processing capability.
2Loss of time
If enrichment data is extracted and indexed in advance, then search latency is reduced, but data processing complexity increases
Solution Approach 1:
The enrichment process is segmented into distinct components: a trigger mechanism that detects data file changes, an extraction module that retrieves enrichment data, and an indexing module that stores the data in the search index. This segmentation allows each component to be optimized independently and simplifies the overall architecture by separating enrichment logic from search logic.
Solution Approach 2:
The system introduces an intermediary enrichment data structure that bridges the data store and search index. This intermediary layer extracts and transforms enrichment data from raw data files into a search-friendly format, simplifying the interaction between the data storage system and search system while reducing the complexity of direct integration.
3Stability of the object's composition
If enrichment data is stored in schema files, then data structure is maintained, but schema modification complexity increases
Solution Approach 1:
The system extracts enrichment data from the schema file structure and stores it separately in the search index. This extraction separates the enrichment functionality from the data storage schema, allowing enrichment data to be added without modifying the original schema file structure. The schema remains stable while enrichment data is maintained in a dedicated indexing structure.
Data Source
AI summary
Systems and methods are provided for implementing content enrichment of document data and data source connector content. A computing system extracts enrichment data associated with a data item and ingests the extracted enrichment data within a data store, by indexing the extracted enrichment data in a search index of the data store. Indexing the extracted enrichment data includes generating a listing within the search index, the listing being generated to be searchable and refinable using a search engine, the extracted enrichment data being retrievable via the search engine. At least a portion of the enrichment data is generated using a large language model. The enrichment data may be stored in an enrichment data field in a schema file of the data item, in some cases, after the schema file has been modified to add the enrichment data field if not already contained in the schema file.