Search Index Update via Field Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search engine indexing systems require reindexing entire documents for updates, leading to inefficiencies in memory usage and processing power, especially when only small changes are made to frequently updated fields.
Innovation Solution
Implementing a method that separates frequently updated fields into an external database, allowing direct updates without reindexing the entire document, while less-frequently updated fields are stored in the internal index file, using a custom codec to manage data between the internal and external storage systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire documents are reindexed for updates, then data consistency is maintained, but memory usage and processing power increase significantly
Solution Approach 1:
The patent divides the index into multiple segments (index segments) that can be independently managed. When a document is updated, only the relevant segments are modified rather than reindexing the entire document collection. This segmentation allows partial updates to maintain data consistency while reducing the overall processing burden.
Solution Approach 2:
The patent extracts frequently updated fields into a separate updateable structure within the index segments, while less frequently updated fields remain in the main index. This extraction allows targeted updates of only the necessary portions of the index, reducing the processing power required while maintaining data consistency for updated fields.
2Reliability
If entire documents are reindexed for updates, then data consistency is maintained, but memory usage increases
Solution Approach 1:
By segmenting the index into multiple manageable parts, the patent reduces the amount of memory that needs to be allocated for any single update operation. Only the memory required for the affected segments needs to be reserved, rather than memory for the entire index, thus reducing overall memory usage while maintaining consistency.
Solution Approach 2:
The patent applies partial action by updating only the necessary portions of the index (specific segments and fields) rather than performing a complete reindexing operation. This partial update approach reduces memory allocation requirements while still ensuring data consistency for the updated information.
3Reliability
If new index files are created for updates, then data consistency is maintained, but garbage collection overhead increases
Solution Approach 1:
The patent performs preliminary actions by pre-allocating and reusing index segment structures rather than creating new index files for each update. This allows the system to prepare the necessary data structures in advance and reuse them, eliminating the need for frequent garbage collection operations and improving overall indexing efficiency while maintaining consistency.
Solution Approach 2:
Instead of creating new index files that require garbage collection, the patent discards the need for frequent file creation by reusing existing index segment structures. The system recovers and reuses memory and data structures from previous operations, eliminating garbage collection overhead while maintaining data consistency through controlled updates.
4Reliability
If frequent field updates require reindexing, then data consistency is maintained, but processing time increases
Solution Approach 1:
The patent segments the index into multiple independent segments, allowing parallel processing of updates. When fields are updated frequently, only the relevant segments need to be processed, reducing the total processing time while maintaining data consistency through segment-level update management.
Solution Approach 2:
The patent applies partial action by performing updates only on the specific segments and fields that have changed, rather than reindexing entire documents. This reduces processing time significantly for frequent field updates while maintaining data consistency through targeted partial updates of the index structure.
Data Source
AI summary
A system and method of indexing documents to support frequent field updates without reindexing may include receiving, from an indexing application, first fields from a document to be indexed. The method may also include receiving, from the indexing application, second fields from the document to be indexed. The first fields are to an index file associated with the indexing application. The index file comprises a directory wrapper around at least a portion of a file system for the indexing application, wherein the wrapper (i) provides indications of when the in-memory file is flushed to the file system, and (ii) controls synchronization of the file system and a data store. The method may further include writing the second fields to a reverse index.