Dynamic Search Index Update via Log File and Generations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search engine index structures are static and inefficient for dynamic updates, leading to high disk space overhead, varying load on disks, cache reload issues, and lack of real-time indexing and crash recovery mechanisms, especially in handling incremental updates and multiversioning.
Innovation Solution
A method for dynamic updating of search engine indices using a log file, generations, delta lists, and checkpoints, allowing for incremental updates, real-time indexing, and efficient crash recovery, while maintaining high query processing efficiency and consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If partitioning and merging is used to handle dynamic updates, then the index can be updated incrementally, but disk space overhead increases to 100% in the worst case
Solution Approach 1:
The index is divided into multiple partitions that can be updated independently. Each partition maintains a subset of the index data, allowing incremental updates to proceed in parallel without requiring complete rebuilding of the entire index structure.
Solution Approach 2:
The invention implements a hierarchical structure where partitions are nested within the overall index framework. Each partition contains its own sub-partitions and metadata structures, enabling efficient space management where unused space in one partition can be reused by others.
2Productivity
If the largest partition is rebuilt during merging, then update capability is maintained, but disk load becomes highly varying and causes lookup performance degradation
Solution Approach 1:
The system dynamically adjusts the update strategy based on current system state. When lookup load is high, update operations are deferred or performed on smaller partitions only. When lookup load is low, background updates can proceed more aggressively, balancing update progress with query performance.
Solution Approach 2:
Updates are performed periodically in controlled batches rather than continuously or all-at-once. The system schedules update operations to occur during low-utilization periods, using periodic background tasks to merge partitions while maintaining normal query operations.
3Adaptability or versatility
If new partitions are introduced, then update flexibility is improved, but cache content is discarded and caches need to be reloaded causing performance drop
Solution Approach 1:
When a new partition is created, the system pre-warms the cache by loading essential metadata and frequently accessed data from the new partition before it becomes active. This preliminary caching reduces the impact when the partition is first introduced and minimizes cache misses during the transition period.
Solution Approach 2:
The invention introduces a cache validation and refresh mechanism that acts as an intermediary between new partitions and the cache system. Instead of discarding all cache content, the system selectively invalidates only the specific cache entries affected by the new partition while preserving unrelated cached data.
4Productivity
If multiple partitions are used, then update distribution is improved, but multiple disk operations are required when a single operation would suffice
Solution Approach 1:
The system merges multiple partition operations into consolidated disk I/O where possible. When multiple partitions require updates, the system batches their operations and performs them in sequence or parallel using shared I/O resources, reducing redundant disk head movements and improving overall I/O efficiency.
Solution Approach 2:
The partition structure uses a unified metadata format and shared index structures that can serve multiple partitions simultaneously. A single disk operation can update metadata that applies to multiple partitions, and shared data structures reduce the need for redundant storage and access operations.
Data Source
AI summary
In a method for a dynamic updating of an index of a search engine, wherein the index is an inverted index comprising a dictionary, a posting file with a posting list for each keyword of the index and a database log, the documents are inserted in the index in small batches called update generations, a list of all occurrences of keywords in the documents of each update generation is generated, the occurrence list is inserted in the database log, and for each keyword entered in the database a reference to a previous entry of the same keyword is created. This previous entry has a reference stored in the mass storage device as the last added entry of all recently keywords.—A search engine performing the method may be implemented on one or more servers with a mass storage device, and comprises a core search engine with a search subsystem and an indexing subsystem for creating a keyword index stored on the mass storage device and with the index realized as a dynamically updateable index.