Dynamic Index Table Management for High-Throughput Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional full-text search engines face limitations in handling high indexing request throughput and large index table sizes, leading to slow indexing and search performance, especially when processing unbounded streaming data that requires real-time processing and efficient deletion of expired data.
Innovation Solution
A method to dynamically adjust the number of index tables based on the number of indexing requests and data volume, allowing for the creation of new index tables when demand increases and deletion of existing ones when demand decreases, thereby optimizing indexing throughput and search performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single inverted index table is used for full-text search, then the search engine can process documents within its capacity, but it cannot handle high indexing request throughput when the number of documents exceeds the limit
Solution Approach 1:
The patent divides a single large index table into multiple smaller index tables (first index table, second index table, third index table, etc.). Each index table stores a subset of the document indices, allowing the system to handle larger numbers of documents while maintaining manageable table sizes. This segmentation resolves the contradiction by distributing the document load across multiple tables.
Solution Approach 2:
The patent implements dynamic management of index tables through a controller that monitors the number of documents and automatically creates or deletes index tables based on current workload. When the number of documents exceeds a threshold, new index tables are created; when documents are deleted, index tables are removed. This dynamic adaptation allows the system to optimize for both throughput and document quantity at different times.
2Quantity of substance
If the number of entries in an inverted index table becomes very large, then more documents can be indexed, but indexing and search operations become slow
Solution Approach 1:
By segmenting the large index table into multiple smaller index tables, each table maintains a manageable number of entries. This keeps indexing and search operations fast while collectively handling a large total number of documents across all tables.
Solution Approach 2:
Each index table is optimized to contain a specific range or subset of document indices, making each table's size and structure appropriate for its specific workload. This local optimization ensures that operations on individual tables remain efficient while the system as a whole handles large quantities of documents.
3Adaptability or versatility
If traditional full-text search engines process streaming data, then they can handle continuous data flow, but they have limitations in capacity and performance due to fixed index table structures
Solution Approach 1:
The controller dynamically creates and deletes index tables based on the incoming streaming data volume and processing requirements. This dynamic structure allows the system to adapt its capacity to match the streaming data workload, overcoming the limitations of fixed index table structures while maintaining high processing performance.
Solution Approach 2:
The index table management system serves multiple functions: it handles both traditional batch indexing and real-time streaming data indexing, manages document additions and deletions, and automatically optimizes the number of index tables. This multi-functionality allows the same system to handle diverse data processing scenarios with high adaptability and performance.
Data Source
AI summary
Embodiments of the present disclosure provide a method, device, and computer program product for managing index tables. There is provided a method of managing index tables, comprising: determining the number of indexing requests for documents, the indexing requests being received within a predetermined period of time; obtaining information related to a processing index table in an index processing system, the processing index table being used for storing indices generated based on the indexing requests, the index processing system further comprising a receiving index table, the receiving index table being used for storing at least a part of the indices in the processing index table; and adjusting, based on the number of indexing requests and the obtained information, the number of processing index tables in the index processing system. Embodiments of the present disclosure can improve indexing throughput of an index processing system when massive data arrive, and avoid the problem of infinite growth of index tables.