Real-Time Indexing for High-Volume Data Search Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional search engines struggle to handle high data updating rates, such as those seen in microblogging platforms, where hundreds or thousands of news stories appear per second, as they can only achieve data updates in minutes, leading to inefficiencies in real-time news searching.
Innovation Solution
A data searching system comprising multiple databases with a fixed maximum capacity, where data is written in real-time and indices are created and updated, allowing for rapid indexing and searching, with the ability to overwrite older data when databases are full, and transmitting these indices for immediate search results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional search engines use periodic index creation, then system complexity is reduced, but data updating speed deteriorates (only minute-level updates)
Solution Approach 1:
The patent divides the indexing system into multiple independent index modules, each responsible for creating and maintaining indices for specific databases. This segmentation allows parallel index creation across multiple databases simultaneously, dramatically increasing data updating speed from minute-level to second-level updates, while each module maintains manageable complexity through specialized responsibility.
Solution Approach 2:
The system performs preliminary index creation and maintenance actions continuously in the background before search requests arrive. Index modules proactively create and update indices as data is written to databases, so that when search operations occur, the indices are already ready, enabling immediate real-time search results without waiting for periodic index regeneration.
2Productivity
If multiple databases are used to store real-time data, then data capacity and updating rate are improved, but index creation time increases
Solution Approach 1:
The patent assigns each database its dedicated index module, enabling parallel index creation operations across multiple databases. This eliminates the sequential bottleneck where one index would block others, allowing index creation time to remain constant even as data processing capacity scales by adding more databases.
Solution Approach 2:
Each index module autonomously manages its corresponding database's indexing without requiring coordination or waiting for other modules. The self-service approach allows independent, concurrent index creation operations, maintaining efficient index creation time while significantly increasing overall data processing capacity through parallel operations.
3Measurement precision
If indices are created for all databases, then search accuracy is improved, but system resource consumption increases
Solution Approach 1:
The patent implements specialized index modules that create optimized, database-specific indices tailored to the characteristics and query patterns of each data source. This local quality approach ensures high search accuracy for each database type while avoiding the resource waste of creating generic, one-size-fits-all indices, as each index is precisely tailored to its specific data domain.
Solution Approach 2:
By segmenting the indexing function into dedicated modules for each database, the system creates only the necessary indices for specific data types rather than comprehensive indices for all data. This reduces overall resource consumption while maintaining search accuracy, as each module efficiently creates minimal required indices for its specialized domain.
Data Source
AI summary
A data searching system is provided for real-time database searching. The data searching system includes a plurality of databases having respective maximum data capacity, and a searching module configured to provide a searching service interface. The data searching system also includes an index module configured to write received data into one of the plurality of databases, to treat the received data as the most up-to-date data to be stored over data having a longer existence time period in the plurality of database when the plurality of databases are filled; and to create indices of the plurality of databases. Further, the data searching system includes a transmission module configured to send the created indices of the plurality of databases to the searching module to provide searching results of the data searching system.