Search Server Webpage Summary Comparison for Crawl Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing method of regular crawling of webpages is inefficient due to the large volume of data, resulting in long crawling times and wasteful use of computing resources, as it often involves re-crawling un-updated webpages.
Innovation Solution
A data updating method where a search server provides a terminal with a search result webpage containing pre-stored summaries, which the terminal uses to compare with current summaries of crawled webpages, sending comparison information to the server to update only the necessary webpages, thereby reducing unnecessary crawling and resource waste.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the search server crawls all webpages regularly according to a preset time period, then the timeliness of search results is improved, but the crawling time and computing resource consumption increase significantly
Solution Approach 1:
The patent extracts only the essential comparison information (webpage summaries) from the crawling process. Instead of crawling entire webpages regularly, the system extracts and compares only the summary content to determine whether full crawling is necessary, thereby reducing time loss while maintaining result timeliness
Solution Approach 2:
The patent applies partial action by performing a simplified comparison operation (summarization and comparison) rather than complete crawling. This partial check is performed on all webpages, while full crawling is performed only on those that show changes, optimizing the balance between timeliness and time consumption
2Reliability
If the search server crawls all webpages regularly, then the freshness of webpage data is improved, but the computing resources are wasted on re-crawling un-updated webpages
Solution Approach 1:
The patent performs preliminary action by conducting a lightweight summarization and comparison check before deciding whether to perform full crawling. This preliminary step identifies which webpages have actually changed, allowing the system to avoid unnecessary full crawling of un-updated webpages and thus reduce computing resource waste
Solution Approach 2:
The patent implements a feedback mechanism where the comparison result (whether the summary has changed) determines the next action. If the summary remains unchanged, no further crawling is performed; if it has changed, full crawling is triggered. This feedback-based decision process optimizes resource allocation by acting only when necessary
3Reliability
If the search server adopts regular crawling of all webpages, then the completeness of search data is improved, but the crawling efficiency decreases
Solution Approach 1:
The patent segments the crawling process into two distinct stages: (1) a lightweight summarization and comparison stage applied to all webpages, and (2) a full crawling stage applied only to webpages that show changes. This segmentation allows the system to maintain data completeness while dramatically improving overall crawling efficiency by avoiding redundant full crawls
Data Source
AI summary
Embodiments of present application provide a data updating method, an apparatus, a search server, a terminal and a storage medium. A search server provides a terminal with a search result webpage including one or more webpages associated with a search keyword, where the search result webpage also includes a pre-stored webpage summary obtained by the search server by crawling webpage content of each webpage in advance; then the terminal crawls a target webpage triggered by a user to obtain a current webpage summary, and the terminal sends comparison information of consistency comparison between the current webpage summary and the pre-stored webpage summary to the search server; and finally the search server updates the pre-stored webpage summary of the target webpage according to the comparison information of the target webpage.