Thematic Web Corpus Construction via Streaming URL Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing web crawling methods are inefficient in building a thematic Web corpus with reasonable computational costs, precision, and recall, as they either suffer from low precision by crawling the whole Web or poor recall by stopping at out-of-topic pages.
Innovation Solution
A computer-implemented method where a client sends a structured query to a server storing a search engine index, and the server responds with a stream of URLs matching the query, allowing focused crawling on the client side, thereby improving precision and recall by filtering out irrelevant pages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard web crawling is used to collect thematic corpus, then recall is improved by following all links, but precision deteriorates due to inclusion of out-of-topic pages
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing topic probabilities for web pages during an initial indexing phase. This allows the system to make informed decisions about which pages to crawl during the corpus building phase, improving precision without sacrificing recall by using pre-established topic knowledge to guide selective crawling.
Solution Approach 2:
The patent introduces an intermediary mechanism using topic probability scores as a mediator between the crawler and the web pages. This intermediary metric allows the system to evaluate and select pages based on their thematic relevance, filtering out out-of-topic pages while maintaining access to in-topic pages through the probabilistic assessment.
2Measurement precision
If focused web crawling with topic probability scoring is used, then precision is improved by reducing out-of-topic pages, but recall deteriorates by stopping at out-of-topic pages
Solution Approach 1:
The patent applies partial action by selectively crawling only a subset of pages based on their topic probability scores rather than exhaustively crawling all pages. This partial crawling approach improves precision by focusing on relevant pages while maintaining acceptable recall by using probability thresholds that ensure inclusion of most in-topic pages.
Solution Approach 2:
The patent changes the parameter used for page selection from binary inclusion/exclusion to probabilistic scoring. By using topic probability as a continuous parameter, the system can fine-tune the balance between precision and recall through threshold adjustment, allowing flexible control over which pages are included in the corpus.
3Quantity of substance
If the server performs extensive storage and ranking operations to provide comprehensive results, then recall is improved, but device complexity and computational costs increase
Solution Approach 1:
The patent extracts the heavy computational tasks of topic probability calculation and page ranking from the server and relocates them to the client side. The server only performs simple queries and returns results, while the client handles the complex processing of topic probabilities and selective crawling, significantly reducing server complexity and computational costs.
Solution Approach 2:
The patent implements self-service by enabling the client to autonomously perform topic probability assessments and make crawling decisions based on pre-computed data. This shifts the computational burden from the server to the client, allowing the server to provide comprehensive results without bearing the full computational load.
Data Source
AI summary
The invention notably relates to a computer-implemented method, performed by a server storing an index of a search engine, for sending, to a client, the URLs of pages of a Web corpus that relates to a theme. The method comprises receiving, from the client, a structured query that corresponds to the theme, the structured query consisting of a disjunction of at least one keyword; determining in the index the group that consists of the URLs of all pages that match the query; and sending to the client the URLs of the group as a stream.Such a method improves the building of a thematic Web corpus.