Web Page Searchable Value Evaluation for E-commerce Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
E-commerce website search engines are inefficient due to a large quantity of irrelevant web pages, which hinder consumers' ability to find relevant products, as many indexed pages lack useful information and are automatically generated without consumer interest.
Innovation Solution
A system comprising a web crawler module, page analysis module, and page index module that evaluates retrieved web pages for searchable value by analyzing metadata, content, and HTTP headers to determine if they provide useful information relevant to user queries, and filters out irrelevant pages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the search engine indexes a large quantity of product and category pages, then the coverage of searchable content is improved, but the search result quality deteriorates due to numerous irrelevant web pages
Solution Approach 1:
The patent applies preliminary action by evaluating web pages during the crawling and indexing phase rather than during search. The page evaluation module analyzes each crawled page and assigns a quality score before indexing, so that only pages meeting certain quality thresholds are added to the search index. This prevents low-quality pages from entering the search results in the first place, resolving the contradiction between comprehensive coverage and result quality.
Solution Approach 2:
The patent implements local quality by applying different evaluation criteria and quality thresholds to different types of web pages. The system analyzes specific local characteristics of each page (such as presence of product information, category information, pricing details) and assigns quality scores based on these localized features. This allows the search engine to maintain high standards for product pages while being more lenient with category pages or informational pages, thereby improving overall search result quality without sacrificing coverage.
2Productivity
If automatically generated web pages are included in the index, then the quantity of indexed pages increases, but the usefulness of search results decreases due to lack of consumer interest
Solution Approach 1:
The patent implements feedback by using the quality evaluation results to control the indexing process. The page evaluation module provides feedback about the quality of crawled pages, and this feedback is used to determine whether to index each page. Pages with low quality scores (indicating automatically generated content without consumer interest) are excluded from the index, while high-quality pages are indexed. This feedback mechanism ensures that indexing throughput is maintained for useful pages while filtering out worthless automatically generated pages.
Solution Approach 2:
The patent applies the taking out principle by extracting and removing low-quality automatically generated pages from the indexing process. The evaluation module identifies pages lacking consumer interest through analysis of content characteristics, and these pages are extracted from the candidate set before indexing. This separation allows the system to maintain high indexing throughput for valuable pages while eliminating useless automatically generated content from the search index.
3Quantity of substance
If irrelevant web pages dominate search results, then the quantity of returned results increases, but the consumer's ability to find relevant products decreases
Solution Approach 1:
The patent applies preliminary action by pre-evaluating and filtering pages during the indexing phase, so that only high-quality relevant pages are stored in the search index. When consumers perform searches, the system returns results from this pre-filtered index, ensuring that all returned results have already been vetted for relevance and quality. This eliminates the need to filter irrelevant pages at query time and directly improves the consumer's ability to find relevant products.
Data Source
AI summary
A system for evaluating web pages for searchable content can be utilized to make an e-commerce search engine more effective by identifying pages with searchable value. In embodiments, a web page exhibiting “searchable value” is a page that provides useful information responsive to a user's query on an e-commerce search engine. One embodiment of a page evaluation system includes a web crawler module, a page analysis module, and a page index module. Web pages may be fetched and evaluated for various signals that indicate whether the web page exhibits searchable value. Examples of signals that may indicate that a web page lacks searchable value include a product page that does not have a product name nor a product image and a category page that does not have products listed in the page content. Web pages that lack searchable value may be excluded from subsequent e-commerce search results.