Incremental Web Search via Delta Index Segmentation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current search engines face inefficiencies due to the need to re-compute every query after an index update, despite most webpages remaining unchanged, and the heavy burden of daily index merges, which wastes resources and reduces effectiveness by not utilizing cached results effectively.

Innovation Solution

The architecture employs incremental computing by storing repeat query results in a frontend cache and using delta and total indices at index servers to selectively re-compute only changed documents, allowing unchanged pages to be served directly from the cache and reducing unnecessary processing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If every query is re-computed after index update, then query results are always up-to-date, but computational resources are wasted and processing time increases

Engineering Contradiction:

Improvequery result freshnessVSAvoidcomputational resource waste

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The index is segmented into changed documents (delta index) and unchanged documents (total index). This allows the system to separate the computation of changed documents from unchanged documents, enabling selective re-computation only for changed documents while serving unchanged documents from cache, thus resolving the contradiction between result freshness and computational waste

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different parts of the index are treated differently: the delta index is re-computed for changed documents, while the total index is served from cache for unchanged documents. This local differentiation allows the system to maintain freshness where needed (changed documents) while avoiding unnecessary computation where not needed (unchanged documents)

Inventive Principle:

Principle #3Local quality

2Reliability

If every query is re-computed after index update, then query results reflect latest changes, but processing time increases significantly

Engineering Contradiction:

Improvequery result freshnessVSAvoidquery processing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The index is segmented into changed documents (delta index) and unchanged documents (total index). This segmentation enables the system to compute only the changed documents instead of recomputing all documents, significantly reducing processing time while maintaining result freshness for changed content

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system pre-computes and stores results for unchanged documents in the total index before index updates. When a query comes in, the system can immediately serve results for unchanged documents from this pre-computed cache, avoiding the need to re-compute them during query processing

Inventive Principle:

Principle #10Preliminary action

3Reliability

If daily index merge updates the whole index, then the index is fully updated, but system performance deteriorates due to heavy processing

Engineering Contradiction:

Improveindex completenessVSAvoidsystem processing performance

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The index is segmented into delta index (changed documents) and total index (all documents). This segmentation allows the system to maintain index completeness by having both indices, while improving processing performance by only computing changes in the delta index during updates rather than recomputing the entire index

Inventive Principle:

Principle #1Segmentation

4Reliability

If unchanged webpages are re-computed, then query results are accurate, but computational waste increases

Engineering Contradiction:

Improvequery result accuracyVSAvoidcomputational waste

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The index is segmented into changed documents (delta index) and unchanged documents (total index). This allows the system to serve accurate results for unchanged documents from the pre-computed total index cache while avoiding computational waste by not re-computing them during index updates

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different parts of the index are treated differently: the delta index is re-computed for changed documents to maintain accuracy, while the total index is served from cache for unchanged documents, avoiding unnecessary computation and computational waste

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS8560509B2Incremental computing for web search

Publication Date: 2013.10.15 GC NORTH ACQUISITION LLC

AI summary

Architecture that performs incremental computing for web searches by employing methods at least for storing the results of repeat queries on unchanged webpages and for computing results for the repeated queries. The architecture includes one or more algorithms for pre-computing query results on index servers, for only selectively choosing index servers whose result for a query change for a query computation process, and for re-using the unchanged web pages stored in the cache and computing results upon changed index and unchanged index separately.