Document Index Update via Aged Content Checksums

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Web-based information retrieval systems face challenges in handling duplicate content and high frequency content changes, leading to resource wastage and poor user search experiences due to crawl skew and indexing issues.

Innovation Solution

A computing device-implemented method and system that compares documents with prior versions to identify common and uncommon portions, calculates checksums based on aged content, and associates documents with duplicate clusters, ensuring that documents with high frequency changes and crawl skew are indexed efficiently.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the indexer indexes all crawled documents including duplicates and versions with frequent content changes, then the index becomes more comprehensive, but it wastes storage space and degrades user search experience

Engineering Contradiction:

Improvecompleteness of indexVSAvoidstorage space wastage

Core Design Contradiction:

ReliabilityVSLoss of substance

Solution Approach 1:

The document is segmented into multiple portions based on content age. The indexer calculates the age of each portion by comparing with prior versions and identifies which portions are aged (stable) versus new (changing). This segmentation allows selective checksum calculation only for aged portions, enabling duplicate detection without processing the entire document including frequently changing content.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different portions of the document are treated differently based on their age characteristics. Aged portions (those unchanged for a threshold period) are used for checksum calculation and duplicate identification, while new portions are excluded. This local quality approach ensures that stable content is indexed for duplicate detection while dynamic content is ignored, preventing storage wastage of duplicate documents.

Inventive Principle:

Principle #3Local quality

2Productivity

If the indexer processes all document versions with frequent content changes, then all documents are available for search, but duplicate identification accuracy deteriorates

Engineering Contradiction:

Improvedocument availability for searchVSAvoidduplicate identification accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

Before performing duplicate identification, the system performs preliminary action by calculating the age of each document portion and determining which portions are stable enough for comparison. This preliminary filtering of aged portions ensures that subsequent checksum-based duplicate detection operates only on stable content, significantly improving identification accuracy while maintaining document availability in the index.

Inventive Principle:

Principle #10Preliminary action

3Quantity of substance

If the indexer handles documents with crawl skew (continuous content growth), then all crawled versions are indexed, but duplicate detection fails due to new content differences

Engineering Contradiction:

Improvenumber of indexed documentsVSAvoidduplicate detection accuracy

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

Documents experiencing crawl skew are segmented into aged portions (common across versions) and new portions (unique to recent versions). The checksum is calculated only from aged portions, allowing duplicate detection to succeed despite continuous content growth. This segmentation enables the system to handle crawl skew effectively while maintaining accurate duplicate identification.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Instead of processing the entire document including all new content, the system applies partial action by using only the necessary aged portions for duplicate detection. This partial processing approach prevents new content from interfering with duplicate identification while still maintaining comprehensive document indexing for search purposes.

Inventive Principle:

Principle #16Partial or excessive action

4Measurement precision

If the indexer compares entire documents for duplicate detection, then comprehensive comparison is performed, but processing time and computational resources increase

Engineering Contradiction:

Improvecomparison accuracyVSAvoidindexing processing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system extracts only the relevant aged portions from each document for checksum calculation and duplicate comparison, excluding new and changing content. This extraction process significantly reduces the amount of data processed during duplicate detection while maintaining comparison accuracy for stable content, thereby reducing indexing processing time and computational resource consumption.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS8001462B1Updating search engine document index based on calculated age of changed portions in a document

Publication Date: 2011.08.16 GOOGLE LLC

AI summary

A system receives a document that includes new content and aged content, and compares the document with a prior version of the document that includes the aged content but not the new content. The system also separates the new content and the aged content based on the comparison, determines ages associated with the new content and the aged content, and determines whether the ages of the new content and the aged content are greater than or equal to an age threshold. The system further calculates a checksum of the document based on the aged content when the age of the aged content is greater than or equal to the age threshold, and the age of the new content is less than the age threshold, and stores the calculated checksum.