Content Indexing via Novelty Detection and Priority Crawling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Search engines face challenges in efficiently indexing and ranking content, particularly in distinguishing between original and copied content, leading to suboptimal search result rankings.
Innovation Solution
A computer-implemented method and system that evaluates submitted content to determine its novelty and authorship by comparing shingles and issuing high-priority crawling requests to search engines, prioritizing original content in search result rankings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard crawling and indexing is used, then all content is indexed, but original content cannot be distinguished from copied content leading to suboptimal search result rankings
Solution Approach 1:
The system performs preliminary actions by receiving notifications about published content before standard crawling occurs, allowing early evaluation and indexing of original content. This preliminary notification mechanism enables the system to proactively identify and prioritize original content rather than passively discovering it during routine crawling.
Solution Approach 2:
The system introduces an intermediary evaluation component that acts as a mediator between content publishers and the search engine indexing system. This intermediary evaluates submitted content to determine originality, then communicates with the search engine to request prioritized crawling and indexing, thereby resolving the information asymmetry between content sources and the search engine.
2Quantity of substance
If all submitted content is crawled and indexed, then comprehensive coverage is achieved, but processing time and computational resources increase significantly
Solution Approach 1:
The system applies local quality by differentiating the processing priority of different content based on their originality evaluation. Original content receives high-priority crawling and indexing with expedited processing, while non-original or duplicate content follows standard processing queues. This selective quality assignment optimizes resource allocation based on content value.
Solution Approach 2:
The system changes the priority parameter of crawling requests based on content evaluation results. When original content is identified, the system issues high-priority crawling requests with adjusted timing parameters, causing the search engine to process and index this content faster than standard content, thereby reducing indexing time for valuable original material.
3Speed
If high-priority crawling requests are issued for original content, then indexing speed improves, but system complexity increases due to evaluation mechanisms
Solution Approach 1:
The system extracts the content evaluation function as a separate, modular component that operates independently from the main crawling and indexing pipeline. This extracted evaluation module receives content notifications, assesses originality, and generates priority directives, thereby simplifying the overall system architecture by separating concerns and making the evaluation mechanism reusable and maintainable.
Data Source
AI summary
A system, computer-readable storage medium storing at least one program, and a computer-implemented method for expedited indexing of content is presented. Submitted content is received via a user interface, where the submitted content is copy-and-pasted content from an entity claiming authorship of the content. A link identifying an Internet address hosting the submitted is also received. Responsive to determining the submitted content is new relative to known content, the submitted content is indexed without visiting the Internet address.