Location Hierarchy Classifier for Web Crawling Precision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions for determining the geographical relevance of documents during web crawling are inefficient due to the complexity of natural language place references, requiring significant processing and often result in crawling non-local content, which wastes resources.
Innovation Solution
A system and method that uses a location hierarchy and trainable location-specific classifiers to determine document association with a particular location by analyzing feature vectors representing occurrence counts of named entities, with the option to apply weighting schemes for improved performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a comprehensive toponymic database is used to identify and disambiguate place references, then measurement precision of location association is improved, but processing time and device complexity increase significantly
Solution Approach 1:
The patent segments the location identification task by organizing locations into a hierarchy (country, state, city, neighborhood levels) and applying different classification strategies at different levels. This allows the system to make quick decisions at higher levels and only perform detailed analysis when necessary, reducing overall processing time while maintaining precision.
Solution Approach 2:
The patent implements dynamic classification by training separate classifiers for different location levels and types, allowing the system to adapt its processing depth based on the document content and location hierarchy level. This dynamic approach optimizes the balance between processing time and precision.
2Measurement precision
If a comprehensive toponymic database is used to identify and disambiguate place references, then measurement precision of location association is improved, but device complexity increases
Solution Approach 1:
The patent divides the complex location identification system into modular components: a location hierarchy structure, level-specific classifiers, and a decision framework. This segmentation reduces overall system complexity by making each component independent and easier to manage.
Solution Approach 2:
The patent introduces a hierarchical dimension to location classification, organizing locations across multiple levels (country, state, city, neighborhood). This dimensional approach simplifies the classification problem by breaking down complex place references into manageable hierarchical steps.
3Productivity
If targeted web crawling with location determination is implemented, then productivity of relevant content retrieval is improved, but loss of time in processing each document increases
Solution Approach 1:
The patent performs preliminary location classification by determining the geographic level (country, state, city) and training appropriate classifiers in advance. This preliminary organization allows for faster processing during actual document crawling, as the system only needs to apply the pre-trained classifiers rather than performing complex analysis in real-time.
Solution Approach 2:
The patent implements dynamic processing by adjusting the depth and type of analysis based on the document content and location hierarchy level. This allows the system to process documents efficiently by applying minimal necessary processing while maintaining high retrieval productivity.
Data Source
AI summary
Provided herein are systems, methods and computer readable media for identifying content to represent web pages and creating a representative image from the content. An example method may include retrieving a web document using a uniform resource locator (URL) contained in a dequeued work item, determining, from the web document, candidate images for creation of the representative image including extracting image references, wherein the image references are extracted by identifying image tags with source attributes, values of which are URLs locating images, filtering the URLs using a blacklist of expressions designed to match the URLs of images comprising one or more predefined undesirable characteristics, and retrieving the images which do not match any of the expressions using an HTTP client, and creating the representative image, comprising at least modifying a chosen image selected from among the candidate images.