Network Crawler Format Conversion for News Recommendation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Network crawling systems face challenges in managing vast amounts of internet information due to varying formats and locations of data on different websites, which hinders the identification of specific information like titles, and is processor-intensive, affecting personalized news recommendation engines.
Innovation Solution
A network crawler extracts titles from media sites by identifying and converting different formats into a common format, storing them in a user profile interest database, and using maximum weighted matching in the word2vec space to recommend news based on user interests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Difficulty of detecting and measuring
If network crawling software crawls the Internet to identify specific types of information, then information classification capability is improved, but processor intensity increases and interferes with other processes
Solution Approach 1:
The system segments the crawling process into separate dedicated crawler processes that operate independently from the recommendation engine. This separation allows the crawling function to be performed without interfering with other processes while maintaining information identification capability.
Solution Approach 2:
The patent introduces intermediary components including a data processor that receives crawled information and performs format conversion, and a recommendation engine that receives processed data. This intermediary structure reduces processor intensity by distributing computational tasks across multiple specialized components.
2Difficulty of detecting and measuring
If network crawling software crawls the Internet to identify specific types of information, then information classification capability is improved, but processor intensity increases
Solution Approach 1:
The crawling system is divided into separate dedicated crawler processes that run independently, segmenting the computational load and allowing the recommendation engine to focus on its core function while crawlers handle information collection.
Solution Approach 2:
The data processor performs preliminary format conversion and data standardization on crawled information before it reaches the recommendation engine. This preliminary action reduces the computational power required by the recommendation engine by pre-processing the data into a usable format.
3Adaptability or versatility
If information from different websites is processed in various computer formats, then website-specific format flexibility is maintained, but information extraction difficulty increases
Solution Approach 1:
The data processor acts as an intermediary that receives information in various website-specific formats and converts them into a standardized internal format. This intermediary layer maintains adaptability to different source formats while simplifying information extraction for the recommendation engine.
Solution Approach 2:
The system changes the format parameter of incoming data from various website-specific formats into a standardized format through the data processor. This parameter transformation enables consistent information extraction across different sources without sacrificing format flexibility.
4Adaptability or versatility
If information location varies dramatically across webpages, then website-specific layout flexibility is maintained, but information identification difficulty increases
Solution Approach 1:
The data processor changes the positional parameter of information elements from various locations on different webpages into a standardized structure. This parameter transformation allows the system to maintain adaptability to different layouts while simplifying the identification of specific information elements.
Solution Approach 2:
The data processor performs preliminary reorganization of information elements from their original varied positions into a standardized format before passing data to the recommendation engine. This preliminary repositioning action reduces identification difficulty while preserving the ability to handle diverse webpage layouts.
Data Source
AI summary
A network crawler crawls one or more media sites to extract a plurality of titles for information contained in the one or more media sites. For example, the network crawler may extract the titles on the one or more media sites by identifying different computer formats and converting the titles in the different computer formats into a common computer format for comparison. User profiles which, includes user's interest are stored in a user profile interest database. A novel personalized news recommendation engine recommends news similar to the interest specified in a user profile, where maximum weighted matching is applied to score the similarity of news against user interest in the word2vec space.