Personal Search Engine with Web Crawler for Custom Database
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional search engines are limited in indexing websites, covering only 4% of the internet, leading to insufficient search results for users, particularly researchers, as they cannot create their own databases or utilize web crawlers to expand search results.
Innovation Solution
A personal search engine with a one-click login function and web crawler that allows users to index websites by querying external search engines, collecting and adding relevant data to a database, while filtering out harmful sites.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a general web search engine is used, then search results are provided based on the engine's database, but the coverage is limited to only 4% of all Internet sites
Solution Approach 1:
The system enables users to perform self-service by allowing them to operate web crawlers themselves to collect and index websites according to their own needs, rather than relying on the search engine company's automated database construction
Solution Approach 2:
The search engine provides multiple functions including both standard search capabilities and user-controlled web crawling functionality, making it adaptable to different user needs from casual browsing to intensive research
2Ease of operation
If the search engine company maintains a centralized database, then search operations are simplified, but users cannot customize the database according to their specific research needs
Solution Approach 1:
The system segments the database creation process into two parts: the search engine company maintains a base database for general use, while individual users can create and maintain their own customized databases through web crawling, allowing both centralized efficiency and decentralized customization
Solution Approach 2:
The system transitions from a static, pre-defined database to a dynamic system where users can actively update and customize their databases by running web crawlers according to their changing research needs
3Loss of information
If users want to search for information on buried sites not covered by search engines, then more comprehensive information could be found, but users lack the capability to crawl and index these sites themselves
Solution Approach 1:
The search engine system acts as an intermediary that provides users with web crawling capabilities without requiring them to build complex crawling infrastructure themselves, bridging the gap between user information needs and technical implementation requirements
Data Source
AI summary
The present invention provides a personal-use search engine and a web crawler equipped with a login function. The present invention can construct and provide a search system and a database searchable and manageable by a user including a researcher or the like, by the user using a personal-use web crawler. After login with one-click SNS login function or login with e-mail, the user adds, to a database, sites crawled by a web crawler on a server accessible from a web browser. Accordingly, the user can obtain a satisfying search result from among data that is widely collected regarding a specific topic and can discover new valuable information in the data.