1239 results about "Web crawler" patented technology

A visual index method provides graphical output from search engine results or other URL lists. Search engine results or a list of URLs are passed to a web crawler that retrieves the web page and other media information present at the associated URL. The web crawler then passes this information to a page renderer which also receives image scale and format information regarding the web pages present at the URLs. The graphical information as well as other media information is then rendered into a reduced graphical form so that the page may be summarily reviewed by the user. Media, visual, or other information may also be downwardly scaled as appropriate or rendered in its original as appropriate (such as with audio data streams). A variety of convenient formats allows the user to quickly and readily scan the presentation at the URL web pages or other data present. Image maps associated with the reduced images may also provide hyperlink access to the linked web page and/or multimedia allowing the links present on the web page in its original to be accessed through the reduced image provided by the web page renderer.

In a pay-for-placement search system, the system makes search term recommendations to advertisers managing their accounts in one or more of two ways. A first technique involves looking for good search terms directly on an advertiser's web site. A second technique involves comparing an advertiser to other, similar advertisers and recommending the search terms the other advertisers have chosen. The first technique is called spidering and the second technique is called collaborative filtering. In the preferred embodiment, the output of the spidering step is used as input to the collaborative filtering step. The final output of search terms from both steps is then interleaved in a natural way.

The current disclosure uses the disciplines of Ontology and Epistemology to implement a context/content-based “multimedia conceptual search and planning”, in which the formation of conceptualization is supported by embedding multimedia sensation and perception into a hybrid database. The disclosed system comprises: 1) A hybrid database model to host concept setup. 2) A graphic user interface to let user freely issue searching request in text and graphic mode. 3) A parsing engine conducting the best match between user query and dictionaries, analyzing queried images, detecting and presenting shape and chroma, extracting features/texture of an object. (4) A translation engine built for search engine and inference engine in text and graphic mode. 5) A search engine using partitioned, parallel, hashed indexes from web crawler result, conducting search in formal/natural language in text and graphic mode. 6) A logic interference engine working in text and graphic mode, and 7) A learning/feedback interface.

Systems and methods are provided for automatically generating and maintaining user profile cookie sets. The user profile cookie sets may be used by a web crawler when gathering data such as advertisement data associated with one or more websites. The cookie sets may be generated by choosing a user profile with a set of user traits, selecting a set of websites related to the user traits, and browsing the selected set of websites using a web crawler while allowing the website to place cookies in storage of the web crawler. The cookie sets may be maintained by selecting a website to browse, selecting a user profile associated with the selected website, loading a previously generated cookie set for the selected user profile into the storage of a web crawler, and loading the webpage while allowing the website to place, update, or replace cookies in the storage of the web crawler.

The present invention provides a system and method for input of images containing faces for accessing, linking, and or visualizing multiple similar facial images and associated electronic data for innovative new on-line commercialization, medical and training uses. The system uses various image capturing devices and communication devices to capture images and enter them into a facial image recognition search engine. Embedded facial image recognition techniques within the image recognition search engine extract facial images and encode the extracted facial images in a computer readable format. The processed facial images are then entered for comparison into at least one database populated with facial images and associated information. Once the newly captured facial images are matched with similar “best-fit match” facial images in the facial image recognition search engine's database, the “best-fit” matching images and each image's associated information are returned to the user. Additionally, the newly captured facial image can be automatically linked to the “best-fit” matching facial images, along with comparisons calculated, and/or visualized. Key new use innovations of the system include but are not limited to: input of user selected facial images for use finding multiple similar celebrity look-a-likes, with automatic linking that return the look-a-like celebrities' similar images, associated electronic information, and convenient opportunities to purchase fashion, jewelry, products and services to better mimic your celebrity look-a-likes; health monitoring and diagnostic use by conveniently organizing and superimposing periodically captured patient images for health professionals to view progress of patients; entirely new classes of semi-transparent superimposed training your face to mimic other similar faces, such as mimic celebrity look-a-like cosmetic applications, and or facial expressions; intuitive automatic linking of similar facial images for enhanced information technology in the context of enhanced and improved organization, classification, and fast retrieval objects and advantages; and an improved method of facial image based indexing and retrieval of information from the web-crawler or spider searched Web, USENET, and other resources to provide new types of intuitive easy to use searching, and/or combined use with current key-word searching for optimized searching.

The present invention provides systems and methods for obtaining information from a networked system utilizing a distributed web crawler. The distributed nature of clients of a server is leveraged to provide fast and accurate web crawling data. Information gathered by a server's web crawler is compared to data retrieved by clients of the server to update the crawler's data. In one instance of the present invention, data comparison is achieved by utilizing information disseminated via a search engine results page. In another instance of the present invention, data validation is accomplished by client dictionaries, emanating from a server, that summarize web crawler data. The present invention also facilitates data analysis by providing a means to resist spoofing of a web crawler to increase data accuracy.

A system and method of distributed metadata searching is disclosed. The present invention permits an extension of the searching and retrieval functions of existing Internet web search engines by utilizing computational resources embodied in user computer systems and search browsers. By distributing the searching and scanning functions to the user level, the present invention reduces the computational and communications burden on Internet web search engines and crawlers, resulting in lower computational resource utilization by Internet search engine providers. Given the exponential growth rate currently being experienced in the Internet community, the present invention provides one of the few methods by which complete searches of this vast distributed database may be performed. The present invention permits embodiments incorporating a Search Manger (1001) further comprising a Service Results Manager (1013), User Profile Database (1012), Service Manager(1013), and Service Database (1014); a Light Weight Application SCANNER (1002); and a Search Engine (1008). These components may be augmented in some preferred embodiments via the use of a Search Browser (1003), Internet Communications (1004); Web Site(s) (1005), Web Crawler(s) (1006), and a Repository Database (1007).

Provided is a method and system for indexing documents in a collection of linked documents. A link log, including one or more pairings of source documents and target documents is accessed. A sorted anchor map, containing one or more target document to source document pairings, is generated. The pairings in the sorted anchor map are ordered based on target document identifiers.

A web-crawler system includes a plurality of network crawlers configured to fetch documents from hosts on a network and a cookie database shared by the plurality of network crawlers. The cookie database stores cookies and associated information for use by the plurality of network crawlers. Each network crawler is configured to retrieve one or more cookies from the cookie database so as to enable access to documents on at least one of the hosts on the network. In some embodiments, each of the network crawlers may be configured to detect any of a plurality of predefined cookie errors associated with fetching a document. In some embodiments, each of the network crawlers may also be configured to detect when a cookie in the cookie database has expired and to obtain a replacement cookie.

A dynamic-content web crawler is disclosed. These New Crawlers (NCs) are located at points between the server and user, and monitor content from said points, for example by proxying the web traffic or sniffing the traffic as it goes by. Web page content is recursively parsed into subcomponents. Sub-components are fingerpinted with a cyclic redundancy check code or other loss-full compression in order to be able to detect recurrence of the sub-component in subsequent pages. Those sub-components which persist in the web traffic, as measured by the frequency NCs (6) are defined as having substantive content of interest to data-mining applications. Where a substantive content sub-component is added to or removed from a web page, then this change is significant and is sent to a duplication filter (11) so that if multiple NCs (6) detect a change in a web page only one announcement of the changed URL will be broadcast to data-mining applications (8). The NC (6) identifies substantive content sub-components which repeatably are part of a page pointed to by a URL. Provision is also made for limiting monitoring to pages having a flag authorizing discovery of the page by a monitor.

A web crawler system has an automatic website crawler and a virtual browser that provides script related information to the website crawler. The virtual browser transforms an HTML document included in a web page of the website into an XML document, and builds a document object model containing document objects in a tree structure based on the XML document. The virtual browser extracts from the DOM scripts that are potentially executable, and executes the extracted scripts using a browser object model provided for the virtual browser containing objects and methods and properties that are used for script execution so as to capture script related information generated by execution of the scripts.

The present invention provides for the efficient downloading of data set addresses from among a plurality of host computers, using a plurality of web crawlers. Each web crawler identifies URL's in data sets downloaded by that web crawler, and identifies the host computer identifier within each such URL. The host computer identifier for each URL is mapped to the web crawler identifier of one of the web crawlers. If the URL is mapped to the web crawler identifier of a different web crawler, the URL is sent to that web crawler for processing, and otherwise the URL is processed by the web crawler that identified the URL. Each web crawler sends URL's to the other web crawlers for processing, and each web crawler receives URL's from the other web crawlers for processing. In a preferred embodiment, each web crawler processes only the URL's assigned to it, which are the URL's whose host identifier is mapped to the web crawler identifier for that web crawler. Each web crawler filters the URL's assigned to it by comparing them against a database of URL's already known by the web crawler and removing the already known URL's. If a URL is not already known to the web crawler, the data set corresponding to the URL is scheduled for downloading.

The invention discloses a Web information extractions system, which is characterized by comprising a retrieve analyzing module, a rule generation module and a data extraction storage module, wherein the retrieve analyzing module comprises a web crawler unit and an HTML resolver; the rule generation module comprises a single-slot extraction rule generation unit and a multi-slot extraction rule generation unit; and the data extraction storage module extracts data from web pages downloaded from the retrieve analyzing module and stores the data in a structural form according to the extraction rule generated by the rule generation module. The system has the following advantages: when single-slot extraction rules are generated, the interface operation is simple and easy to understand; for generating multi-slot extraction rules, the system provides a graphical interface to help a user label so as to save the time and the physical power for the user; for pre-generated extraction rules and mission sequences, the system provides two ways to achieve the extraction and the storage of batch tasks; and the system can finish the tasks of the extraction and the storage in preset period and time according to the parameters configured by the user.

Different URLs that actually reference the same web page or other web resource are detected and that information is used to only download one instance of a web page or web resource from a web site. All web pages or web resources downloaded from a web server are compared to identify which are substantially identical. Once identical web pages or web resources with different URLs are found, the different URLs are then analyzed to identify what portions of the URL are essential for identifying a particular web page or web resource, and what portions are irrelevant. Once this has been done for each set of substantially identical web pages or web resources (also referred to as an “equivalence class” herein), these per-equivalence-class rules are generalized to trans-equivalence-class rules. There are two rule-learning steps: step (1), where it is learned for each equivalence class what portions of the URLs in that class are relevant for selecting the page and what portions are not; and step (2), where the per-equivalence-class rules constructed during step (1) are generalized to rules that cover many equivalence classes. Once a rule is determined, it is applied to the class of web pages or web resources to identify errors. If there are no errors, the rule is activated and is then used by the web crawler for future crawling to avoid the download of duplicative web pages or web resources.

The invention provides a similar web page duplicate-removing system based on a parallel programming mode, comprises a web page content pre-processing module, a web page eigenvector extracting module,a web page feature fingerprint calculation module, a web page fingerprint on-line duplicate-removing module, a web page fingerprint distributed batch duplicate-removing module and a computing platformbased on specific distribution. The system can complete links of carrying out unified conversion of text content encoding, standardization of document structure, web page noise content abortion, thematic content analysis and identification of web pages, lexical segmentation of continuous text content, and the like on the web pages obtained by crawling of web crawlers, thereby forming eigenvectorswhich can present the web pages. Relative algorithms can be used to obtain web page fingerprints which present web page characteristics aiming at the vector. The system provided by the invention accurately and fast detects fully complete repetition or approximate repetition of the web page contents caused by site mirroring, web document transshipment, and the like on the condition of massive amount of data of Internet and completes corresponding repetition-removing works, thereby enhancing the storage efficiency of search engines and bringing better use experience for the search engines.