62 results about "Web scraping" patented technology

Disclosed are combinations of authentication, session management and web scraping implemented on a mobile device to support a rich mobile application using secure connections to existing websites to access data sources. The mobile application presents information in logical units rather than screen by screen, and fetches data in the background for low perceived delay. The mobile application provides consistent navigation using the 12-key or QWERTY keypad. The mobile application maintains a history of screens, allowing the user to easily return to a prior screen. A web server allows phrases to be configured on-line by an individual user and downloaded to that user's mobile device to simplify data entry on the mobile device. A method of embedding user profile information in a signed application executable file that allows applications to be pre-configured per user. A licensing mechanism that supports multiple distribution channels.

A crawler that is either based on an interactive mode of operation or includes an interactive mode along with one or more other modes, such as automatic or manual. Similar to an automatic mode crawler, the crawler traverses web sites, web content and links. However, if the crawler encounters a structure that requires human interaction, such as a form, a radio button selector, a drop down selector, a human verification test, etc., the crawler pauses and prompts a user to take action.

The present invention provides a method, system, and computer program to help a user without any programming knowledge create data extraction rules for collecting data from websites at scale. A user only needs to provide a web page Universal Resource Locator (URL), then mark and assign the needed data to its type. For example, on an e-commerce website, this data can be the product name, price, description, and so forth. Marking is done by highlighting the correct part of the web page. This creates a data extraction rule that describes the web template of full website and can be used thereafter for automated web scraping from all pages on a particular website.

The invention discloses a webpage screening method and a webpage screening device. The method comprises that preset seed webpage is captured; uniform resource locator (URL) information included by the seed webpage is captured; webpage mass fraction corresponding to the URL information is calculated; the URL information is divided into corresponding candidate gather according to preset network address information; the URL information which is not greater than the preset pressure quota is screened out from each candidate gather, the URL information which is screened out and corresponding to the webpage mass fraction which is not lower than the webpage mass fraction and corresponding to arbitrary residual URL information in the relative candidate gather is screened out. The captured pressure value corresponding to the network address is ensured based on the preset pressure quota. The webpage corresponding to the URL information which is screened out is regarded as the target captured webpage. The method lowers the risk of the capturing webpage failure or the risk of banning site so that the goal of improving the success rate of capturing the webpage is achieved.

The invention discloses a web page crawling method and a spider. The method comprises the following steps: injecting seed URL into a Web database; generating a URL list based on the Web database; feeding back the URL in the URL list to a web page crawler; crawling the webpage by the web page crawler according to the fed back URL comforming to the corresponding visit mode of the web page; and updating the URL state in the Web database and injecting newly found URL based on the crawled web page, wherein the visit mode comprises requesting parameter socket, responsing parameter socket, requesting the corresponding relationship between the requesting parameter socket and the responsing parameter socket; the requesting parameter socket comprises requesting parameter, as well as the mapping relationship of the requesting parameter socket and the responsing parameter socket; the responsing parameter socket comprises a responsing parameter, as well as the extraction position information about the extraction position of the responsing parameter in http respongsing message.

The invention is suitable for the technical field of computers and provides a web page collecting method and device as well as a browser. The web page collecting method comprises the following steps of receiving a web page collecting command, and obtaining a web page link corresponding to a web page; calling a web page grabbing server in a cloud server farm to grab web page content corresponding to the web page link according to the web page link; saving the web page content in a cloud storage server in the cloud server farm. According to the web page collecting method and device as well as the browser, which are disclosed by the invention, the cloud storage of the collected web page content is realized, and the long-term validity of the collected web page is ensured, so that the collected web page content is not limited by time and access addresses, and the function of a bookmark of the browser is expanded.

The invention discloses an equipment and method for capturing second-level domain information associated with a main domain. The equipment comprises a webpage capturing module and a first domain capturing module, wherein the webpage capturing module is configured to be used for capturing the content of a webpage with address being set to be the main domain; the first domain capturing module is configured to analyze the content of the captured webpage, and obtain the second domain information associated with the main domain. According to the equipment and method for capturing the second-level domain information associated with the main domain, the second-level domain information associated with the main domain of the website can be effectively and comprehensively obtained, and therefore a webmaster user can know the domain visiting situation of the website more comprehensively.

The application discloses an HTML template-based method for releasing graphics and text information via a television. The method comprises the steps as follows: setting a template file which is suitable for HTML/xHTML format played by the television; wherein the template file comprises a display frame and a formal parameter, and the display frame defines a display mode of metadata; the formal parameter is embedded in a specific position of the display frame; using a data capturing technology to capture the metadata from a webpage and storing the metadata in a database; extracting the metadata from the database and filling in the template file to replace the formal parameter, and rendering and generating a graphics and text page; converting the graphics and text page into a picture format; converting the graphics and text page in picture format into a PAL video signal of a television program channel, and playing on a television screen. The application further discloses an HTML template-based system for releasing the graphics and text information via the television, and a content releasing server.

The embodiment of the invention discloses a method and a device for reading webpage resources, and electronic equipment. The method is applied to the webview controls of an Android operating system of 4.0 to 4.3 versions. The method comprises the following steps: if the loading state of webpage resources to be fetched is loading completion, obtaining the URL (Uniform Resource Locator) information of the webpage resources to be fetched, wherein the webpage resources to be fetched correspond to an obtained webpage fetching request; according to the package name of an application program which constructs the current webpage, obtaining a resource cache file path mapped by the package name; extracting a binary data file under the resource cache file path, and traversing the binary data file to obtain an information field matched with the URL information; and inquiring information before the matched information field, obtaining preset symbolic information, obtaining a webpage resource file corresponding to the URL information according to the information before the symbolic information and a filename calculation strategy, and reading the webpage resource file under the resource cache file path. The method and the device can be applied to improve web resource utilization efficiency.

A web page grabbing method is provided. A target web page on a website is grabbed, the target web page including a web page corresponding to a Hypertext Markup Language 5 (H5) content and a web page corresponding to a non-H5 content. The web page corresponding to the H5 content is detected according to web page source code of the target web page. Dynamic rendering is performed on the web page corresponding to the H5 content, to obtain a rendered web page. Content details information corresponding to the H5 content is extracted from the rendered web page.

The invention discloses a filtering expression and rendering engine based method for automatically monitoring update of a dynamic webpage. A user appoints an interested part in the webpage as a concerned point through a visualized interface, and an application or a client automatically generates a filtering expression corresponding to the concerned point; a server renders the dynamic webpage by utilizing the rendering engine to obtain the same page seen by the user, and extracts the concerned point of the user; and when the concerned point of the user is updated, the server pushes the update content to the user in time. According to the method, a customizable dynamic webpage monitoring program is realized by helping the user appoint the concerned point and utilize the rendering engine for automatically inspecting webpage update at the server, the problem of lack of customization for a conventional information subscription mode (such as RSS (really simple syndication)) is solved, the defect of incapability of analyzing the dynamic webpage in conventional webpage capture is also overcome, and the efficiency for obtaining webpage information update by the user is improved.

The embodiment of the invention belongs to the technical field of artificial intelligence, and relates to a policy big data mining method and device, computer equipment and a storage medium. In addition, the invention also relates to a blockchain technology, and the target policy data of the user can be stored in the blockchain. According to the application, data capture operation is performed onthe policy object URLs in the policy object list through the webpage capturer to obtain the initial policy data, and the initial policy data is automatically integrated to obtain the target policy data convenient for workers to check, in the whole implementation process, related information can be obtained without manually querying policy data, the problems of large manpower resource consumption and high manpower cost are effectively solved, meanwhile, the accuracy of information extraction can be ensured, and the data extraction rate is increased.

The invention provides a system and method for recognizing the posture of single pig body based on a stacked hourglass network. The method comprises the steps: acquiring image data and video data on the spot through photographing equipment, acquiring the image data through a web scraping technology, learning a preprocessed data set through the training of an improved stacked hourglass network model, automatically acquiring a skeleton diagram of a single pig in a pigsty within a shooting controllable range, and determining the category of the pig according to obtained pig body posture data; andwhen a certain posture of the pig is abnormal, warning and prompting a breeder to enter the pigsty to further check the health condition of the specific pig. With the system and the method, pigs in afarm can be effectively managed, and the pressure of technicians such as feeders is relieved to a certain extent.

The invention provides a web page grabbing method and a web page grabbing system based on big data. The web page grabbing method comprises the following steps of receiving a web page request of a user; classifying the big data according to key word classification of the web page; and transmitting the classified web page which corresponds with the web page request to the user. The web page grabbingmethod and the web page grabbing system provided by the invention have an advantage of high convenience in grabbing the web page.

In an illustrative embodiment, methods and systems for identifying prospective new clients based upon review of current clients include accessing a book of business of a user of a transactional platform to identify current clients, identifying, key terms relevant to each of the clients, automatically performing Internet searches, each search using different groupings of the key terms, automatically deriving from web sites of the search results information regarding a number of prospects, and presenting prospect information to the user. The key terms may be identified through performing web searches and web scraping of online information related to the current clients. The clients may be categorized based upon organizational attributes prior to searching. An initial set or sets of key termsmay be filtered through performing a number of data analyses on the key terms.

The invention discloses a dynamic webpage crawling method. The method comprises following steps: arranging at least two queues, crawling url of web-pages to be crawled and priorities, storing them into at least two queues and scheduling according to priorities of url stored in at least two queues; receiving elements of at least two queues called in order to obtain url of elements to be analyzed; and obtaining webpage content by analyzing url of queue elements. The dynamic webpage crawling method has following beneficial effects: procedures for crawling analyses and url of a link library can be scheduled simultaneously according to priorities so that webpages of higher priorities can be crawled firstly; by scheduling at least two queues, de-queuing efficiency and en-queuing efficiency of webpages can be improved; the time complexity is logN so that webpage crawling efficiency can be greatly improved.

The invention relates to a cascade crawling method for multi-level pages based on web crawlers. The method comprises: grabbing an upper-level page, storing grabbed data in an upper-level page data analysis table, and setting main key values for objects needing to continue to grab a lower-level page in the upper-level page data analysis table, wherein the main key values corresponding to the objects are different; grabbing a subordinate page and storing the captured data in a subordinate page data analysis table; setting a foreign key value for the lower-level page data analysis table, obtaining a main key value of an object corresponding to a lower-level page from an upper-level page data analysis table, and taking the main key value as the foreign key value of the lower-level page data analysis table, thereby realizing associated query of an upper-level webpage and a lower-level webpage after grabbed data falls to the ground. According to the method, a data acquisition mode capable ofrestoring logics before and after the webpage is provided, the webpage capture integrity is ensured, the data is stored according to the original webpage hierarchy sequence, and the associated multi-hierarchy page data can be conveniently obtained.