311 results about "Data scraping" patented technology

A hybrid prediction system may aggregate electronic data to identify and initially predict an outcome of a future event and subsequently update the initial prediction. The system may include at least one processor and a memory. The processor may access data scraped from the Internet. The data may be associated with at least one future event. The processor may further store the scraped data, determine, from the scraped data, an initial prediction of the outcome of the at least one future event, generate, from the scraped data, an initial likelihood indication associated with the initial prediction, and transmit the initial prediction and the initial likelihood indication to a device associated with one or more users. The processor may further receive proprietary information, store the proprietary information, determine, using the scraped data and the proprietary information, a subsequent likelihood indication, and transmit the subsequent likelihood indication to the device.

The invention discloses a method and a system for fetching business data. The method includes configuring rule data required during fetching operation; reading the rule data, creating web page resource fetching tasks according to the rule data, and storing fetched web page resources according to configuration rules in a classified manner; creating data analysis tasks for the fetched web page resources, analyzing HTML (hypertext markup language) documents for the fetched web page resources to acquire required resource URIs (uniform resource identifiers) and filtering out the resources with incomplete data; creating resource download tasks, and downloading the resource URIs acquired by means of analysis in a breakpoint resume manner to acquire resource data; storing the resource data or fetching other resource data according to the integrity of the resource data and transmitting reporting information if the fetching operation cannot be completed normally. The method and the system have the advantage that problems that large quantities of resources are consumed when data are acquired while business data cannot be acquired by means of configuring relevant information are solved.

The invention discloses a product database forming method based on Internet data and a system. The method includes the steps of capturing webpage data with the theme relevance higher than a preset threshold value by the adoption of the focused crawler technology, performing structuralized storage on the captured webpage data, automatically classifying the structuralized storage webpage data according to the categories which products belong to, performing statistics on the frequency and the time of occurrence of attributes of the products in the webpage data after the automatic classification, performing weighting calculation on the frequency and the time of occurrence of the attributes of the products according to preset weighting, acquiring the decision value of the attributes of the products, and determining the sort order of the attributes of the products according to the decision value of the attributes of the products. The system comprises a data capturing module, a structuralized storage module, a data classifying module and an attribute deciding module. According to the product database forming method based on the Internet data and the system, a user can acquire comprehensive and summarized information without needing to collect and sort product information on the Internet, real-time performance of data is ensured, and real-time requirements of the user are met.

The invention relates to a data collection method based on an intelligent grasping system for science and technology service information. The method comprises the following steps: 1, data grasping: in terms of configuration of a crawler, a user releases a grasping task through a configuration module and a start module of a client and sets a webpage for grasping and corresponding rules; 2, loading of timing grasping tasks: the task released by the user is loaded dynamically to a timing grasping task list; 3, page downloading; 4, page parsing: pages in a queue are parsed; 5, adding to URL to be grasped; 6, data processing and storage: page data is subjected to parsing extraction processing, and extracted two-dimensional structured data is stored. The method can meet the requirement of the crawler for generality and the requirement of the science and technology service system for grasping, and convenient extension and plug-in type development are realized; parser rule configuration, width and depth of the pages subjected to grasping, grasping threads, database configuration or index configuration is added to specific service logic, and information can be grasped and collected intelligently.

The invention particularly discloses a single-image robot disordered target grabbing method based on pose estimation and correction. The method comprises the steps: S1, generating an image data set ofa to-be-grabbed object model; S2, constructing a convolutional neural network model according to the image data set in the step S1; S3, importing the two-dimensional image of the to-be-grabbed objectinto the trained convolutional neural network model to extract a corresponding confidence map and a vector field; S4, obtaining a predicted translation amount and a predicted rotation amount of the to-be-grabbed object; S5, finding the optimal grabbing point of the object to be grabbed and calculating the measurement translation amount of the depth camera; S6, performing grabbing safety distancecorrection according to the predicted translation amount of the object to be grabbed and the measured translation amount of the depth camera, executing correction data grabbing if the correction succeeds, and entering S7 if the correction fails; and S7, repeating the steps S3-S6. The disordered target grabbing method provided by the invention has the characteristics of high reliability, strong robustness and good real-time performance, can meet the existing industrial production requirements, and has a relatively high application value.

The invention discloses a recommendation method based on a knowledge-aware hypergraph neural network, which comprises the following steps: step 1, constructing a user hypergraph, and initializing a hyperedge by taking an article having an interaction relationship with a user as an entity node; constructing an article hypergraph, wherein the initial hyperedge takes an article having an interaction relationship with any user having the interaction relationship with the article as an entity node; step 2, convolution calculation is performed on the user hypergraph and each article hypergraph; and step 3, calculating the inner product of the user and all the articles to obtain the interaction score of the user and all the articles. The method has the beneficial effect that the auxiliary information of the articles is integrated into the vector representation of the user, so that the articles of which the user is more likely to have the interaction relationship can be selected from the numerous articles. The invention discloses a recommendation system based on a knowledge-aware hypergraph neural network. The recommendation system comprises: a data capture module; a knowledge perception hypergraph construction module; a domain convolution module; a hyperedge convolution module; and a prediction module. The method has the beneficial effects of simple operation and accurate prediction.

The embodiment of the invention discloses a computer robot crawling task distribution method and device, and a computer robot data crawling method and device. The computer robot data crawling method comprises the following steps: a crawled historical page is stored into historical crawling data, whether a historical page corresponding to the URL(Uniform Resource Locator) cluster of a crawling task is in the presence in the historical crawling data or not is judged when the crawling task is in the presence, first target page data can be directly extracted from an available historical page if the historical page is in the presence and the existing historical page is available, the part of the historical page does not need to be repeatedly crawled, and system resources are saved. Meanwhile, the network bandwidth usage rates of all computer robots are detected in fixed time, the mean value E(w) and the variance D(w) of the network bandwidth usage rate of each computer robot are calculated and stored, and then, the availability of each computer robot is calculated according to E(w) and D(w). For the available computer robots, the computer robot for executing tasks is selected according to the descending order of an availability probability so as to reasonably distribute computer robot resources.

The invention discloses a user data processing system which comprises a user data capture module, a data integration processing module, a quantitative classification statistics module, a residence track analysis module, a store resource database, a store resource updating module, a management server and a guide push initiation module. The user data capturing terminal is respectively connected withthe quantitative classification statistics module and the data integration processing module; wherein the quantitative classification statistics module is connected with the comprehensive quantitative evaluation module through the resident trajectory analysis module, the commodity resource database is connected with the comprehensive quantitative evaluation module, and the management server is respectively connected with the data integration processing module, the comprehensive quantitative evaluation module, the store resource database and the guide push initiation module. According to the invention, the user position information and the consumption record are analyzed to determine the preference and interest of the user, so that the guidance information can be accurately pushed to guidethe user to shop, the satisfaction degree of the user is greatly improved, and the sales of commodities in the shopping mall is promoted.

The invention provides a data grabbing method and a data grabbing system. The data grabbing method comprises the steps that a plurality of regular expressions are configured; the regular expressions are selected from the regular expressions in sequence according to the preset sequence of the regular expressions, data correlated with target data are matched in a target file according to the selected regular expressions, if the correlated data are matched, the correlated data are grabbed and returned, the selection of the regulation expressions from the regular expressions is terminated, the matching operation in the target file is terminated, and if the correlated data are not matched according to the regular expressions, prompt information is returned. By means of the technical scheme, when one system needs to obtain data of other systems, corresponding data can be obtained from other systems in a matching mode only by configuring the corresponding regular expressions, and data sharing can be conveniently carried out between the different systems.

The invention provides a resource fetching method and a device. An embodiment of the invention obtains the resource fetching configuration data, and then performs the operation of the data fetching according to the fetching configuration data; a complete fetching script is no need to be compiled, so that the operation is very simple and the accuracy is high, thus the efficiency and reliability of resource fetching are improved.

The invention provides a data fetching method and device of an OTT (over the top) application. The data fetching method comprises the following steps: indicating a to-be-tested terminal to fetch signaling layer data generated when the OTT application sends a signaling in business of the OTT application arranged in the to-be-tested terminal, wherein the signaling layer data comprise signaling information of a signaling sent every time and time information corresponding to signaling sent every time; fetching application layer data generated by the OTT application from the data output end of a simulative network device, wherein the application layer data comprise information of an application data packet sent every time and time information corresponding to the application data packet sent every time; and relating the signaling information of the signaling layer data matched with the time information with the application data packet information of the application layer data to form behavior data corresponding to the OTT application. By utilizing the data fetching method and device, the technical problem that the synchronous fetching of the signaling layer data and network layer data can not be realized so as not to guide an operator to optimize network behaviors of the OTT application in the prior art is effectively solved.

The application discloses an HTML template-based method for releasing graphics and text information via a television. The method comprises the steps as follows: setting a template file which is suitable for HTML/xHTML format played by the television; wherein the template file comprises a display frame and a formal parameter, and the display frame defines a display mode of metadata; the formal parameter is embedded in a specific position of the display frame; using a data capturing technology to capture the metadata from a webpage and storing the metadata in a database; extracting the metadata from the database and filling in the template file to replace the formal parameter, and rendering and generating a graphics and text page; converting the graphics and text page into a picture format; converting the graphics and text page in picture format into a PAL video signal of a television program channel, and playing on a television screen. The application further discloses an HTML template-based system for releasing the graphics and text information via the television, and a content releasing server.

The present invention provides a data capture method and system. The method comprises the following steps of receiving a data range of a user needing to capture; according to the data range, carrying out the data capture by a Baidu search algorithm and a Google search algorithm separately; taking the same results in the Baidu search algorithm and the Google search algorithm as the capture results. The technical scheme provided by the present invention has an advantage of good capture effect.

The invention discloses a behavior analysis method for construction personnel of a power substation engineering construction project, and relates to an intelligent video system based on a machine learning method and a chromatographic partition management technology. A computer network communication technology, a video monitoring technology and a safe area management technology are adopted to analyze and process construction site monitoring data. Function division and safety attributes of a regional grid working plane are set, color identification is performed on regions, real-time monitoring is performed on environment, facility and equipment state changes, threshold early warning is set, binding interaction is performed on personnel and the identified regions and equipment, and data capture and classification are performed on related information. According to the invention, real-time monitoring can be realized, abnormal conditions can be judged, and an alarm can be given at the fastest speed and in the maximum mode, so that beforehand early warning can be effectively carried out.

The invention discloses a public opinion data capturing method device and equipment and a storage medium. The method comprises the following steps of acquiring the configuration parameters of a data capturing task set by a user through a configuration parameter setting interface, wherein the configuration parameters include data capture entry address, filtered keywords, and data capture rule information; after the data capturing task is started, capturing the entry address and the filtered keywords according to the data to obtain page data; based on the page data, capturing the public opiniondata according to the data capturing rule information. The scheme of the invention can set the configuration parameters of the self-configuration of the interface based on the configuration parameterscalled by the user to capture of the public opinion data, thereby realizing the public opinion monitoring and improving the monitoring efficiency.

The invention relates to the technical field of data analysis and acquisition, in particular to a webpage data capture method. Data information of some websites with access right is captured quickly and effectively by means of establishing a data channel of concurrent execution and defining a data capture process of the websites. Facing to ERP (enterprise resource planning) software developers, a scheme for quickly conveniently defining data capture of corresponding websites is provided, and trouble of artificially accessing to website downloaded information is avoided by timed automatic data capture of a background.

The invention discloses a database system applied to a real-time big data scene. The database system applied to the real-time big data scene is characterized by comprising a data capturing module which is used for defining the range for capturing a data source, automatically capturing webpage data, performing content extraction and repetition exclusion, and performing context analysis; a feature database module which is used for storing Cookie data, advertisement position dada and linked data; a real-time database module which is used for performing indexing and sharding storage on the data in the feature database module; an advertisement putting module which is used for obtaining advertisement feature data from the real-time database module, comparing the advertisement feature data with the data in the feature database module, and finally determining whether to put advertisement to specific users and which advertisement is put to the users. The database system disclosed by the invention has the beneficial effects that as a three-level storage structure of internal memory, SSD and hard disk is adopted, the system not only provides high-speed real-time data access ability, but also can effectively cope with super-large scale data storage and has the disaster recovery ability.

The invention aims to provide a method and equipment for capturing data of a website. The method includes selecting a non-visited link from all links in a current root page according to information of a topological structure of the website and acquiring a next-layer page directed by the non-visited link; judging whether the next-layer page is a target information page or not according to a first preset rule; utilizing the next-layer page as a current root page when the next-layer page is not the target information page, and executing a step a and a step b repeatedly until a first preset condition is met; capturing the target information page when the next-layer page is the target information page; and utilizing a previous root page as a current root page when a second preset condition is met, and executing the step a, the step b, a step c1 and a step c2 repeatedly. Compared with the prior art, the method and the equipment have the advantages that a depth-first traversal mode is adopted, the target data of the integral website can be captured, the target data capturing accuracy is guaranteed, and the data capturing efficiency is improved.