150 results about "Web crawler" patented technology

The present application relates to a data processing method and system, in particular to a vocational education multi-source data federal governance method and system, belonging to the field of vocational education informatization and big data technology, which first acquires and preprocesses vocational education multi-source data through network crawler and OCR technology; then constructs a knowledge graph in the form of triplets and maps it to a multi-dimensional tensor space; then evaluates the compatibility of multi-source data through topological feature analysis, and performs feature fusion that maintains topological invariants; finally, based on a quantum probability field model, knowledge reasoning and services are provided, the present application uses tensor representation to improve the expression ability of complex relationships, ensures the quality of heterogeneous data fusion through topological persistence, and introduces quantum probability theory to handle uncertainty in knowledge services, effectively solving the problems of multi-source heterogeneous vocational education data, complex relationships and insufficient service accuracy, and providing a comprehensive data federal governance solution for the vocational education field.

The invention discloses a multi-dimensional hotspot extraction method based on data analysis, relates to the technical field of data processing, and solves the problem that features and hotspot feature sets of text information data, image video data and voice information data are difficult to extract respectively after data is automatically captured in real time and preprocessed by using a web crawler technology. The hot events are difficult to identify by using a clustering algorithm, and analysis and evaluation of time dimension, space dimension and emotion dimension are performed on the hot events; and the development trend of the hot event is difficult to predict. According to the method, text information, image video and voice information data are automatically captured through a web crawler, multi-modal fusion is carried out after preprocessing and hot spot feature extraction, a hot spot event is identified by utilizing a clustering algorithm, and multi-dimensional analysis, evaluation and prediction are carried out on the hot spot event.

A system for dynamic cybersecurity scoring using traffic fingerprinting and score improvement, that uses a web crawler that sends message prompts to external hosts and receives responses from external hosts, a time-series data store that produces time-series data from the message responses, and a directed computational graph module that analyzes the time-series data to produce a weighted score representing the overall cybersecurity state of an organization.

The invention discloses a self-adaptive multi-dimensional evaluation high-quality scientific and technological information screening method and system, and relates to the technical field of information screening, and the method comprises the following steps: collecting data from a database by using an API interface and a web crawler technology, and establishing a multi-source data set; establishing a preprocessing data set; performing feature extraction on the preprocessed data set, and establishing a multi-dimensional feature set after standardization processing; carrying out dimension evaluation under a six-dimensional evaluation channel on the multi-dimensional feature set, and establishing a dimension score; and after the dynamic weight mapped by the dimension score is adaptively configured, weighted calculation is executed, and a scientific and technological information screening result is generated. The technical problems that in the prior art, due to the fact that the scientific and technological information screening dimension is single, the evaluation weight is fixed and different scenes are difficult to adapt, the screening result is insufficient in accuracy and comprehensiveness are solved, and the purposes of achieving self-adaptive multi-dimensional evaluation and high-quality screening of the scientific and technological information and improving the screening efficiency are achieved. And the accuracy and comprehensiveness of scientific and technological information screening are improved.

The invention relates to the technical field of asset assessment. The asset evaluation intelligent system based on the AI technology is provided, and the system comprises the steps that web crawler capture processing is conducted on a data source of a public network channel, and a first type of heterogeneous data is obtained; aPI interface docking processing is carried out on the data source of the enterprise internal system, and second-class heterogeneous data is obtained; performing logic splicing processing on the first type of heterogeneous data and the second type of heterogeneous data to generate multi-source heterogeneous asset evaluation data; performing model matching processing on the asset value characteristic matrix based on the asset type, and selecting an estimation model architecture; performing interpretability analysis processing on the dynamic evaluation parameters to generate a key factor contribution degree and an interpretable evaluation conclusion; and performing document processing on the contribution degree of the key factor and the interpretable evaluation conclusion to generate an interactive evaluation document so as to improve the multi-source data integration capability, enhance the risk dynamic evaluation adaptability and strengthen the real-time relevance and interpretability of the evaluation document.

Embodiments of the technology described programmatically decrease the number of spam Uniform Resource locators (URLs) that are accessed from untrusted domains when the subdomain prefix is above a threshold probability of having been randomly generated. In this regard, prior to adding a discovered set of URLs to a crawl queue of a web crawler, the URLs are filtered into URLs from trusted domains and untrusted domains determined by a statistical model. The trusted domain URLs are added to the crawl queue, and the remaining URLs are sandboxed to filter out spam URLs. The subdomain prefixes of the sandboxed URLs are applied to a neural network to determine the probability that the subdomain prefixes are randomly generated. When a subdomain prefix is above a threshold probability of having been randomly generated, the subdomain is determined to be a spam subdomain and can be blocked.

A vulnerability documentation system detects vulnerabilities having outdated or undocumented formatted descriptions for corresponding remediation actions. A web crawler crawls the Internet for configuration data for software / firmware affected by the detected vulnerabilities and descriptive content for the remediation actions. The vulnerability documentation system prompts and LLM with a prompt for each detected vulnerability comprising instructions to generate a formatted description for remediation actions using the crawled configuration data / descriptive content. The vulnerability documentation system then populates natural language descriptions of remediation actions from the formatted descriptions and pushes the natural language descriptions to affected devices.

A system for dynamically updating data for course generation, the system including a web crawler operating on a server, wherein the web crawler is configured to identify one or more predetermined HTML elements on a plurality of web pages, identify isolated data as a function of the one or more predetermined HTML elements, compare the isolated data to a modification baseline and generate regulatory data as a function of the isolated data and the comparison, and a memory the memory containing instructions configuring at least a processor to receive the regulatory data, classify the regulatory data to one or more guideline categorizations, identify a plurality of course modules, modify the one or more course modules as a function of the regulatory data and transmit a notification associated with the one or more modified course modules to an end user.

The invention provides a method and system for detecting an unauthorized URL of a website based on automatic operation of a common user, belongs to the technical field of network security, and can at least partially solve the problems of low detection efficiency, poor accuracy and excessive dependence on manual operation in the prior art. Automatically logging in a website by using the registration information, processing a verification link, and obtaining session information; obtaining a JS file through a web crawler in a normal user login state; analyzing the JS file, extracting static, dynamic and relative URLs in combination with lexical analysis, grammatical analysis and sandbox simulation execution, and unifying the static, dynamic and relative URLs as absolute URLs; de-duplicating and normalizing the URL; accessing the URL through a common user session by using a network test tool, obtaining response data, comparing the response data with a high-authority user reference response, and judging an unauthorized URL; the method has the beneficial effects that automatic and comprehensive detection of the unauthorized URL is realized, and the detection efficiency and accuracy are remarkably improved.

The invention belongs to the technical field of landslide identification and prediction, and particularly relates to a landslide distribution probability prediction method and system based on remote sensing data, and the prediction method comprises the following steps: step 1 to step 6, selecting a research area, capturing remote sensing data, describing a curved surface form of the research area, and constructing a digital elevation model; after screening, introducing elevation data into MATLAB software, and drawing a three-dimensional topographic map in a three-dimensional coordinate system; constructing a landslide probability model, simulating variables in the landslide, deducing seismic data, and deducing slope data; constructing a WGEN model, and deducing a precipitation density function of the research area based on GAMMA distribution to realize precipitation simulation; the prediction system comprises a web crawler module, a local database, a processor and a touch screen. According to the method, the elevation and rainfall of the plateau area are associated, multiple models are constructed, the landslide distribution probability accuracy is high, and risk early warning can be carried out within 3 hours after an earthquake.

Disclosed herein are embodiments of systems, methods, and products comprises an analytic server, which automatically manages appointment scheduling. The analytic server receives a customer request to schedule an appointment. The analytic server determines the required data from both customer and service provider for making the appointment. The analytic server retrieves customer data comprising requested service attributes, user preferences, users attributes from internal database and external data source. The analytic server retrieves service providers' data comprising provider service attributes, providers' attributes from internal database and external data sources. The analytic server accesses external data source by web crawling various websites. The analytic server executes an artificial intelligence model to predict user preferences and needs. The analytic server determines potential service providers best matching the customer's input or predicted preferences. The analytic server generates an appointment for each matching service provider and transmits an electronic message comprising the appointments to customer device.

The invention provides an LLM-based engineering project scheme compliance examination method. The method comprises the following steps: Step 1, data preparation; implementing multi-dimensional data acquisition through a distributed web crawler system; step 2, processing various types of data of images, texts and tables by using a large language model (LLM) technology; step 3, performing Prompt fine tuning optimization on the data, so that the large language model can better understand and generate related contents of compliance review of the engineering project scheme; step 4, performing automatic compliance check based on rule and mode matching; by defining a series of rules and modes, automatic compliance review of project documents is realized; and Step 5, establishing a risk generation assessment and early warning mechanism based on the large model. Through automatic compliance inspection, the workload of manual auditing is reduced, and document specifications are ensured; and an intelligent risk assessment and early warning mechanism identifies and notifies potential risks in real time, so that compliance risks are effectively reduced, and the safety and the overall level of project management are improved.

The invention discloses a mobile application privacy policy compliance detection method based on a pre-training model, and aims to realize efficient and intelligent privacy policy compliance automatic detection. The method comprises the following steps: firstly, constructing a hierarchical privacy policy compliance detection index system according to domestic related laws and regulations and standards; secondly, collecting an original text of the privacy policy through a web crawler technology, and constructing an unlabeled corpus after cleaning and structured processing; thirdly, constructing a multi-label classification data set based on a mode of combining a large language model and manual review; mapping the text and the label to a unified semantic vector space by adopting a text-label joint embedding strategy, and inputting the text and the label into a multi-granularity classification model; according to the model, on the basis of an ERNIE pre-training model, context feature enhancement and deep semantic interaction are realized through a bidirectional long-short-term memory network, a self-attention mechanism and a text-label cross attention mechanism, so that the multi-label classification performance is remarkably improved; finally, according to a label prediction result output by the model and a preset index system, compliance judgment is automatically completed, and a structured detection report is generated. According to the invention, the automation degree and efficiency of privacy policy compliance detection are effectively improved.

A method and device for evaluating the influence effect of an academic recommendation algorithm based on a social robot, the method comprising: collecting academic papers, citation relationships and author information through a data interface or a web crawler, and screening target personnel data according to a research field; constructing a virtual social robot and establishing a research interest vector of the virtual social robot based on historical literature and research theme information of the target personnel; presetting multiple literature acquisition strategies for the social robot to form differentiated experimental conditions; controlling the social robot to perform operations such as paper retrieval, access and click recommendation on an academic platform, and recording literature access paths, recommendation results and browsing sequence data in real time; calculating the research theme distribution of the social robot according to a literature set contacted in the experiment, and comparing the research theme distribution with the initial research interest, so as to obtain the change degree of the research direction; and comparing the change results of the research direction of the social robot under different strategies, and quantitatively evaluating the influence degree of the recommendation system on the research direction evolution of the scientific researchers.

The invention discloses a website data analysis method and device based on a large model technology, and belongs to the field of fusion of web crawlers and artificial intelligence. According to the method, a local knowledge base drives a large model to generate a precise XPath rule, and the precise XPath rule is packaged into a decoupling plug-in and integrated into an existing crawler framework; a dynamic feedback optimization mechanism is innovatively designed, rule validity is monitored in real time by using a composite verifier, and XPath reconstruction and knowledge base updating are automatically triggered. According to the method, the high concurrency performance of a traditional crawler is reserved, the rule generation efficiency and the system robustness are remarkably improved, the manual maintenance cost is reduced, and the method is suitable for a website data collection scene with a complex structure or frequent updating.

The application discloses a kind of personalized recommendation method based on big data, the present application will elastic search data storage search and based on user's collaborative filtering algorithm and based on item's collaborative filtering recommendation algorithm is combined.For new user, recommend the top 15 data of highest click rate in elastic search database.For old user, the recommendation result of the user-based collaborative filtering algorithm and the item-based recommendation algorithm is linearly weighted fusion, and the top 10 data is selected for recommendation.The application can update the database regularly through the high automation of web crawler.Secondly, by combining TF-IDF algorithm with collaborative filtering algorithm, the sparsity problem of data can be alleviated to a certain extent.

This invention discloses a flood situation perception method and apparatus based on a weighted LDA algorithm, belonging to the field of flood disaster risk management technology. The method includes: acquiring posts related to flood events through web crawling technology, performing data cleaning and secondary processing to obtain multiple target words and storing them in a dataset; calculating the weight of each target word in the posts using a term frequency-inverse document frequency algorithm; comprehensively determining the number of flood topics using perplexity, consistency index, and Jensen-Shannon divergence; introducing the weights of each target word in the posts into the LDA model, sampling the LDA distribution using a weighted Gibbs sampling algorithm, and estimating the flood post-topic distribution and topic-word distribution to analyze and perceive the flood development trend. This invention can improve the accuracy and interpretability of topic identification, thereby achieving comprehensive perception and accurate understanding of the flood situation.

The invention provides an optical cable security situation information acquisition processing method and device driven by a large language model, and the method comprises the steps: searching a webpage which may have situation information according to a keyword, and transmitting a request to a query API of a search engine according to a search result to receive a URL searched by the search engine; according to the URL retrieved by the search engine, using a web crawler tool to access a webpage to obtain a corresponding text; extracting security situation information and situation original text fragments from the main body text by using a large language model, and arranging the security situation information and the situation original text fragments into formatted contents; retrieving the most similar situation from the text vector database according to the situation information by utilizing a retrieval enhancement generation technology; performing matching judgment on the situation information and the most similar situation by utilizing a large language model; performing matching judgment on the situation information and the situation original text fragment by utilizing a large language model; and storing the structured situation information into a situation structured database, and storing the situation original text fragments into a text vector database.

The application relates to a network crawler scheduling method associated with user behavior heat, and belongs to the technical field of network information acquisition. Network resource data is acquired through a network crawler, scheduling data is analyzed, the scheduling data comprises user preference behavior data for resources and resource historical update frequency data; user behavior heat values of the resources and resource update frequencies are respectively calculated according to the scheduling data, a resource update crawling queue is constructed on the basis, and the network crawler is scheduled according to the resource update crawling queue. The application firstly associates the user behavior heat for resources with the resource acquisition frequency, and schedules the network crawler according to the resource access heat, so as to provide the user with the resource collection effect closest to real-time update on one hand, and reduce the crawling frequency of non-popular resources, so as to reduce the information acquisition cost and save the network bandwidth on the other hand.

The embodiment of the invention provides a crawler method, device and system and electronic equipment. The crawler method, device and system are used for reducing the development workload and improving the expansibility of a crawler system. The method comprises the following steps: analyzing a configuration file to generate crawler reference data; the crawler reference data is used for triggering the test tool to execute automatic webpage operation; based on the crawler reference data, target data are obtained, and the target data are data generated in the automatic webpage operation process of the test tool; and according to the target data, generating a data file meeting knowledge base construction requirements. According to the method, crawler reference data is generated through a configuration file, and a test tool is triggered by the crawler reference data to perform automatic webpage operation, so that a web crawler is realized. In this way, the development workload can be reduced, and the problems that the code and service coupling degree is high, and expansibility is poor can be solved.

The application discloses a false movie review detection method based on a graph attention neural network, first constructs a network crawler to collect data of a movie platform and manually labels comment text information to construct a data set; uses collected user, movie and comment data to construct a heterogeneous graph, confirms nodes in the heterogeneous graph and extracts node features, uses a TF-IDF algorithm, normalization and a ConSERT model to extract the features of the nodes; constructs a graph attention network based on the heterogeneous graph, including a node-based attention mechanism and a meta-path-based attention mechanism, and obtains comment node embedding containing semantic information for classification. The application first proposes a way of using a heterogeneous graph to capture node context semantic information to detect false movie reviews, has better effect than a traditional text-based classification method, and provides a method and a train of thought for future false movie review detection.

The embodiment of the invention provides a data access method and device, equipment and a storage medium. The method comprises the following steps: sending a target access request to a server to instruct the server to generate a corresponding target access script according to a preset computational intensive algorithm, a signature algorithm and a temporary access certificate; obtaining a target access script returned by the server, and executing the target access script to generate a corresponding access token; and sending the access token to the server side to indicate the server side to feed back corresponding target access data to the client side after the access token passes the verification. According to the embodiment of the invention, safe and effective data access can be realized, the data access of the web crawler is effectively intercepted, the safety and reliability of the data access are further improved, and the defense capability for executing the data access aiming at the web crawler is remarkably enhanced.

The invention provides a sports event rebroadcasting information aggregation system and method based on intelligent analysis, and the system comprises a data collection layer which comprises a web crawler engine, a request management module, an anti-crawling bypassing module and a dynamic page rendering module, and is used for collecting original HTML data from an electronic program guide forecast website; the intelligent analysis layer comprises a preprocessing cleaning module, a target match matching module, a time processing module and a double traversal analysis module and is used for intelligently analyzing the collected original HTML data; the data storage layer is used for storing the analyzed structured data; and the exception handling mechanism is used for handling network request exception of the data acquisition layer, analysis exception of the intelligent analysis layer and storage exception of the data storage layer. The invention aims to solve the problems of large data error, weak dynamic adaptation, low storage efficiency and the like in the prior art, and is suitable for automatic acquisition, standardized processing, structured storage and efficient scene retrieval of sports event rebroadcast information.

The application discloses a kind of based on the waste plastic detection method of improvement M2Det, the described method includes: through camera real-time sampling and the way of web crawler, collection multiple waste plastic data;Original picture is randomly divided into training set, verification set and test set;Through data enhancement method expansion existing data set and obtain a large amount of data;Using improved M2Det algorithm establishes the identification model of waste plastic;Using deformable convolution instead of ordinary convolution, using ResNeXt-101 as main network instead of ResNet-101 to improve feature extraction capability, establish final detection model;Based on the detection model trained, through camera shooting picture obtains final detection result.The application effectively solves the problem that waste plastic sorting background is complex and the problem that different plastic shape is greatly different, improves the identification precision of detecting multiple common waste plastic objects, and is helpful to practical production application.

The application discloses a multi-dimension hot spot extraction method based on data analysis, relates to the technical field of data processing, and solves the technical problems that it is difficult to automatically and timely capture data by using a network crawler, and it is difficult to respectively extract feature and hot spot feature set of text information data, image and video data and voice information data after preprocessing; it is difficult to use a clustering algorithm to recognize hot spot events, and it is difficult to analyze and evaluate hot spot events in time dimension, space dimension and emotion dimension; and it is difficult to predict the development trend of hot spot events. The application automatically captures text information, image and video and voice information data by using a network crawler, carries out multi-modal fusion after preprocessing and hot spot feature extraction, recognizes hot spot events by using a clustering algorithm, and analyzes, evaluates and predicts hot spot events in multiple dimensions.

The present invention generally relates to the field of taxonomy identification. Identifying product taxonomy from a product abbreviation is currently performed manually and consumes lot of time and effort. Hence, embodiments of present disclosure provide an automated method for identification of product taxonomy from product abbreviation. First, a brand name of product abbreviation is predicted using a Large Language Model (LLM) and a brand list. Then, possible expansions of the abbreviation are generated based on the brand name using acronym expansion dictionary. Among the generated possible expansions, a relevant one is identified using the LLM. Later, web scraping and web search techniques are applied on the relevant expansion to obtain associated top k matches of a supergroup, a product group and module of the predicted relevant expansion. Finally, product taxonomy is predicted based on the top k matches using a LLM augmented taxonomy classification by Retrieval Augmented Generation.

The invention relates to a traditional village culture digital protection method and system, and relates to the field of digital technology, and the method comprises the steps: obtaining a building shooting image shot by an unmanned aerial vehicle and a specific shooting position corresponding to the building shooting image; determining an image recognition state according to the building shot image, and determining an actual damage position according to the specific shooting position and the building shot image when the image recognition state is consistent with the damage state; determining a contrast shot image according to a web crawler technology and image similarity calculation, and determining a contrast mapping image in the contrast shot image; determining an image conflict value according to any two contrast mapping images, and concluding the contrast mapping images according to the image conflict value to determine an image conclusion set; and determining an effective contrast set in the image induction set, and updating the building shooting image according to the effective contrast set for digital building model construction. The method has the effect of improving the traditional village architectural culture protection effect.

The invention belongs to the technical field of electronic digital data processing, and particularly relates to a new energy automobile maintenance data analysis and management system which comprises a multi-source data acquisition module, a data storage module, a data analysis module, a visual display module, a prediction and early warning module, a system management module and a feedback and optimization module. According to the new energy automobile maintenance data analysis and management system, based on the multi-source data acquisition module, the maintenance record of a new energy automobile is obtained by establishing interface butt joint with a data source, and various data in the driving process of the automobile are collected and uploaded in real time through a sensor; climate conditions, road condition information and part market price fluctuation data, related to maintenance, of different regions are obtained through a web crawler technology, then the external data are combined with maintenance and use data of a vehicle, fault causes and maintenance cost are analyzed from a more comprehensive perspective, and then comprehensiveness of analysis results is guaranteed.

The disclosure provides a cloud platform interface generation method, which can be applied to the technical field of cloud computing. The method comprises the following steps: obtaining cloud platform data through a web crawler technology; classifying the cloud platform data to determine first target data and second target data, wherein the first target data comprises environmental information data and application information data, and the second target data comprises application node information and container information; establishing a target model for the first target data and the second target data, wherein the target model comprises an application layer, a template layer and a container layer; and encapsulating the target model to generate interface information, wherein the interface information comprises an interface calling method and an interface document. The disclosure also provides a cloud platform interface generation device, equipment, a storage medium and a program product.

The present application relates to a kind of brand influence dynamic monitoring method based on big data driving, comprising the following steps: collecting the webpage data of various network media and various forums by network crawler and carrying out pretreatment to webpage data;Media public opinion sentiment classification model is built, and the public opinion information in the webpage data of various network media after pretreatment is classified by media public opinion sentiment classification model;Hot topic mining model is built, and hot topic in the webpage data of various forums after pretreatment is mined by hot topic mining model, meanwhile, hot topic tracking model is built, and hot topic is tracked by hot topic tracking model;Brand influence trend prediction model is built, and the influence trend of the brand is predicted by the public opinion information related to the brand corresponding to the model and relevant hot topic, which provides early warning capability for brand party.