391 results about "Web crawler" patented technology

The invention discloses a multi-modal data fusion method and system based on a large-model intelligent agent, belongs to the technical field of artificial intelligence, big data processing and intelligent agents, and aims to solve the technical problem of how to effectively integrate data of different modalities such as texts, images and voices by using the large-model intelligent agent. According to the technical scheme, the method comprises the steps that data collection and preprocessing are conducted, specifically, various modal data of texts, images and voice are collected through a web crawler, an API interface, a camera and microphone equipment, the collected data are preprocessed, the preprocessed multi-modal data are obtained, and the data quality is ensured; feature extraction and mapping: extracting corresponding modal features from the preprocessed multi-modal data through CNN and Transform models, mapping the different modal features to the same space, and combining the aligned features to form comprehensive feature representation; carrying out multi-modal fusion processing; and performing intelligent decision and feedback.

The invention relates to the technical field of data processing, in particular to a knowledge graph construction method for multi-source heterogeneous data of power transformation main equipment and a related system.The method comprises the steps that multi-source heterogeneous data collection is carried out, and for structured database data, a database connection tool is adopted, and needed data is obtained by writing SQL query statements; analyzing the semi-structured data by using a corresponding analysis library to extract key information, and converting the key information into a structured data format; for unstructured text data and webpage data, a natural language processing technology and a web crawler technology are adopted; according to the knowledge graph construction method for the multi-source heterogeneous data of the power transformation main equipment, the data utilization rate is increased, and the data can be enhanced and supplemented by introducing a data enhancement algorithm in secondary data processing.

The invention discloses a cooking robot menu management method and system based on big data and a medium, and relates to the technical field of cooking robot menu management. The cooking robot menu management method comprises the following steps: a data acquisition step: collecting various menu data through a web crawler technology; a data preprocessing step: cleaning the collected menu data; a data analysis and classification step: analyzing the preprocessed menu data by using a data mining algorithm; a menu recommendation and customization step: recommending a proper menu to the user by using a collaborative filtering algorithm or a content-based recommendation algorithm according to historical ordering records, health requirements and personal taste preference information of the user; and an intelligent cooking control step: the cooking robot receives the customized cooking instruction. According to the invention, the modules cooperatively and efficiently operate, links of data acquisition, processing, analysis, recommendation, cooking control and the like are closely linked, the system response is rapid, and the user operation experience and cooking efficiency are greatly improved.

The invention discloses an electronic product personalized marketing method based on a big data accurate portrait, and relates to the technical field of big data processing, and the method comprises the following steps: multi-source data collection and cross-domain fusion processing: obtaining a multi-source data set related to an electronic product from each platform of the Internet by using a web crawler and a streaming processing engine, the multi-source data set is subjected to cleaning classification and alignment fusion processing to generate multi-modal data, and then the multi-modal data is updated in real time to dynamically reflect data changes, so that a foundation is laid for subsequent data analysis; according to the method, multi-source data is collected by using a web crawler and a streaming processing engine, quantitative analysis is performed on the data by using a Hofscode culture dimension model and a Maslow demand level theory, a user portrait is constructed by using a distributed data flow engine Apache Flink to dynamically update a label, and a GAN (GAN Adversarial Generative Network) anomaly detection and evaluation analysis method is applied, so that the user portrait is obtained. And comprehensive and accurate data acquisition and processing are realized.

A system and method for an integrated subsystem that identifies resale consumer goods available for sale whereby picture or text descriptions of items of firsthand consumer goods may be used as an input or search query where a user may enter size, style, and color parameters for clothing and fashion items. The plug-in uses a web crawler to maintain a database of resale consumer goods for sale and may isolate a single clothing item from a clothed person to perform a search query for resale versions of that item as well as provided improved specifically tailored results based on what the user is looking for at that specific time.

The invention discloses an automatic financial data collection system based on a web crawler technology. The system comprises an intelligent crawler collection module, a financial data processing module and a data collection module. The intelligent crawler acquisition module comprises a dynamic scheduling crawler cluster unit, a multi-format capture engine and a metadata marking unit; the dynamic scheduling crawler cluster configures a routing algorithm based on reinforcement learning, and dynamically adjusts an IP proxy strategy through an anti-crawling feature matrix; the multi-format capture engine is integrated with a PDF / HTML / JSON heterogeneous data analysis channel; the metadata marking unit generates a data source label by adopting multi-layer coding conversion; and the financial data processing module comprises a cross-format analyzer, a data cleaning pipeline and a structured storage unit. According to the scheme, through innovative designs such as multi-modal data analysis, self-adaptive anti-creeping confrontation, dynamic resource scheduling and intelligent association analysis, the core problems of low heterogeneous data integration efficiency, weak dynamic anti-creeping confrontation capability and the like are solved.

A system and methods for cybersecurity rating using active and passive external reconnaissance, comprising a web crawler that send message prompts to external hosts and receives responses from external hosts, a time-series data store that produces time-series data from the message responses, and a directed computational graph module that probes, scans, and fingerprints devices within a cyber-physical graph and analyzes the results as time-series data to produce a weighted score representing the overall cybersecurity state of an organization.

The invention discloses a multi-modal data management method based on a large model technology, and relates to the technical field of combination of artificial intelligence and a large data technology, and the method comprises the following steps: collecting multi-modal data of a target body through a web crawler, processing the collected multi-modal data, extracting features in the processed multi-modal data, and storing the features in a database; a multi-modal feature vector is formed through combination of an attention mechanism and a weighted summation method, association between different modal features is learned by using a multi-modal auto-encoder and a multi-modal Transform model, a multi-modal feature space is constructed, and a multi-modal feature vector is obtained. According to the method, a web crawler acquisition technology, a multi-modal feature extraction and fusion technology and a modern information technology are closely combined, multi-modal data such as texts, images, audios and videos are accurately captured, and high-quality multi-modal feature vectors and a unified multi-modal feature space are obtained; and real-time and comprehensive monitoring of the whole process from acquisition to processing and storage of the multi-modal data is achieved.

The invention provides a Web automatic penetration test method and device, electronic equipment and a storage medium, and belongs to the technical field of website vulnerability tests.The method comprises the steps that a web crawler collects target Web information to form a basic set; detecting vulnerability acquisition information; matching the adaptive script, and generating an intelligent load to verify vulnerability availability; constructing a knowledge graph to determine vulnerability association, and evaluating success rate and risk; planning an optimal attack route by using reinforcement learning, and determining attack depth and range, namely attack path risk features; and evaluating the risk by the dynamic scoring model, and generating a visual report containing detailed information, results and repair and improvement suggestions. According to the automatic penetration testing method, manual intervention is reduced in the whole process, dependence on professional experience is reduced, efficiency is improved, and the problems that traditional testing is tedious and low in efficiency are solved.

The invention provides an intelligent data cleaning method in a big data environment, and belongs to the technical field of big data processing, and the intelligent data cleaning method in the big data environment comprises the steps of data acquisition and preprocessing, intelligent detection and identification, intelligent cleaning strategy making, cleaning execution and verification, continuous optimization and monitoring and the like. Data are collected through a database connection tool, a file reading interface and a web crawler, and format conversion and standardization processing are carried out. Abnormal values and outliers are detected through a machine learning algorithm and statistical analysis, a cleaning strategy is made, and verification is executed. A feedback mechanism is established, and the data cleaning process is monitored in real time. The problems that a traditional data cleaning method depends on manpower, is low in efficiency and lacks flexibility and adaptability are solved, the data cleaning efficiency, accuracy and reliability are improved, the method adapts to big data diversity and dynamics, the data quality is improved, the decision risk is reduced, and the actual application requirement is met.

The invention discloses a marketing copywriting generation method and system, belongs to the technical field of information processing, can improve the timeliness and pertinence of marketing copywriting, and ensures that the marketing copywriting highly meets the specific requirements of users and the characteristics of target industries. Comprising the following steps: regularly crawling a plurality of target websites at a preset time interval based on a web crawler technology to obtain marketing hotspot data of each industry; constructing a marketing database according to the marketing hotspot data; obtaining a marketing copywriting demand of a user; according to a preset screening condition, screening out data matched with the marketing copywriting demand from a marketing database by utilizing a retrieval enhancement generation technology to obtain a first information set; converting the marketing copywriting demand into a first cue word, and converting the first information set into a second cue word; generating prompt text information according to the first prompt word and the second prompt word; and inputting the prompt text information into a preset language model to generate a marketing copywriting.

The invention discloses a personalized customized vi design method and system, and relates to the technical field of visual identification design, the design method comprises the following specific steps: S100, data acquisition: docking with an enterprise internal management system to obtain core data, using a web crawler tool to collect Internet related information, and using the web crawler tool to store the core data; according to the method, data in multiple aspects of industry attributes, brand positioning and color preference of target audiences of an enterprise are deeply analyzed by utilizing an artificial intelligence technology, highly intelligent color analysis and recommendation are realized, and the color matching degree of the industry, the color preference of the target audiences and multiple dimensions of color coordination correction factors are comprehensively considered. According to the method, the color scheme meeting the enterprise image and the market demand can be accurately recommended to the user, the intelligent color recommendation not only improves the design efficiency, but also ensures that the color scheme is highly matched with the enterprise positioning and audience preference, and the visual impact and brand recognition degree of the design are enhanced.

The invention relates to an intelligent visual crawler configuration system and method based on a large model, and the system comprises a data input module, an AI feature analysis module, a rule generation module, and a visual configuration module, the data input module is responsible for receiving a target website or webpage content, and obtaining an HTML source code or a dynamically rendered complete webpage structure; the AI feature analysis module is used for automatically analyzing structural features and semantic features of a webpage by utilizing a pre-trained AI large model; the rule generation module is used for automatically generating a crawling rule based on an analysis result, wherein the crawling rule comprises a page turning path, list area positioning and field extraction logic; and the visual configuration module is used for providing a visual user interface, displaying an analysis result and allowing a user to manually adjust the generated rule. The problems that an existing web crawler technology is low in efficiency and poor in adaptability in the aspect of webpage rule configuration are solved, the rule generation efficiency is remarkably improved through the system and method, the system and method are suitable for more unknown types of webpages, and the dynamic adaptation and continuous optimization capacity is achieved.

A wearable electronic medical device that stores initial baseline biometric data in memory and determines patient-specific thresholds for monitored biometric parameters. Biometric sensors detect changes in these parameters in response to therapeutic treatment or disease progression, and exceeded thresholds activate actions such as patient notifications, changes in treatment, and transmission of data. Trusted receivers receive suggested diagnoses and treatment options determined by an AI-powered web crawler. The treatment includes electrocuetical and pharmaceutical treatment, and the physiological change includes an indication of a change in the cardiovascular condition. The device determines patient-specific thresholds by applying statistical weighting to monitored biometric parameters and determining an acceptable data set.

The invention relates to the technical field of agriculture, in particular to a large-model-driven agricultural knowledge graph analysis method and system, and aims to realize standardized processing of multi-source heterogeneous data through a three-stage preprocessing process and combine with a field adaptive large-model training technology so as to realize the large-model-driven agricultural knowledge graph analysis method and the large-model-driven agricultural knowledge graph analysis method and the large-model-driven agricultural knowledge graph analysis system. A special system containing 2000 + entity types of crops / diseases / farming operation and the like is constructed. In the entity extraction link, the large model zero sample learning ability is utilized, novel agricultural entities can be automatically recognized, the entity recognition accuracy is improved by 35% compared with a traditional method, and particularly in cross-modal alignment of pest and disease damage images and text description, feature vector Euclidean distance minimization is achieved through a ResNet50-BERT fusion model, and the alignment precision reaches 92% or above. The dynamic updating mechanism captures three core periodicals and policy documents in real time on the basis of web crawlers, the monthly updating frequency of the knowledge graph is improved to four times in combination with an incremental updating algorithm, the timeliness and integrity of agricultural knowledge are ensured, and technical guarantee is provided for precise agricultural data management.

The invention relates to the technical field of information management, and discloses a project cloud platform data management method based on a process market, and the method comprises the steps: obtaining process demand information inputted by an enterprise user, and obtaining an enterprise name of the enterprise user; crawling enterprise process information through a web crawler tool, and performing multi-level classification to construct a process market platform; obtaining the business scope and the enterprise scale of the enterprise user according to the enterprise name, and determining the enterprise type of the enterprise user according to the business scope; performing keyword extraction on the process demand information input by the enterprise user to determine a target keyword of the process demand information; and matching a demand flow list in a flow market platform according to the enterprise type, the enterprise scale and the target keyword, and pushing the demand flow list to the enterprise user. According to the method, an intelligent process market platform is constructed, keyword extraction and analysis are carried out on demands of enterprise users, automatic aggregation and accurate matching of process resources are realized, and the process adaptation accuracy is remarkably improved.

The invention belongs to the technical field of hotel operation virtual simulation analysis, and particularly discloses a hotel operation virtual simulation analysis method and system and a storage medium, and the method comprises the steps: integrating construction data, historical operation logs and other multi-source data, and carrying out the multi-dimensional correlation analysis through the simulation initial data; the problem of decision deviation caused by the fact that a traditional method depends on artificial experience is solved, and operation bottleneck problems such as a service efficiency threshold value and resource allocation conflicts can be accurately recognized through a three-dimensional simulation model, so that high-precision data support is provided for optimization decision. Competition pricing data are collected in real time through a web crawler tool, competition influence factors are monitored, the check-in waiting loss probability is analyzed, in combination with environment monitoring data, competition opponent strategy change and market change data can be synchronized in real time, dynamic pricing weight parameters are loaded in a three-dimensional simulation model, the customer waiting duration and the loss rate are reduced, and the user experience is improved. And the technical efficiency of service resource scheduling is improved.

The invention relates to a data processing method and system, in particular to a vocational education multi-source data federation treatment method and system, and belongs to the technical field of vocational education informatization and big data. Then constructing a knowledge graph in a triple form and mapping the knowledge graph to a multi-dimensional tensor space; secondly, multi-source data compatibility is analyzed and evaluated through topological features, and feature fusion maintaining topological invariants is executed; finally, knowledge reasoning and service are provided based on a quantum probability field model, the complex relation expression ability is improved through tensor representation, the heterogeneous data fusion quality is guaranteed through topology durability, the uncertainty in knowledge service is processed by introducing the quantum probability theory, and the reliability of the knowledge reasoning and service is improved. The problems of multi-source heterogeneous, complex relation, insufficient service accuracy and the like of vocational education data are effectively solved, and a comprehensive data federation governance solution is provided for the vocational education field.

The invention provides a personalized safety risk early warning method and system integrated with rapid help calling and a medium, and relates to the technical field of safety early warning, and the method comprises the steps: collecting and obtaining an information platform data set through employing a crawler technology, and carrying out the safety risk recognition, and obtaining a risk region feature set and a risk region position set; calculating to obtain a real-time distance information set of the risk area position set and the current position information of the target user; performing multi-dimensional risk assessment on the risk area feature set and the real-time distance information set to obtain risk area hidden danger factors; and constructing a dynamic safety early warning strategy library, carrying out strategy analysis on the hidden danger factors of the risk area, determining a target safety risk early warning strategy, and carrying out dynamic safety early warning on the target user. The technical problems of incomplete safety risk identification and insufficient early warning accuracy caused by limited data sources in the prior art are solved, and the technical effects of improving the real-time performance and accuracy of risk early warning and the flexibility of emergency response are achieved.

The invention discloses an automatic ransomware collection and analysis method and system based on a large language model in the technical field of network security, and the method comprises the steps: constructing a target data source, collecting a first sample of a ransomware, and capturing the first sample which is actually triggered in an enterprise-level network environment and context behavior data of the first sample from the first sample; an automatic crawler system is established by adopting an open source web crawler framework to establish real-time connection with a malicious code platform, the popularity of a first sample on the malicious code platform is analyzed, and an optimization task is set. According to the method, the natural language processing capacity and the automation technology of a large language model are combined, efficient capture, accurate analysis and quick response of the samples are achieved, an anti-virtualization detection mechanism of the blackmail virus is effectively coped with by implementing an anti-virtualization detection confrontation strategy, and an intelligent protection chain is constructed for network security.

The invention discloses an application method for classification and recognition of non-abandoned images, and the method comprises the steps: S1, obtaining an initial non-abandoned image dataset which comprises Shenembroidery and related non-abandoned images collected from a museum, a public dataset or a web crawler, guaranteeing the image to be clear and complete, carrying out the format standardization of the image, uniformly adjusting the image to be 224 * 224 in resolution, and storing the image in a jpg format, through data enhancement methods such as horizontal overturning, rotation, brightness and contrast adjustment and the like, the image diversity is increased, and about 15 enhanced versions are generated from each original image. The method comprises the steps of data set preparation and enhancement, model improvement, transfer learning, training process, software design and test and the like. According to the method, spatial pyramid pooling and transfer learning are introduced, so that the recognition accuracy of the model on non-abandoned images is effectively improved, and a test result shows that the recognition accuracy of the improved model reaches 98.65%. The Shen embroidery image recognition software designed in the invention also shows high recognition precision, and has wide popularization and application values.

The invention provides a real estate data cleaning method based on the Internet, and belongs to the technical field of data cleaning, and the method comprises the steps: collecting real estate data from a plurality of Internet data sources by using a web crawler technology, monitoring the collection process based on a monitoring tool to generate monitoring data, and setting a transmission set for each piece of real estate information under each data source; respectively counting the type distribution of each house type; performing anomaly identification on the real estate information, and when the anomaly type is related to transmission anomaly, analyzing the update type of the corresponding transmission set to perform anomaly update on the corresponding real estate information; when the exception type is irrelevant to the transmission exception, analyzing an input behavior corresponding to the internet data source, searching and identifying similar information by using an identification field corresponding to exception information, and performing error updating; based on distribution of all types and authority of each internet data source, all house types are arranged in sequence, and format conversion and classified storage are carried out. And the quality and availability of the real estate data are effectively improved.

The invention provides a method and a system for constructing a time sequence knowledge graph in the water conservancy field based on a T-ATLOP model, which are applied to the water conservancy field, mainly comprise two key levels, namely a data layer and a knowledge extraction layer, and realize semi-automatic time sequence knowledge graph construction. The method comprises the following steps: carrying out batch document processing on water conservancy census data by utilizing a large language model through a prompt project, crawling water conservancy consultation of water conservancy halls of all provinces and cities by a web crawler, constructing a time sequence knowledge graph data set HE-DOCU through two-step method labeling of large language model assistance and heuristic strategy labeling, and carrying out a knowledge extraction layer part on the basis of an ATLOP model. A new model T-ATLOP is provided for knowledge extraction. And finally, storing the knowledge tetrad of the knowledge extraction layer by using a graph database to form a water conservancy field time sequence knowledge graph. According to the method, the more perfect water conservancy field time sequence knowledge graph is constructed by fully utilizing the water conservancy information, so that the accuracy and the intelligent level of querying related knowledge by hydrological employees can be improved.

The invention relates to the technical field of safety production management and intelligent analysis, in particular to a production safety accident key cause link identification method based on a complex network, which comprises the following steps of: acquiring structured production safety accident report data from a safety production supervision platform by utilizing a web crawler technology; a TF-IDF algorithm is combined with an industrial standard term library to optimize Jieba word segmentation, and a chi-square statistical method is adopted to calculate the correlation between each risk keyword and an accident category. By fusing text mining, statistical analysis and complex network theories and combining field dictionary optimization word segmentation, multi-accident model adaptation and triple centrality index analysis, the problems that a traditional method is single in analysis dimension, depends on subjective experience and is insufficient in technology integration are solved; efficient utilization of unstructured data, scientific layering of risk factors and accurate identification of key cause links are realized, and objectivity and effectiveness of safety management of a complex production system are effectively improved.

There are provided systems and methods for a guided web crawler for automated identification and verification of webpage resources. A service provider, such as an online transaction processor, may provide a guided web crawler and / or resources for such crawler for execution by computing devices of users. Users may load different pluggable modules to the guided web crawler, which are associated with specific web crawling tasks. Web crawling tasks may correspond to identification and verification of webpage resources on a webpage, such as a location, placement, use of, and / or number of appearances of the resource. The web crawler may use code from the pluggable module being executed to parse and / or crawl webpage data for a webpage and identify requested resources. Thereafter, the guided web crawler may automate resources to use, display, and / or interact with the identified and verified resource.

The invention discloses a historical soil survey text-based land utilization information extraction method and system. The method comprises the following steps of: establishing a plant name dictionary of three land utilization types of paddy fields, dry lands and other forest lands in a research area; reading a vegetation description text Vi of a sampling point from a historical soil survey text of a research area, judging whether the Vi contains plants in a plant name dictionary or not, if so, assigning a land utilization type corresponding to the plant to a related variable of a species name, assigning a coverage value corresponding to full coverage to the coverage, storing the coverage in a species information list, and if not, assigning the coverage value corresponding to the full coverage to the species information list; otherwise, using a regular expression and an information extraction method to obtain names and coverage degrees of main species, and storing the names and the coverage degrees in a species information list; for each element in the species information list, extracting a corresponding life type from a preset plant website by using a web crawler technology; and inferring the land utilization type of the sampling point according to the life type and the corresponding coverage degree. The method is high in efficiency.

The present application relates to a data processing method and system, in particular to a vocational education multi-source data federal governance method and system, belonging to the field of vocational education informatization and big data technology, which first acquires and preprocesses vocational education multi-source data through network crawler and OCR technology; then constructs a knowledge graph in the form of triplets and maps it to a multi-dimensional tensor space; then evaluates the compatibility of multi-source data through topological feature analysis, and performs feature fusion that maintains topological invariants; finally, based on a quantum probability field model, knowledge reasoning and services are provided, the present application uses tensor representation to improve the expression ability of complex relationships, ensures the quality of heterogeneous data fusion through topological persistence, and introduces quantum probability theory to handle uncertainty in knowledge services, effectively solving the problems of multi-source heterogeneous vocational education data, complex relationships and insufficient service accuracy, and providing a comprehensive data federal governance solution for the vocational education field.

Disclosed in the present disclosure is a fast evaluation method of site seismic liquefaction hazard based on artificial intelligence algorithm: establishing a historical seismic and site information database, the database including a demand input module, a web crawler module, a data processing module, and a database module connected in sequence; a neural network model performs prediction to acquire a post-earthquake site dominant frequency; and, on the basis of the post-earthquake site dominant frequency, acquiring a site earthquake damage degree and seismic performance parameters. The present disclosure solves the problem of fast evaluating post-earthquake site earthquake damage and site seismic performance parameters, and can rapidly evaluate the site liquefaction or softening earthquake damage degree and site seismic performance parameters in given earthquake conditions.

The invention discloses a multi-dimensional hotspot extraction method based on data analysis, relates to the technical field of data processing, and solves the problem that features and hotspot feature sets of text information data, image video data and voice information data are difficult to extract respectively after data is automatically captured in real time and preprocessed by using a web crawler technology. The hot events are difficult to identify by using a clustering algorithm, and analysis and evaluation of time dimension, space dimension and emotion dimension are performed on the hot events; and the development trend of the hot event is difficult to predict. According to the method, text information, image video and voice information data are automatically captured through a web crawler, multi-modal fusion is carried out after preprocessing and hot spot feature extraction, a hot spot event is identified by utilizing a clustering algorithm, and multi-dimensional analysis, evaluation and prediction are carried out on the hot spot event.

The invention discloses a fine-grained crop disease image classification method based on image text fusion, and belongs to the technical field of image data processing, and the method comprises the following steps: 1, carrying out the image data collection in an agricultural website through a web crawler method, carrying out the image data selection through an agricultural field expert, and carrying out the image data collection; taking the selected image as a crop disease image; 2, extracting an image high-dimensional feature map by using a pre-trained image feature extractor; 3, weakening interference information in the image by using a scab extraction algorithm, and highlighting scab features; step 4, a pre-trained BERT model is utilized to embed text description for each image, visual and text information is fused into a unified feature space, and the weight of each piece of modal information is dynamically adjusted; and step 5, performing accurate classification of crop diseases by using the fused features. According to the method, various disease types can be effectively distinguished under various conditions.