58 results about "Web scraping" patented technology

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses an agent-based webpage information processing method, device and equipment and a medium. And distributing the data to a capturing agent to obtain webpage original data. According to the method, model context information is generated by extracting webpage source information and release time information, page structure representation is constructed, key information nodes are positioned, structured data are generated, then multi-scene decision data are generated in combination with the structured data and the model context information in the analysis process, and a decision result is output through a decision reasoning module. By constructing a uniform data processing flow, webpage capture, context generation, structured analysis, analysis processing and decision reasoning are organically linked, efficient acquisition and intelligent processing of webpage information are realized, and timeliness and accuracy of decision are improved.

The invention discloses a method and a system for intelligently acquiring and analyzing webpage merchant information. Automatic acquisition of the merchant information is realized through five main steps of intelligent webpage capture, intelligent field identification, accurate address information analysis, intelligent merchant grouping and automatic page discovery. According to the method, a traditional rule-based extraction method is abandoned, information such as merchant names, phone numbers and addresses in webpages is automatically recognized by using context semantic analysis and mode recognition technologies, and accurate association grouping of fields is realized through a dynamic weight multi-dimensional similarity calculation method; and the merchant information can be continuously discovered and collected by using an automatic page discovery mechanism without manually specifying a collection path. The technical bottlenecks that specific rules need to be written for different websites, webpage structure changes are difficult to deal with, multi-merchant information grouping cannot be processed and the like in a traditional method are broken through, the method adapts to various webpage structure changes, and the automation degree, accuracy and efficiency of merchant information collection are greatly improved.

Systems and processes are disclosed for enhancing cybersecurity and optimizing software repositories through integration of web crawling, web scraping, feature engineering, and advanced machine learning algorithms to detect phishing attempts, prevent account takeover fraud, and identify unused code in repositories. The system collects and refines data from various sources, including transaction logs, customer databases, device details, external data sources, and historical fraud data, to build comprehensive datasets. Feature engineering creates new, meaningful features from the refined data, which are used to train and evaluate machine learning models. The best-performing models are deployed in production to monitor incoming communications and transactions in real-time, flagging suspicious activities and optimizing codebases. This processing ensures timely detection and prevention of security threats while maintaining efficient software development processes. Robust protection is provided against evolving cyber threats and enhances software performance and security through continuous learning and adaptation.

An AI-powered HR recruitment platform system that aggregates job postings in real time using API and web scraping technologies, classifies roles using NLP, and ranks candidates using trust-weighted AI models based on blockchain-verified qualifications, thereby achieving increased transparency, data protection, and verifiable fairness over traditional HR systems. The components can be used individually or in any combination without departing from the scope of the invention.

An artificial intelligence (AI)-powered system for policy evaluation, bias mitigation, and regulatory compliance. The system retrieves AI policies from multiple sources, including APIs, document parsing, and structured web scraping, ensuring real-time updates. It applies machine learning and natural language processing (NLP) to generate unbiased policy summaries and detect cybersecurity, ethics, and compliance gaps. A policy optimization engine analyzes governance trends and refines recommendations based on adoption feasibility. The system includes a sandbox testing framework that simulates regulatory and societal impacts to assess policy effectiveness. This AI-driven approach enhances fairness, transparency, and cybersecurity resilience in AI governance, enabling policymakers to proactively align with global regulatory frameworks and mitigate policy risks.

A smart spider and web scraping system with custom templates, artificial intelligence for custom attribute extraction, artificial intelligence for providing quick fixes for machine learning extracted web page data, and direct HTTP request extraction without crawling.

An intelligent, predictive customer loyalty system for social media marketing, consisting of: a data acquisition unit configured to collect user interaction data, demographic attributes, and sentiment analysis metrics from multiple social media platforms via platform-specific application programming interfaces (APIs), web scraping techniques, and direct data ingestion pipelines; a real-time engagement prediction engine comprising a machine learning model trained on historical engagement patterns, sentiment fluctuations, and content performance metrics, with the model dynamically predicting user responses, determining the optimal engagement time, and prioritizing interaction opportunities based on a probabilistic ranking technique; a real-time engagement prediction processing unit configured to generate and optimize social media posts, ads, and interactive media using a neural network-based natural language generation (NLG) model, where the model processes contextual user behavior and trending topics to dynamically adapt marketing content; an AI-powered automation unit for interactive engagement, comprising a deep learning-based chatbot engine integrated into a dialogue management system, with the chatbot engine analyzing the user's intentions, sentiment development, and engagement history to enable predictive, context-aware conversations with users; a cloud-connected processing unit that includes an AI inference accelerator configured to perform deep learning calculations for real-time engagement prediction units, personalized content recommendations, and chatbot response generation, with the processing unit integrated into a distributed computing framework to optimize inference latency.

Disclosed herein are system, method, and computer program product embodiments for improving web scraping technology by using machine learning to generate parsing expressions. A system receives a request to identify an element in a first document at a target web page. The system downloads and modifies the first document by adding an index value as an attribute to a tag for the element. A query is submitted to a large language model (LLM), including the modified first document, a description of the element, and a request asking the LLM to identify the element based on the description. The system obtains, from the LLM, the index value assigned to the element. The system generates an expression defining a path to the element in the first document using the index returned by the large language model. The system downloads a second document, and parses data of a second element using the expression.

A system for automated real-time publication processing may comprise a data collection module configured to automatically gather data from multiple external sources. A data analysis module may be configured to process the gathered data using artificial intelligence models. A data visualization module may be configured to generate interactive visual representations of the processed data. An update control module may be configured to automatically update published content and maintain version history with timestamps. The data collection module may utilize application programming interfaces and web scraping tools to gather data from government databases and real-time data feeds. The data analysis module may employ machine learning libraries to process and analyze the collected data. The data visualization module may use visualization tools to create interactive charts and graphs. The update control module may use Git-based version management and automated scheduling scripts to implement updates and record modification dates.

A smart spider and web scraping system with custom templates, artificial intelligence for custom attribute extraction, artificial intelligence for providing quick fixes for machine learning extracted web page data, and direct HTTP request extraction without crawling.

The invention discloses a vertical domain intelligence analysis method based on a large language model. The method comprises the following steps: receiving a natural language analysis demand of a user; based on pre-constructed vertical domain specific task planning knowledge, intelligently decomposing the demand into a subtask list corresponding to a preset dimension by using a large language model, and semantically matching a proper information acquisition tool for each subtask to generate a structured task plan; the system schedules and executes tool sets of network search, webpage capture, internal query and the like according to a plan, and performs cross verification and cleaning on collected fragmented information; and inputting the verified data into a large language model for integration analysis, and optionally carrying out multi-round iteration self-evaluation optimization to automatically generate a high-quality vertical domain intelligence analysis report. According to the method, full-process and high-intelligence automatic processing of information analysis in the vertical field from demand understanding to report generation is realized, and the method has excellent planning capability, process interpretability, result reliability and excellent modular expansibility.

The invention discloses a webpage data extraction method based on a large language model, which comprises the following steps of: generating an Xpath sequence webpage grabber by utilizing the large language model, and processing diversified and variable network environments through a two-stage framework: in the first stage, inversely checking and removing HTML (Hypertext Markup Language) noise by utilizing LLM (Language Language Model) information extraction capability, and in the second stage, inversely checking and removing HTML (Hypertext Markup Language Model) noise; self-adaptive generation of an Xpath action sequence is carried out according to the hierarchical structure of the HTML; according to the combination of an external evaluation mechanism and a local evaluation mechanism of LLMs, a plurality of Xpath action sequences generated on different webpages in one stage are integrated, and a general grabber specific to a website is generated. According to the method, a baseline method is always exceeded under zero sample setting, higher efficiency is shown in large-scale webpage information extraction tasks, the method can quickly adapt to different website and task requirements, dependence on LLMs is reduced when similar tasks are processed, and therefore the efficiency of processing a large number of webpage tasks is improved.

The invention relates to an automatic method for technical assistance in the correction of vulnerabilities of a computer system, where the aforementioned method comprises at least the following steps: a) receiving information relative to computer system vulnerabilities by text, voice, file input or through data exchange; b) launching tools based on artificial intelligence algorithms and machine learning to understand the information and requests entered relative to vulnerabilities; c) performing parsing and text mining of the information received and process it through intelligent data matching with information relative to the computer vulnerabilities present in a database or through interaction with external online vulnerability databases; d) in case of unsatisfactory results, starting an extended online search by web scraping using deep learning algorithms and unsupervised machine learning; e) updating an internal vulnerability database with the additional information found; and f) generating a vulnerability report accompanied by relative remediation.

This application relates to a method, system, and computer device for preventing web scraping. The method includes: when a client sends an access request to a target website, determining whether the target website has configured anti-scraping measures based on preset anti-scraping information; if the target website has configured anti-scraping measures, injecting an SDK into the access response, generating a first access response, and returning the first access response to the client; the access response is generated by the target website based on the access request; the client asynchronously submits its runtime information to a human-machine interface verification unit based on the first access response; when the client sends an access request to a subpage of the target website, determining whether the interface corresponding to the subpage is a protected interface; if the interface corresponding to the subpage is a protected interface, the human-machine interface verification unit performs human-machine interface verification based on the client's runtime information; if the human-machine interface verification result is successful, an access request is sent to the subpage. This method can prevent the target website from being accessed by malicious programs.

To provide a system.SOLUTION: The system includes: means for collecting financial economy information by using a public API or a web scraping technology on the Internet; means for shaping the collected data and normalizing the data; means for storing the shaped and normalized data into a database; means for searching a database for data according to a request from a user; and means for sending the acquired data to a browser of the user.SELECTED DRAWING: Figure 1

A method and a system for generating a labeled benchmark dataset are disclosed. The method includes obtaining a plurality of sources related to a thin section using web scraping and extracting a plurality of images from the plurality of sources related to the thin section, the plurality of images including a plurality of thin section images and a plurality of non-thin section images. Further, the method includes determining the plurality of thin section images from the plurality of extracted images and generating a classification of the plurality of thin section images based on a given classification criteria. The geological thin-section based machine learning models is trained based on the generated classification of the plurality of thin section images and a wellbore drilling plan is generated based on the geological thin-section based machine learning models.

The present invention generally relates to the field of taxonomy identification. Identifying product taxonomy from a product abbreviation is currently performed manually and consumes lot of time and effort. Hence, embodiments of present disclosure provide an automated method for identification of product taxonomy from product abbreviation. First, a brand name of product abbreviation is predicted using a Large Language Model (LLM) and a brand list. Then, possible expansions of the abbreviation are generated based on the brand name using acronym expansion dictionary. Among the generated possible expansions, a relevant one is identified using the LLM. Later, web scraping and web search techniques are applied on the relevant expansion to obtain associated top k matches of a supergroup, a product group and module of the predicted relevant expansion. Finally, product taxonomy is predicted based on the top k matches using a LLM augmented taxonomy classification by Retrieval Augmented Generation.

A web scraping system configured with web scraping countermeasure resolution technology. The system comprises a configuration manager, a browser stack configured as a web browser client, a custom solver comprising a web scraping countermeasure solver; an application programming interface (API) gateway server operatively connected with the configuration manager and is configured to obtain a browser stack configuration and a session strategy from the configuration manager, and a session analysis server comprising a response analyzer configured to process a response from the target website to the target webpage request to solve a web scraping countermeasure challenge from the target website and provide an antibot solution to the custom solver.

The embodiment of the invention relates to a big data enterprise environment public service system, and the system comprises a data collection module which is used for processing unstructured data through a webpage capturing algorithm and a multi-source AP I calling mechanism, and obtaining enterprise environment information; the data processing module is used for dynamically adjusting the task priority and the acquisition frequency of data acquisition by monitoring the change of a data source and a user query record in real time; the user interface module is used for receiving request information from a plurality of users and returning corresponding results according to the enterprise environment information; the request information comprises a Web browser, a mobile application, a desktop client and an AP I calling request; and the website function module is used for receiving the request information sent by the user interface module and carrying out certificate query, enterprise evaluation and point exchange according to the Web browser, the mobile application, the desktop client, the AP I calling request and the enterprise environment information.

The invention discloses a price comparison method based on webpage capture and text similarity. The method comprises the following steps: step 1, constructing a Scrioy framework tool to obtain information of each website, and constructing a database; 2, carrying out vectorization processing on the large-scale text data, and converting the text data into a digital form capable of being calculated and analyzed; step 3, constructing a cosine similarity algorithm; and 4, designing a user interaction interface. According to the invention, through the technical combination of Scrapy dynamic crawling, TF-ID vectorization, cosine similarity matching and interactive UI, three major pain points of data lag, inaccurate matching and result redundancy of a traditional price comparison tool are solved, and localization optimization is carried out for the Chinese market.

A computer-implemented data processing method of validating legitimacy of a plurality of social media followers of a selected social media account owner, comprising steps, carried out by a social media follower scrubber tool, of: receiving an uploaded spreadsheet from a user, the spreadsheet including results of a web scraping operation, where a web scraping tool has been used to scrape data regarding the plurality of social media followers of the social media account owner selected by the user, where the spreadsheet has a plurality of rows, with each row representing one of the plurality of followers and a plurality of columns, with each column representing a characteristic feature related to the plurality of followers; and presenting the user with a drag and drop graphical user interface functionality allowing the user to rearrange and rename the columns of the uploaded spreadsheet in accordance with a native spreadsheet format.

Disclosed herein are system, method, and computer program product embodiments for improving web scraping technology by using machine learning to generate parsing expressions. A system receives a request to identify an element in a first document at a target web page. The system downloads and modifies the first document by adding an index value as an attribute to a tag for the element. A query is submitted to a large language model (LLM), including the modified first document, a description of the element, and a request asking the LLM to identify the element based on the description. The system obtains, from the LLM, the index value assigned to the element. The system generates an expression defining a path to the element in the first document using the index returned by the large language model. The system downloads a second document, and parses data of a second element using the expression.

This invention introduces a server-based artificial intelligence-supported security and evacuation system designed to protect oil wells against threats such as warfare, terrorism, and chemical attacks The system comprises four main functions: detection of a warfare or terrorism atmosphere, rapid shutdown of the oil well, personnel evacuation, and detection of chemical weapon attacks. Within the server system, various technologies are utilised, including web scraping, Support Vector Machines (SVM), Convolutional Neural Networks (CNN), YOLO object detection, Text-to-Speech (TTS), Natural Language Processing (NLP) algorithms, as well as physical components such as Metal-Oxide Sensors (MOS) and Electrochemical Sensors. This integration ensures early threat detection, prompt response, and minimisation of casualties.

A method includes receiving a request from a client web browser, generating a first instruction responsive to the request, modifying header information in the first instruction to produce a first header-modified instruction, forwarding the first header-modified instruction to a first target website, receiving a first response from the first target website, modifying, using a man in the middle (MITM) proxy, the first response to produce a modified first response, and sending a result to the client web browser responsive to the modified first response.

Provided herein are system, apparatus, device, method and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for caching media resources during a scraping operation. Web resources needed by webpage are stored in a cache that is used by multiple browsers that are scraping the webpage. When an unexpired entry for the web resource is present in the cache, a browser retrieves the web resource and cache instead of making a request from the webpage. This offers a technological improvement of reducing the traffic burden on proxy servers needed to forward the scraping requests and responses.

1. The name of the design product: webpage data scraping graphical user interface for electronic devices. 2. The use of the design product: an electronic device. 3. The design points of the design product: in the graphical user interface. 4. The picture or photo that best indicates the design points: front view. 5. The use of the graphical user interface: for accessing applications that allow scraping or capturing webpage data, such as accessing webpage scraping APIs or using artificial intelligence assistants. 6. The human-computer interaction mode of the graphical user interface: it can be interacted through touching or clicking the graphical user interface shown on the display screen of the electronic device or operating the buttons of the electronic device.

A web scraping system configured with web scraping countermeasure resolution technology.

The invention relates to a computer-implemented method for processing web scraping jobs, using a plurality of database servers (404A-404N) operating independently of one another and each being configured to manage data storage to at least a portion of a job database (314) that stores status of web scraping jobs while the web scraping jobs are being executed, the method comprising: - receiving a web scraping request from a client computing device (102); - when the web scraping request is received, selecting one of the plurality of database servers (404A-404N) that is identified as enabled in a table (1008); and sending a job description specified by the web scraping request to the selected database server (404A-404N) for storage in the job database (314) as a pending web scraping job; - repeatedly checking health of each of the plurality of database servers (404A-404N); and - based on the health checks, determine whether each of the plurality of database servers (404A-404N) are to be enabled or disabled in the table (1008).

A system is provided that alerts users to newly surfaced third-party digital listings matching predefined criteria, wherein said criteria are executed continuously or on-demand through distributed compute nodes. The system receives input for searching at least one online marketplace, the input received from a user device and providing description of a desired item of merchandise and criteria associated with the item. The system also launches web scraping of select online marketplaces in search of the item and continuously monitors the select online marketplaces using Python-based headless scrapers. The system also parses and normalizes data gathered during the web scraping with listings processed as they appear in near real time, identifies items of the data matching the received input, and formats and pushes alerts to the user device describing the identified items. The criteria initially comprise keywords, minimum and maximum prices, and selected online marketplaces to be searched.

An automated device for cross-border e-commerce and a method for controlling same are disclosed. The automated device according to one embodiment of the present invention comprises: a database that stores data; and a processor that controls the automated device, wherein the processor may, when a preset word representing a popular search term is identified while scraping a web page, extract keywords searched at a preset frequency or higher and store the keywords in the database, search a shopping platform using the extracted and stored keywords, store a shopping mall URL address searched on the shopping platform in the database, and store product data of the shopping mall in the database using the stored shopping mall URL address.