39 results about "Headless browser" patented technology

A method for automatic detection and remediation of security posture in web-applications using large vision models is fulfilled in the ongoing description by (a) initiating a headless browser as an agent to access an administrative section of a web-application, (b) enabling a pre-trained large vision model to navigate through a web user-interface of the web-application using a state transition graph, (c) determining subsequent navigation actions of the navigated web-user interface using screenshots of the navigated web-user interface with the large vision model, (d) detecting and analyzing a final state of navigation sequence of the administrative section to extract security attributes, (e) monitoring and collecting data associated with security posture of the web-application based on the security attributes, and (f) initiating automated corrective actions through a security posture remediation module upon identifying a security issue in the web-application.

The invention discloses a dynamic webpage content complete collection system and method based on a browser rendering engine. A CDP controller main module calls a command to start a headless browser process, and a network monitoring module registers a Network. Response Received event monitor; a DOM (Document Object Model) monitoring module starts a Mutation Observer to monitor the change of a DOM (Document Object Model) tree; the interactive simulation module preloads a mouse click and scroll event instruction set; capturing JSON / XML data returned by an interface in real time by using a network monitoring module, intercepting API response data, generating a DOM snapshot by using a DOM monitoring module, and extracting key structure changes; the DOM monitoring module provides a complete rendering HTML, the network monitoring module provides request metadata, and the interactive simulation module provides an operation time sequence log and packages generated complete webpage data into a WARC file. According to the method, document collection is upgraded to application-level interaction, and a technical implementation path is provided for collection scenes of dynamic data-intensive websites (such as financial public opinion monitoring and e-commerce price tracking) in which Internet resources are stored for a long time.

The invention provides a method and device for acquiring OG data of multiple social media platforms through links, and belongs to the technical field of data acquisition and processing. The method comprises the following steps: acquiring URL links from a plurality of social media platforms and common webpages as source data, wherein the source data correspond to webpage OG data lt; carrying out metagt; the label form is embedded into the lt of the HTML; carrying out headt; a label; performing validity check and standardization on the URL link, capturing webpage content by using an asynchronous HTTP (Hyper Text Transport Protocol) request library, and loading a complete HTML (Hyper Text Markup Language) on a dynamic webpage through a headless browser; the HTML positioning result is analyzed; carrying out metagt; tag: extracting an OG standard field and a platform extension field, and deducing a missing field from other HTML tags; mapping each platform OG field into a unified standard field; and outputting the OG data containing fields such as title and the like through the standardized API interface. According to the method, the multi-platform OG data acquisition efficiency, accuracy and safety are improved, good expansibility is achieved, and the cross-platform data integration requirement is met.

The invention relates to the technical field of artificial intelligence, and discloses a small language webpage adaptive collection method and device based on a large language model.The method comprises the steps that a target small language webpage is loaded and rendered through a headless browser, and a document object model tree structure is obtained; carrying out analysis and semantic annotation on the document object model tree structure based on a large language model, identifying dynamic content nodes and generating a self-adaptive content positioning rule; based on a self-adaptive content positioning rule, performing semantic vectorization on the at least two continuously collected webpage contents, and calculating semantic similarity; querying a preset frequency mapping rule according to the semantic similarity, dynamically adjusting the initiation frequency of a subsequent acquisition request, and generating an acquisition strategy; and executing the acquisition strategy to access the target webpage, and injecting an operation sequence for simulating human interaction behaviors in the acquisition process. Complete collection of the dynamic content of the small language webpage can be achieved, and continuity and stability of the collection process are guaranteed.

A computer-implemented method is provided that enables remote control of a headless browser. A message from a client to open a connection to a headless browser running on a device remote from the client is received. A command to provision the browser is sent to the headless browser. The command may configure the browser to appear as if the browser is controlled by a human. A command for controlling the headless browser is received from the client. The command is sent to the headless browser for execution. A response to the command is received from the headless browser. Finally, the response is forwarded to the client.

The invention discloses a Web honeypot automatic script identification and countering method based on a large language model and related equipment, and the method comprises the steps: obtaining a real IP address of an attack request through a WebRTC protocol; analyzing the attack request through a headless browser and a large language model, judging whether the attack request is an automatic script behavior or not, countering the attack request if the attack request is the automatic script behavior, and guiding the attack request to a honeypot if the attack request is not the automatic script behavior. The method can improve the accuracy of automatic script recognition, improves the honeypot stability, and can be widely applied to the technical field of network security.

The present disclosure relates to a method and device for implementing a headless browser cluster. The method comprises: assigning a corresponding accessible IP address to each server in a plurality of servers; wherein the plurality of servers form a cluster; installing a headless browser in each server; after receiving a headless browser starting instruction, starting a preset number of headless browser instances, the started headless browser instances forming a headless browser instance pool; after detecting that a headless browser instance acquisition instruction is received, acquiring a target headless browser instance available from the headless browser instance pool; acquiring a network socket endpoint address of the target headless browser instance; and sending the network socket endpoint address of the target headless browser instance to a client.

The invention discloses a website function accessibility evaluation method and device in an IPv6 single-stack environment, and belongs to the technical field of network function detection. The invention aims to solve the problem that in the prior art, website resource dependence and function accessibility cannot be comprehensively detected in an IPv6 single-stack environment. The method comprises the following steps: dynamically creating an isolated IPv6 single-stack network environment, starting a headless browser to capture all resource requests of a webpage, constructing a resource dependency graph, carrying out IPv6 reachability test on each resource, calculating a weighted accessibility score and generating a visual report. According to the method, the IPv6 single-stack user access behavior can be accurately simulated, the target website and all dependent resources of the target website can be comprehensively evaluated, a quantitative accessibility score and a visual dependency relationship visualization graph are provided, and accurate and operable diagnosis information is provided for website developers and operation and maintenance personnel.

The invention relates to the technical field of computers, and discloses a distributed document generation method based on a message queue and a headless browser, and the method comprises the steps: receiving a document generation request, packaging the document generation request into a standard task object, and verifying the validity of a rendering target; splitting an original task into sub-tasks, and calculating task fingerprints and time-carrying task fingerprints; the core parameters and the service data loads are separately stored; converting the effective sub-tasks into rendering instructions, and performing queue arrangement according to the calling mode of the sub-tasks; and taking a headless browser as a rendering processing unit, continuously consuming a rendering instruction from the message queue, carrying out webpage rendering, converting the rendered webpage into a document, and responding to a document generation request. The headless browser is used as a core rendering processing unit, high-fidelity restoration of webpage content is achieved, and the visual quality and accuracy of the generated document are improved.

The application provides a kind of active defense-oriented dynamic deception domain name honeypot generation method and system, it is related to network security active deception technical field, specific scheme: based on headless browser access and save the web page of real business, the type of the web page is analyzed and different strategies are recorded to obtain web variables, based on re-renderer component rendering the web variables are cloned to obtain simulation web page, the simulation web page is packaged and combined with known container template to build container image to obtain honeypot device container;Based on the honeypot device container, attack logs are collected and marked, and the optimal cycle is obtained based on the multi-objective optimization algorithm, and the cycle of the false domain name bound to the honeypot device container is dynamically adjusted based on the optimal cycle to obtain the domain name honeypot.The application improves the problem that domain name is difficult to adapt to the change of attacker strategy and is difficult to dynamically optimize the defense strategy according to the behavior of the attacker.

The application discloses a kind of methods for assisting large model networking query, the method is by receiving and encapsulating as standardization demand to parse user query request, matching dynamic residential proxy IP pool, IP dynamic rotation and load balancing mechanism are started, generate standardization collection task, realize multi-platform data scraping through multi-tool cooperation, anti-crawling avoidance and high concurrency scheduling, clean, double check and structure transformation are carried out to original data, add standardization field annotation, push structured data to large model and monitor data satisfaction degree;The application is through dynamic residential proxy IP pool and IP rotation mechanism, avoid regional restriction and anti-crawling interception, with headless browser, API identification technology, adapt to dynamic and waterfall page, improve collection integrity and efficiency, after cleaning, check and structure transformation, combined with automatic completion query condition and daily optimization mechanism, the accuracy, timeliness and stability of large model real-time query response are greatly improved.

The application discloses a front-rear end cooperative large PDF file intelligent generation method, relates to the technical field of computer software, and comprises the following steps: front-end rendering: the front end renders the work order data page by combining a headless browser with a Vue framework and an ElementUI component library, and dynamically displays structured work order details through a responsive data binding mechanism; data fragmentation: large work order data sets are split into N subtasks based on a dynamic fragmentation strategy; and small PDF files are generated in parallel: a containerized rendering cluster based on Kubernetes deployment management is used to allocate rendering tasks to each container; the small PDF files generated by the application through careful design with the help of the Vue and ElementUI frameworks are more attractive in terms of text layout, color matching and graphic display, and the large file after merging is more beautiful than the file generated by a traditional back end, thereby improving user experience.

The invention discloses a flow chart-based training data generation method, training data generation equipment and a medium, relates to the technical field of flow chart processing, and discloses a flow chart-based training data generation method, which comprises the following steps of: analyzing natural language input information based on a deep seeking model, obtaining a JSON object containing attributes between nodes and edges of the flow chart; constructing a hypertext markup language template for rendering the flow chart according to the random visual style, and embedding the JSON object into the hypertext markup language template to obtain a hypertext markup language file; generating an image file of the hypertext markup language file based on the headless browser; and determining annotation information of the flow chart according to the image file, packaging the annotation information to a JSON object to obtain a JSON annotation file, and inputting the JSON annotation file and the image file as training data into the deep learning model. On the basis, the model training effect is improved through diversified training samples.

The invention provides a proxy AI-based phishing webpage intelligent detection system and method, and the system is characterized in that the system comprises a link input module which is used for receiving a to-be-detected URL, and carrying out the preliminary filtering; the headless browser control module is used for controlling a headless browser to load a page and managing the life cycle of the page; the page content extraction module is used for extracting page information from the headless browser; manual analysis requirements are reduced through automatic interaction, the detection efficiency is improved, and the method is suitable for large-scale mail security scanning scenes; an interaction path and a decision process can be recorded, a transparent log is provided for security analysis, and subsequent optimization and auditing are facilitated.

The invention provides a data processing method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: in response to a received first operation request for a front-end page, generating a screenshot request and sending the screenshot request to middleware service; the screenshot request is generated based on screenshot operation on at least part of page information including third-party resource information in the front-end page; the middleware service starts a headless browser based on the screenshot request, and loads the front-end page through the headless browser; the middleware service performs screenshot on at least part of page information of the headless browser to obtain a target image, and sends the target image to the front-end page; and the front-end page generates target content corresponding to the first operation request based on the target image. According to the data processing method and device, the electronic equipment, the storage medium and the program product, screenshot of cross-domain resources can be achieved.

The embodiment of the invention discloses a signature method and device for a webpage snapshot, and the method comprises the following steps: loading a target webpage in a uniformly configured headless browser environment, automatically detecting and removing dynamic contents in the target webpage according to a preset dynamic element recognition rule, and generating purified webpage document object model (DOM) data; in the webpage state after the de-dynamic processing, respectively collecting standardized webpage DOM data and a webpage rendering image snapshot; hash abstracts of the webpage DOM data and the webpage rendering image snapshot are calculated respectively, and two abstract values obtained through calculation are combined to generate a webpage snapshot signature; and storing the webpage DOM data, the webpage rendering image snapshot, the webpage snapshot signature and the corresponding metadata information in a secure storage medium together. According to the embodiment of the invention, the consistency and reproducibility of webpage snapshot contents can be improved, the tamper-proof capability of webpage evidence storage is enhanced, and the accuracy and robustness of tamper detection are improved.

The invention relates to the technical field of testing, and provides a testing method and device, equipment and a medium, which can create at least one virtual display in a Docker container without a graphical interface so as to provide normal operation conditions for a test case needing a header browser. Circularly detecting the occupancy condition of the environment variable lock, and configuring the environment variable lock corresponding to each virtual display according to the detection result corresponding to each virtual display and the display identifier of each virtual display, each virtual display and the corresponding head browser are bound according to the environment variable lock corresponding to each virtual display, so that starting failure caused by the fact that threads of the virtual displays preempt the environment variables is avoided; according to the embodiment of the invention, the test case corresponding to the header browser is executed in parallel based on each virtual display, so that the problem that the header browser cannot run on a non-graphical interface to efficiently execute an automatic test can be effectively solved in a test scene needing to depend on real graphical interface interaction.

The invention discloses a business mail automatic processing method and system based on robot process automation. The method comprises the following steps: automatically analyzing a theme, a text and an attachment of a received mail through a natural language processing technology, extracting key information such as an enterprise ID (Identity), a chemical identifier and a mail type, and converting the key information into structured data; identifying task types according to the data and selecting corresponding automatic processes; a headless browser is used for simulating and logging in a third-party system, and automatic processing including attachment downloading, analysis, screenshot and multi-language translation is executed; the execution state is monitored in real time, the success rate and the average time are counted to optimize the performance, and meanwhile, various abnormities are captured and reports are generated to trigger the processing flow; and cleaning the resources after execution is finished, and generating a notification mail based on a result to send and archive. According to the scheme, full-process automation of mail processing is realized, and efficiency and accuracy are remarkably improved.

The invention discloses a front-end and rear-end collaborative intelligent generation method for a large PDF (Portable Document Format) file, which relates to the technical field of computer software and comprises the following steps: front-end rendering: a front end renders a work order data page through a headless browser in combination with a Vue framework and an Element UI (User Interface) component library, and dynamically displays structured work order details through a responsive data binding mechanism; data fragmentation: segmenting the large-scale work order data set into N subtasks based on a dynamic fragmentation strategy; generating small PDF files in parallel; deploying a managed containerized rendering cluster based on Kubernetes, and distributing a rendering task for each container; according to the method, the front end is elaborately designed by means of Vue and Element UI frames, the generated small PDF file is more attractive in character typesetting, color matching and graphic display, the attractiveness of the combined large file is far better than that of a traditional file generated at the rear end, and the user experience is improved.

The invention provides an active defense-oriented dynamic deception domain name honey point generation method and system, and relates to the technical field of network security active deception, and the specific scheme is as follows: accessing and storing a webpage of a real service based on a headless browser, analyzing the type of the webpage, and recording by adopting different strategies to obtain webpage variables; rendering the webpage variable based on a re-renderer component, then performing page cloning to obtain a simulation webpage, packaging the simulation webpage, and constructing a container mirror image in combination with a known container template to obtain a honey point equipment container; and collecting and marking an attacker request based on the honey point device container to obtain an attack log, obtaining an optimal period based on a multi-objective optimization algorithm, and dynamically adjusting the period of binding the false domain name to the honey point device container based on the optimal period to obtain the domain name honey point. The problems that the domain name is difficult to adapt to the strategy change of an attacker and the defense strategy is difficult to dynamically optimize according to the behavior of the attacker are solved.

The invention relates to the technical field of station group management methods, in particular to a station group management method based on low codes. According to the method, the scrolling behavior of the user is dynamically simulated through the headless browser, the problem that a traditional static crawler cannot capture JavaScript rendering content and hidden texts is solved in combination with the random speed and a waiting mechanism, and the scrolling speed is dynamically adjusted, for example, scrolling parameters are calculated based on DOM node density and AJAX request frequency, the randomness of human browsing is simulated, so that the user experience is improved. The method comprises the steps of effectively avoiding the recognition of an anti-crawling mechanism on a fixed rolling mode, ensuring that page contents are completely rendered through a display waiting mechanism, capturing a complete DOM structure and hidden texts including lazy loading resources and sensitive elements generated by an AJAX dynamic request, and avoiding the omission of key risk information; the technical problem of relatively low identification efficiency in the process of identifying the station group risk filliform in the prior art is solved.

The invention provides a cross-platform poster generation method and device, computing equipment and a computer readable storage medium. According to the method, a headless browser and Canvas drawing are combined and applied to the field of game poster automatic generation. Firstly, a headless browser is used for rendering at a server side, complete support of HTML5, CSS3 and Canvas is ensured, and the consistency of display effects of the design at all terminals is ensured; and in combination with a template engine and a Canvas API, poster generation driven by dynamic data is supported. A developer can use an HTML / CSS design template, different user and activity data can be automatically filled through placeholders, flexible layout is achieved, and hard coding of layer coordinates is not needed. All clients call the generation service through a unified back-end interface, and different templates or rendering parameters are automatically selected without repeated implementation for each platform. The scheme of the invention does not depend on a specific platform, is suitable for all environments supporting Chrome / Node. Js, and ensures the consistency of poster styles output by multiple terminals.

The invention discloses a webpage operation task execution method and system, a storage medium and computer equipment, and the method comprises the steps that a user interaction interface responds to a task creation instruction, creates a webpage operation task, and adds the webpage operation task into a task queue; monitoring the task queue by each service instance in a back-end service instance cluster, and when monitoring that a newly added webpage operation task exists in the task queue, judging whether to obtain an execution permission of the webpage operation task or not based on a self state and a preemption state of the webpage operation task; and for the target service instance which obtains the execution permission, the target service instance updates the state of the target service instance to a working state, starts a target headless browser in a task execution unit based on the webpage operation task, and executes a target webpage operation through the target headless browser until the webpage operation task is completed. And closing the target headless browser, and updating the state of the target headless browser to be an idle state.

The embodiment of the invention discloses a data quality inspection method and device, electronic equipment and a storage medium, and relates to the technical field of data quality intelligent inspection. The method comprises the steps that a headless browser technology, a User-Agent camouflage technology and window simulation are adopted to collect a first screenshot and a second screenshot corresponding to key indexes in a target billboard area of a first terminal and a target billboard area of a second terminal respectively; performing standardization processing and key area enhancement on the first screenshot and the second screenshot; the standardization processing comprises size normalization, color unification and background purification; calling a deep learning model to intelligently identify the processed first screenshot and second screenshot to obtain first terminal data and second terminal data, and comparing to obtain a difference value; and setting a difference threshold value, and automatically triggering an alarm mechanism when the difference value exceeds the difference threshold value. According to the invention, the problems of data inconsistency, difficulty in troubleshooting and lack of real-time monitoring in the prior art are solved.

Disclosed is phishing classifier that classifies a URL and content page accessed via the URL as phishing or not is disclosed, with URL feature hasher that parses and hashes the URL to produce feature hashes, and headless browser to access and internally render a content page at the URL, extract HTML tokens, and capture an image of the rendering. Also disclosed are an HTML encoder, trained on HTML tokens extracted from pages at URLs, encoded, then decoded to reproduce images captured from rendering, that produces an HTML encoding of the tokens extracted, and an image embedder, pretrained on images, that produces an image embedding of the image captured. Further, phishing classifier layers, trained on the feature hashes, the HTML encoding, and the image embedding, process the URL feature hashes, HTML encoding and image embeddings to produce a likelihood score that the URL and the page accessed presents a phishing risk.

This invention provides a method and system for determining the homology of illegal websites, relating to the field of network monitoring technology. The determination method includes: using a headless browser to realistically render the entry addresses of two websites to be determined, and recursively crawling all pages and link information within the corresponding two-level backlink range, starting from the entry address, to obtain the corresponding raw collected data; performing data normalization and cleaning on the raw collected data to obtain the set of second-level backlink domains and statistical data of the two websites to be determined; obtaining multi-feature data based on the set of second-level backlink domains and statistical data; and fusing the various multi-feature data to obtain homology probability data. This invention elevates homology determination from a single, superficial similarity judgment to a comprehensive quantitative assessment based on multiple evidence such as structure, attribution, and content, thereby significantly improving the accuracy of homology association for illegal websites disguised by methods such as domain switching, mirroring, and template reuse.

A computer-implemented method is provided that enables remote control of a headless browser. A message from a client to open a connection to a headless browser running on a device remote from the client is received. A command to provision the browser is sent to the headless browser. The command may configure the browser to appear as if the browser is controlled by a human. A command for controlling the headless browser is received from the client. The command is sent to the headless browser for execution. A response to the command is received from the headless browser. Finally, the response is forwarded to the client.

The invention discloses a webpage main body information extraction method and device and a medium, and relates to the technical field of Internet information processing. The method comprises the following steps: acquiring a document object model (DOM) tree of a webpage and visual rendering information of at least one HTML element through a headless browser; based on the visual rendering information, dividing the webpage into at least one visual block by using a visual separation algorithm, and calculating a visual feature score of the visual block; performing weighted fusion on the visual feature scores of the DOM nodes in the DOM tree and the structural feature scores of the DOM nodes to obtain comprehensive weight scores of the DOM nodes; and determining the DOM node with the comprehensive weight score greater than a preset weight threshold as a container node of the main body content, so as to perform information extraction on the container node and obtain main body information of the webpage. Therefore, webpage main body information extraction with high accuracy, good robustness and excellent efficiency can be realized.

The invention provides a map screen capture method and system, electronic equipment and a storage medium. The method comprises the steps that screen capture configuration parameters are acquired; in response to the screen capture instruction, performing multiple times of screen capture on the map by using the screen capture configuration parameters and the headless browser to obtain a plurality of map screen capture pictures; and splicing the plurality of map screenshot pictures to obtain a final map screenshot picture. According to the scheme, the map is subjected to multiple times of screen capture through the screen capture configuration parameters and the headless browser, and the multiple map screen capture pictures obtained through multiple times of screen capture are spliced to obtain the large-range final map screen capture picture.

The invention discloses a page translation content detection method, a related device and a detection tool, and relates to the field of artificial intelligence, the detection tool comprises a browser plug-in layer, a headless browser, a content extraction engine, a preset language model and a report generation module, and the method comprises the steps that the browser plug-in layer triggers page initialization loading and traverses a page; in the traversing process, page DOM change is triggered through the headless browser, and after change is completed, the current page is recorded; the content extraction engine extracts and cleans text content of a current page to obtain translated text content, the preset language model analyzes the translated text content and marks translated abnormal content to obtain a detection result, and the report generation module generates a translation detection report according to the detection result. According to the method, page DOM change is automatically triggered, detection is carried out after change is completed, automatic detection is carried out through the preset language model, omission is reduced, the translation detection report is output, translation abnormity can be conveniently determined, and the test efficiency is effectively improved.