368 results about "Screen capture" patented technology

The present disclosure provides techniques and solutions for automatically correcting software tests. When a test failure is detected, it is determined whether a screenshot or code associated with a second version of a software artifact includes a user interface element that has a semantically equivalent identifier to a user interface element in a screenshot or code associated with a first version of the software artifact. Identifying a semantically equivalent identifier can include determining a section of the user interface, in the screenshot of the code, of the first version of the software artifact where the user interface is located and searching the corresponding section of the user interface of the second version of the software artifact. A definition of the software test can be updated to reference the semantically equivalent user interface element.

The invention discloses a screen shooting traceability method, device and equipment and a readable storage medium, which are applied to the field of computer software, and comprise the following steps: encoding watermark information to obtain a watermark information sequence; embedding the watermark information sequence into the image through an R channel, a G channel and a B channel of RGB (Red, Green, Blue) to obtain an image with a watermark; and acquiring the shot image with the watermark image, and tracing to obtain watermark information based on the shot image. According to the method, the watermark is generated by using the RGB channel of the single pixel image, so that the visual concealment of the watermark image is improved, the difficulty in perceiving of human eye recognition is ensured, and the visual quality of a user is not influenced; the watermark embedding position is not influenced by an original image carrier, and watermark information can be hidden in any area of the image; and the watermark information recovery success rate is high.

The invention discloses a watermark imperceptible embedding and recovering method based on a deep learning network structure. Firstly, a mask guide watermark embedding scheme is designed, a mask generation module is constructed by using a residual dense feature extraction module and an attention mask generation module, and a watermark is adaptively guided to be embedded into an image texture rich area so as to improve the invisibility of the watermark. And secondly, constructing a watermark decoding network based on comparative learning, and by comparing a loss function, taking the decoding features of the same watermark image under different noise conditions as positive samples and taking the decoding features of different watermark images as negative samples so as to enhance the consistency of the decoding features, thereby improving the robustness of the watermark. The deep learning network structure can improve the robustness of the watermark in a real screen shooting scene, and has a huge application value in copyright protection and traceability tracking.

The disclosed techniques provide improved methods of operating system (OS) search. Users are enabled to search for documents, emails, presentations, content they entered into a web form, meetings they participated in, and other interactions they had with their computing device. To accomplish this, screenshots are periodically captured and indexed. Machine learning models are used to infer embeddings for visual elements of the screenshots and / or text extracted from the screenshots. A full text index of a relational database may also be populated with text extracted from the screenshots. The embeddings and full text index may then be used to retrieve screenshots in response to a user history query. For example, screenshots of embeddings within a defined distance of an embedding of the user history query may be selected. Query results from different embedding indices and relational databases may be ordered by applying different weights to different kinds of search scores.

A system and method includes connecting an interface device to a port of a vehicle to be in communication with an electrical system of the vehicle for monitoring the vehicle, and running a selected one of multiple available calibration and / or repair programs via the interface device to generate a calibration / repair data file of the vehicle, where the data / log file is generated in one of a plurality of possible native file formats. A report generation tool acquires screen captures and / or screen recordings to document the data / information displayed during the steps of a calibration / repair procedure. An evaluation tool extracts calibration / repair data from the data / log file, where the evaluation tool is configured to extract calibration / repair diagnostic data from data / log files in each of the plurality of possible native file formats, and outputs the calibration / repair diagnostic data to a database in a common format from which detailed calibration / repair diagnostic reports are generated.

Disclosed are techniques for synthesizing large amounts of human-computer interaction data that is representative of real-world user data. An automated screenshot capture engine may cause an automated agent to use an application or a website in a manner designed to mimic real-world human-computer interaction. Screenshots are captured to record how a user might interact with the application. Metadata, such as window location and size, may be obtained for each screenshot. Screenshots and corresponding metadata may be automatically annotated with a large language model to indicate the context of the application and / or computer system when the screenshot was captured. Data created in this way may be used to validate AI-based software application features or to train (or retrain) a machine learning model that predicts human-computer interactions. Automated synthesis of training data significantly increases the scale of data that can be obtained for training while also reducing computing and financial costs.

Various systems and methods for contextual capture and processing of screen capture data are disclosed. An example method for screen capture data processing in a computing device may include: determining an active screen of the computing device based on a user interaction event; identifying a graphics rendering event associated with a software application presented in the active screen; identifying screen capture data in a buffer (e.g., of graphics processing circuitry such as a GPU) that corresponds to the graphics rendering event; and communicating a contextual screen update event via an application programming interface (e.g., a an API operated by a GPU driver). The receipt of this contextual screen update event can be used by an AI engine to control whether to perform contextual processing on particular frames of the screen capture data.

The invention provides an anti-screen-shooting robust text image watermarking method fusing edge attention gating and multi-scale cavity convolution, which comprises the following steps: step 1, designing a message processor for preprocessing a secret message, adding an expansion network ExpandNet and a channel attention SENet in the message processor, and carrying out edge attention gating and multi-scale cavity convolution on the expansion network ExpandNet and the channel attention SENet; inputting a secret message in a binary form and outputting a message feature map; step 2, constructing a text watermark framework based on edge attention gating and multi-scale cavity convolution fusion; adding a functional module in an encoder of the end-to-end watermark framework; 3, designing a noise layer, and performing simulation attack on the model in the watermark model training process; and step 4, designing an optimization mode of the loss function. According to the method, the robustness and imperceptibility of the watermark are remarkably improved.

A network security system that classifies webpages and uses the classification of the webpages to enforce relevant security policies is disclosed. To classify a webpage, a screenshot of at least a portion of the webpage is captured. An embedding engine generates a subject image embedding of the screenshot. The subject image embedding is classified by an image classifier that includes an index of training image embeddings each having a label of their classification and an associated approximate nearest neighbors model trained to identify the label of the closest training image embedding to the subject image embedding and a score representing their similarity. The webpage is classified based at least in part on the score and the label from the image classifier. The network security system applies security policies to requests from client devices that identify the webpage as its destination based at least in part on the webpage classification.

The invention discloses a method and equipment for realizing mouse and virtual key control based on an inertial measurement unit, and a computer readable storage medium. The method comprises the following steps: acquiring original three-dimensional motion data of the inertial measurement unit to generate a standard motion feature flow; forming screen key layout information according to the screenshot of the current active application program; determining an inferred running state of the current active application program to obtain an effective virtual key set containing function annotations; determining a current effective interaction mode according to the updated human-computer interaction situation information, the updated current application program inference operation state and a user input instruction; and selecting a mapping algorithm to process the standardized motion feature flow according to the currently effective interaction mode to generate at least one of a mouse cursor movement instruction, a mouse key instruction and a virtual key operation instruction. The method has the advantage that the operation efficiency and the user experience of IMU-based mouse control in multiple scenes are improved.

The invention discloses a vision enhancement robust digital dark watermarking method based on deep learning, a storage medium and equipment, and belongs to the technical field of digital image processing and information security. According to the method, a visual enhancement module comprising a self-adaptive watermark region adjustment module and a frequency enhancement module is constructed, and a multi-region local loss training mechanism and a screen shooting simulation module are combined, so that the visual quality of a watermark image is improved, and meanwhile, the robustness of the watermark image for resisting physical attacks such as screen shooting is enhanced. The specific processing flow comprises the following steps: acquiring an original image and watermark information; embedding the watermark information into the image by using an encoder, wherein an embedded region is optimized by a self-adaptive region guide matrix generated based on the image edge and gray information; in the model training process, a noise layer simulating screen shooting physical distortion is introduced, and the area weight is dynamically adjusted according to residual error distribution in the later stage of training so as to focus and optimize the image quality; watermark information is extracted from an image which may suffer from an attack by using a decoder.

The invention discloses an automatic filling method and system based on YOLO and RPA, and the method comprises the steps: collecting a form interface image through screen capture, inputting the image into a YOLO model for processing, and recognizing form elements to generate bounding box information; reading data from an external data source, matching a corresponding relationship between the data and fields according to a preset mapping rule, verifying a data format, and obtaining a data set passing verification; using RPA to simulate user operation, inputting data passing verification into a form field, and formatting to obtain a filled form interface; checking fields required to be filled and data formats, and determining the validity of the form; simulating and clicking a submission button to complete form operation, and recording a submission result; and monitoring potential errors in the whole process, performing classified processing, recording detailed log information, and obtaining an optimized filling record. According to the method, the automation degree and accuracy of form processing are remarkably improved, the manual operation burden is reduced, the business process efficiency is optimized, and the method is suitable for complex form filling scenes.

A method for automatic detection and remediation of security posture in web-applications using large vision models is fulfilled in the ongoing description by (a) initiating a headless browser as an agent to access an administrative section of a web-application, (b) enabling a pre-trained large vision model to navigate through a web user-interface of the web-application using a state transition graph, (c) determining subsequent navigation actions of the navigated web-user interface using screenshots of the navigated web-user interface with the large vision model, (d) detecting and analyzing a final state of navigation sequence of the administrative section to extract security attributes, (e) monitoring and collecting data associated with security posture of the web-application based on the security attributes, and (f) initiating automated corrective actions through a security posture remediation module upon identifying a security issue in the web-application.

The invention relates to the technical field of data backup and storage, and discloses a universal chatting record backup and storage method and system.The universal chatting record backup and storage system comprises a multi-source data acquisition unit, an information storage unit, an operation recording unit and a supplementary acquisition unit. One-stop efficient acquisition is realized, and the data processing efficiency is improved; classified storage and structured processing are carried out according to data characteristics, original data storage and efficient query analysis are considered, and data structure changes are flexibly coped with; the operation recording unit completely records the whole operation process by using a screen recording and Hash algorithm, ensures that the operation record is tamper-proof, and provides authenticity guarantee for the chat record as evidence in scenes such as commercial disputes and lawsuit; the supplementary acquisition unit breaks through the limitation of a data interface by means of technologies such as screenshot and optical character recognition, realizes complete acquisition of chat records of a complex communication platform, and enhances the universality and adaptability of the system.

A system and method automatically evaluate participant online testing and automatically provide feedback regarding test results of a participant. Automatic evaluation includes processing video frames, audio, timestamped screenshots, and initial test results of the participant answering multiple test questions and transcribing audio. The system and method utilize optical character recognition on the screenshots to generate a set of text for each screenshot and compare the set of text from each timestamped screenshot to determine which screenshots are associated with each test question. The system and method further determine time periods for each question, segment the transcribed audio, the screenshots, and the video frames by question and select a screenshot and a video frame for each test question from the segmented screenshots and video frames. The system and method utilize a first AI tool and a second AI tool to determine whether the participant demonstrated mastery for each test question.

The invention relates to a display device and a circle selection screenshot method. The display equipment comprises a display and a controller, and the controller is configured to perform screen capture on a display interface of the display in response to a received preset pressing instruction to obtain a screen image, control the display to display the screen image under the condition that a dynamic picture is displayed on the current interface, perform screen capture in the screen image according to a circled area, and display the screen image according to the circled area. And when the current interface displays the static picture, deleting the screen image, performing screenshot in the static picture according to the circled region to obtain the region image, and performing image recognition according to the character in the region image to display the object information of the character in the region image. By the adoption of the method, when the dynamic picture is displayed on the displayer, it can be guaranteed that the picture seen by a user when circling drawing is completed is consistent with the circling display interface, and therefore the accuracy of circling screenshot is guaranteed.

A machine learning model for extracting information from web pages is prepared. The preparation includes generating respective representations of a first set of web pages, including embeddings from screenshots and bounding boxes of the web pages for multi-phase training of the model. In a first phase of training of the model, multiple loss functions associated with respective prediction tasks are optimized jointly, including a markup language element prediction task and a prediction of overlap between bounding boxes and screenshot subdivisions. In a second phase of training, using output of a hidden layer of the model (whose parameters were learned in the first phase) as input, a loss function is optimized to achieve a target web information extraction objective. The trained version of the model is stored.

A text extraction method and a related device are provided. The method includes performing screen capture on a user interface of a first application, to obtain a screenshot of the first application, displaying the screenshot at a target layer, where the target layer is above a layer at which the user interface is located, and performing character recognition on the screenshot. When a character selection operation for the screenshot displayed at the target layer is detected, the method includes highlighting a selected character at the target layer, and when a drag operation for the selected character is detected, dragging the selected character to a second application.

The invention relates to the field of computers, and particularly provides an automatic translation method and system, based on OCR (Optical Character Recognition), firstly, a user selects a screen as a screen capture area, recognizes the capture area, translates into a target character, displays a translation result, monitors the selected area of the screen in the background, and translates the target character into the target character. Judging whether the content of the text area changes or not. Compared with the prior art, the OCR technology can be used, the translation area is selected as the black box model, and software can be simply Chinese when foreign language software source codes do not exist. A screen capture technology is used, monitoring and dynamic translation of the selected area can be started only by selecting the area once, and screenshot translation does not need to be performed all the time.

To provide an information processing device which acquires a screenshot posted by a user, so as to offer complementary information for complementing a content of the screenshot to viewers of the screenshot.SOLUTION: An information processing device includes an acquisition section, a generation section, and a provision section. The acquisition section acquires a screenshot posted by a user. The generation section generates complementary information for complementing a content of the screenshot on the basis of information on the screenshot and attribute information on viewers of the screenshot. The provision section provides the complementary information together with the screenshot to the viewers.SELECTED DRAWING: Figure 1

The invention discloses a server in-band data obtaining method, device and equipment and a storage medium, and relates to the technical field of servers, and the method comprises the steps: communicating with a management server in batches by means of a substrate management controller, obtaining screenshots in batches by means of a prefabricated guidable optical disc system, automatically extracting text information through a pre-trained optical character recognition model, and obtaining the in-band data of the server through the pre-trained optical character recognition model. And then the in-band data is automatically screened and output, so that the problem that batch processing cannot be performed due to the fact that manual single operation and recording are depended traditionally is solved, and efficient and automatic acquisition of the in-band data of the large-scale server cluster is realized.

The invention provides a method for carrying out universal non-intrusive automatic operation on equipment with a man-machine interface by using a multi-mode large model agent. The method comprises the following steps: step 1, signal acquisition: acquiring an output signal of target equipment by edge equipment; step 2, data uploading: the edge computing device reads a video stream of the edge computing device or converts the original signal acquired in the step 1 into an analyzable digital signal, then divides the digital signal into independent screenshots and preprocesses the screenshots, and then uploads the processed screenshots and an identification result to a server; step 3, instruction generation and issuing: the server generates a subsequent operation instruction after analyzing the data, and returns the subsequent operation instruction to the edge computing device; 4, the edge computing device converts the operation instruction into a specific HID signal and sends the specific HID signal to the target device, and automatic operation closed loop is completed. The method has the beneficial effects that the target equipment can be automatically controlled in a mode which is completely the same as that of a human operator, and the method can be applied to any electronic equipment which can be operated by human beings.

A graph-forming process generates a graph having nodes that represent a plurality of previously captured screenshots. The graph-forming process relies on a plurality of machine-trained models to identify edges between pairs of the nodes. The edges represent relationships among the screenshots. The graph-forming process then trains a graph neural network (GNN) based on the graph. The training produces a plurality of target embeddings associated with respective nodes in the graph. A retrieval process retrieves a previously captured screenshot using the plurality of target embeddings. The retrieval process involves adding a new node to the graph that represents the query and using the GNN to produce a query embedding associated with the new node. The retrieval process then finds at least one target embedding that matches the query embedding and retrieves a screenshot associated with the matching target embedding.

To provide a generation system or the like that can generate a user-desired screen capture without failure.SOLUTION: A screen capture instruction is accepted on the basis of a user's input operation. A screen is captured at instruction timing at which the instruction is accepted. It is determined whether or not the screen capture satisfies specified conditions. If it is determined that the specified conditions are not satisfied, content information based on the instruction timing is changed. The screen capture is generated on the basis of the changed content information.SELECTED DRAWING: Figure 3

A method of gathering information about a process being performed by a user of a computing device having application programs and separate monitoring software installed thereon is described. The user performs the process by performing actions via a sequence of application user interface (UI) screens. The method comprises capturing screenshots of at least some application UI screens in the sequence to obtain a sequence of application screenshots; processing the sequence of application screenshots using multiple different trained machine learning (ML) models to extract a corresponding sequence of application UI screen metadata, the multiple different trained ML models including an object detection model and a text recognition model; using the sequence of application UI screen metadata to generate a representation of the process; and storing the representation of the process on the computing device and / or transmitting the representation of the process to another device different from the computing device.

Effective Retrieval-Augmented Generation (RAG) pipelines face significant challenges when processing domain-specific technical documents that have diverse content types like text, figures, equations, and tables. To address this challenge, a context-oriented RAG system can be implemented for various domain-specific applications. The RAG system can include a lightweight, two-stage architecture to facilitate contextual understanding: a content analysis and enrichment pipeline for structured metadata extraction and a query processing pipeline for context-aware retrieval. In some cases, tabular data is processed using a dual-stream approach: semantically via text and visually via screenshots. The embedding vectors and the metadata can be stored in a visual data management system. The RAG system, utilizing the visual data management system, can answer questions and precisely retrieve technical information in a way that can preserve structural relationships and semantic connections across different modalities.

The invention discloses a device and method for automatically identifying a PC application scene and recommending a corresponding AI tool, and the method comprises the steps: capturing a current screen of a PC, and transmitting the current screen to an AI expansion screen device connected with the PC; using a multi-mode AI large model to identify and obtain the content of the screenshot, and determining the current application scene of the PC; automatically recommending a corresponding AI tool according to the application scene, and displaying a function button of the AI tool; and in response to the operation on the function button, executing corresponding AI processing, and displaying a processing result on a touch screen of the AI extended screen equipment. According to the method and the device, the most suitable AI tool can be automatically analyzed and activated according to the real-time change of the PC screen content, manual operation of a user is not needed, the working efficiency is improved, and the user experience is improved.

The invention discloses a data sharing method and related equipment, for a sender, sender equipment can provide a system-level file encryption control sharing function without depending on the fact that the sender equipment must install a certain application, and the sender can select a file to be shared from a system through the sender equipment and set an authority control strategy. Sending to a receiver device in one or more different ways; for the receiver, the receiver equipment can provide a data management and control function, the receiver can receive and open the shared file through the receiver equipment, and the system can dynamically adjust the permissions of different functions of the application access system based on the permission control strategy set in the file. For example, a receiving party can be controlled to be incapable of performing operations such as copying, screen capturing and printing on the file, and an attacker cannot perform the operations on the file even if attacking an application for opening the file. In this way, the scene support degree, safety, eco-friendliness and the like are all improved, and the user experience is better.

The invention discloses a clinical test sensitive data automatic shielding method and system based on a mobile terminal. The method comprises the following steps: capturing a clinical test data image through a mobile equipment camera, and automatically detecting and cutting a document frame to obtain an effective area; classifying and identifying the images as paper or screenshots by adopting the trained convolutional neural network, and applying a corresponding processing strategy; extracting text and position information by using an OCR (Optical Character Recognition) technology, and inputting the fine-tuned BERT model to combine context semantic analysis and position a sensitive entity; performing irreversible shielding operation on the sensitive content according to the position information to ensure data desensitization; and storing or outputting the desensitized image. By implementing the method provided by the invention, the effective shielding of sensitive data can be realized on the mobile equipment with limited resources while the efficient and accurate processing requirement is met, so that the strict privacy protection requirement is met, the data security and privacy can be ensured, and meanwhile, the method has high operation convenience and real-time performance.

Detection of a malicious application or website by a transaction processing application includes inputting a transactional request having a transactional data record and a screenshot; sending the input data record and input screenshot to a backend controller; requesting to a prompt selector, a string comprising a feature-extraction prompt; sending the input screenshot and the received prompt string to a Large Vision Model, LVM; receiving a string having risk classification features from said LVM; verifying the received string by a format parser; if the received string fails the verification, requesting by the backend controller, a string having a feature-extraction prompt which explicitly mentions format parsing compatibility, and repeating the preceding steps; sending the received string to a risk classification model for providing a risk classification; sending the risk classification to the backend controller; determining if the application or website is determined as malicious, and accepting or rejecting the transactional request accordingly.