444 results about "Screen capture" patented technology

The present disclosure provides techniques and solutions for automatically correcting software tests. When a test failure is detected, it is determined whether a screenshot or code associated with a second version of a software artifact includes a user interface element that has a semantically equivalent identifier to a user interface element in a screenshot or code associated with a first version of the software artifact. Identifying a semantically equivalent identifier can include determining a section of the user interface, in the screenshot of the code, of the first version of the software artifact where the user interface is located and searching the corresponding section of the user interface of the second version of the software artifact. A definition of the software test can be updated to reference the semantically equivalent user interface element.

A management node local to a customer site of a private communications network stores a model. The model is a compact version of a visual language model remote from the customer site. A first screen shot of a dashboard of telemetry data measured from the private communications network is accessed. A prompt is formulated comprising the first screen shot and information to adapt the model to the private communications network via few shot learning. The prompt is submitted to the model. An output is received from the model comprising textual information about anomalies or trends depicted in the first screen shot. The output is checked against data from a statistical model of the telemetry data, the statistical model being independent of the model. In response to the check being successful, an action is triggered to manage the private communications network according to the output.

The invention discloses a screen shooting traceability method, device and equipment and a readable storage medium, which are applied to the field of computer software, and comprise the following steps: encoding watermark information to obtain a watermark information sequence; embedding the watermark information sequence into the image through an R channel, a G channel and a B channel of RGB (Red, Green, Blue) to obtain an image with a watermark; and acquiring the shot image with the watermark image, and tracing to obtain watermark information based on the shot image. According to the method, the watermark is generated by using the RGB channel of the single pixel image, so that the visual concealment of the watermark image is improved, the difficulty in perceiving of human eye recognition is ensured, and the visual quality of a user is not influenced; the watermark embedding position is not influenced by an original image carrier, and watermark information can be hidden in any area of the image; and the watermark information recovery success rate is high.

The invention discloses a watermark imperceptible embedding and recovering method based on a deep learning network structure. Firstly, a mask guide watermark embedding scheme is designed, a mask generation module is constructed by using a residual dense feature extraction module and an attention mask generation module, and a watermark is adaptively guided to be embedded into an image texture rich area so as to improve the invisibility of the watermark. And secondly, constructing a watermark decoding network based on comparative learning, and by comparing a loss function, taking the decoding features of the same watermark image under different noise conditions as positive samples and taking the decoding features of different watermark images as negative samples so as to enhance the consistency of the decoding features, thereby improving the robustness of the watermark. The deep learning network structure can improve the robustness of the watermark in a real screen shooting scene, and has a huge application value in copyright protection and traceability tracking.

The disclosed techniques provide improved methods of operating system (OS) search. Users are enabled to search for documents, emails, presentations, content they entered into a web form, meetings they participated in, and other interactions they had with their computing device. To accomplish this, screenshots are periodically captured and indexed. Machine learning models are used to infer embeddings for visual elements of the screenshots and / or text extracted from the screenshots. A full text index of a relational database may also be populated with text extracted from the screenshots. The embeddings and full text index may then be used to retrieve screenshots in response to a user history query. For example, screenshots of embeddings within a defined distance of an embedding of the user history query may be selected. Query results from different embedding indices and relational databases may be ordered by applying different weights to different kinds of search scores.

A system and method includes connecting an interface device to a port of a vehicle to be in communication with an electrical system of the vehicle for monitoring the vehicle, and running a selected one of multiple available calibration and / or repair programs via the interface device to generate a calibration / repair data file of the vehicle, where the data / log file is generated in one of a plurality of possible native file formats. A report generation tool acquires screen captures and / or screen recordings to document the data / information displayed during the steps of a calibration / repair procedure. An evaluation tool extracts calibration / repair data from the data / log file, where the evaluation tool is configured to extract calibration / repair diagnostic data from data / log files in each of the plurality of possible native file formats, and outputs the calibration / repair diagnostic data to a database in a common format from which detailed calibration / repair diagnostic reports are generated.

Large language models (LLMs) and visual-language models (VLMs) are able to provide robust results based on specified formatting and organization. Although LLMs and VLMs are designed to receive natural language input, users often lack the skill, knowledge, or patience to utilize LLMs and VLMs to their full potential. By leveraging screen understanding, AI prompts (or “pills”) may automatically be generated for artificial-intelligence (AI) assistance and query resolution in a VLM / LLM environment. Using an image encoder, a current screenshot is processed into an image embedding and compared to text embeddings representing screenshot activities. By identifying the text embedding having the closest similarity to the image embedding, a screen activity being performed by the user may be determined. Suggested AI prompts (or “pills”) may then be generated in real-time to assist the user in performing the screen activity.

Systems, tools and methods to allow for the deterministic replay of nondeterministic events and testing of interactions with systems and interfaces. The system may record or retrieve user sessions corresponding to a first version of an application or user interface and replay the events of the user session with a second version of the application or user interface. The system may test updates to code by comparing screenshots or outputs generated by the second version against those generated by the first version to identify unintended differences or errors. The sessions used to test the updates may also be generated by a script or by a machine learning model trained on previous user sessions.

Disclosed are techniques for synthesizing large amounts of human-computer interaction data that is representative of real-world user data. An automated screenshot capture engine may cause an automated agent to use an application or a website in a manner designed to mimic real-world human-computer interaction. Screenshots are captured to record how a user might interact with the application. Metadata, such as window location and size, may be obtained for each screenshot. Screenshots and corresponding metadata may be automatically annotated with a large language model to indicate the context of the application and / or computer system when the screenshot was captured. Data created in this way may be used to validate AI-based software application features or to train (or retrain) a machine learning model that predicts human-computer interactions. Automated synthesis of training data significantly increases the scale of data that can be obtained for training while also reducing computing and financial costs.

Various systems and methods for contextual capture and processing of screen capture data are disclosed. An example method for screen capture data processing in a computing device may include: determining an active screen of the computing device based on a user interaction event; identifying a graphics rendering event associated with a software application presented in the active screen; identifying screen capture data in a buffer (e.g., of graphics processing circuitry such as a GPU) that corresponds to the graphics rendering event; and communicating a contextual screen update event via an application programming interface (e.g., a an API operated by a GPU driver). The receipt of this contextual screen update event can be used by an AI engine to control whether to perform contextual processing on particular frames of the screen capture data.

The invention provides an anti-screen-shooting robust text image watermarking method fusing edge attention gating and multi-scale cavity convolution, which comprises the following steps: step 1, designing a message processor for preprocessing a secret message, adding an expansion network ExpandNet and a channel attention SENet in the message processor, and carrying out edge attention gating and multi-scale cavity convolution on the expansion network ExpandNet and the channel attention SENet; inputting a secret message in a binary form and outputting a message feature map; step 2, constructing a text watermark framework based on edge attention gating and multi-scale cavity convolution fusion; adding a functional module in an encoder of the end-to-end watermark framework; 3, designing a noise layer, and performing simulation attack on the model in the watermark model training process; and step 4, designing an optimization mode of the loss function. According to the method, the robustness and imperceptibility of the watermark are remarkably improved.

A network security system that classifies webpages and uses the classification of the webpages to enforce relevant security policies is disclosed. To classify a webpage, a screenshot of at least a portion of the webpage is captured. An embedding engine generates a subject image embedding of the screenshot. The subject image embedding is classified by an image classifier that includes an index of training image embeddings each having a label of their classification and an associated approximate nearest neighbors model trained to identify the label of the closest training image embedding to the subject image embedding and a score representing their similarity. The webpage is classified based at least in part on the score and the label from the image classifier. The network security system applies security policies to requests from client devices that identify the webpage as its destination based at least in part on the webpage classification.

The invention discloses a method and equipment for realizing mouse and virtual key control based on an inertial measurement unit, and a computer readable storage medium. The method comprises the following steps: acquiring original three-dimensional motion data of the inertial measurement unit to generate a standard motion feature flow; forming screen key layout information according to the screenshot of the current active application program; determining an inferred running state of the current active application program to obtain an effective virtual key set containing function annotations; determining a current effective interaction mode according to the updated human-computer interaction situation information, the updated current application program inference operation state and a user input instruction; and selecting a mapping algorithm to process the standardized motion feature flow according to the currently effective interaction mode to generate at least one of a mouse cursor movement instruction, a mouse key instruction and a virtual key operation instruction. The method has the advantage that the operation efficiency and the user experience of IMU-based mouse control in multiple scenes are improved.

The invention provides a game real-time intelligent translation method and device based on interactive reinforcement learning. The method comprises the following steps: determining a corresponding screen capture mode according to current game picture information and user operation habit information; dynamically sensing the current game picture according to the screen capture mode; performing text recognition on the image data in the user attention area by using a context-enhanced image recognition model, and obtaining a recognition text according to a context knowledge base exclusive to the game scene and user operation historical information; inputting the recognition text, the corresponding image data and user operation historical information into a multi-modal fusion translation model to generate a target language text; and based on the corresponding relationship between the recognition text and the target language text, rendering the target language text on the original foreign language region in an augmented reality mode. According to the method, real-time, automatic and immersive translation of the game picture content is realized, a large number of continuous texts can be processed, and the translation result better conforms to a game scene.

The invention discloses a vision enhancement robust digital dark watermarking method based on deep learning, a storage medium and equipment, and belongs to the technical field of digital image processing and information security. According to the method, a visual enhancement module comprising a self-adaptive watermark region adjustment module and a frequency enhancement module is constructed, and a multi-region local loss training mechanism and a screen shooting simulation module are combined, so that the visual quality of a watermark image is improved, and meanwhile, the robustness of the watermark image for resisting physical attacks such as screen shooting is enhanced. The specific processing flow comprises the following steps: acquiring an original image and watermark information; embedding the watermark information into the image by using an encoder, wherein an embedded region is optimized by a self-adaptive region guide matrix generated based on the image edge and gray information; in the model training process, a noise layer simulating screen shooting physical distortion is introduced, and the area weight is dynamically adjusted according to residual error distribution in the later stage of training so as to focus and optimize the image quality; watermark information is extracted from an image which may suffer from an attack by using a decoder.

The invention relates to the technical field of network information security, in particular to a network information security monitoring method based on artificial intelligence, which comprises the following steps: step 1, interface information acquisition: monitoring a user interface in real time and capturing all information on the interface, including a desktop, an application window and document content, capturing a snapshot of the user interface at high frequency through a screen capture tool, extracting a video stream from the user interface and separating an audio stream, and extracting text information from an image by using an optical character recognition technology; according to the scheme, the character, image and sound information of the user interface is collected in real time, advanced artificial intelligence algorithms such as a convolutional neural network and potential semantic analysis are used for deep risk analysis, prohibited information grades are accurately divided, flexible and diverse processing modes such as pop-up window shielding are adopted according to the grades, and the prohibited information classification efficiency is improved. The network information security is comprehensively, accurately and effectively monitored, the defects of the traditional technology are overcome, and a powerful guarantee is provided for the health and stability of the network environment.

The invention discloses an automatic filling method and system based on YOLO and RPA, and the method comprises the steps: collecting a form interface image through screen capture, inputting the image into a YOLO model for processing, and recognizing form elements to generate bounding box information; reading data from an external data source, matching a corresponding relationship between the data and fields according to a preset mapping rule, verifying a data format, and obtaining a data set passing verification; using RPA to simulate user operation, inputting data passing verification into a form field, and formatting to obtain a filled form interface; checking fields required to be filled and data formats, and determining the validity of the form; simulating and clicking a submission button to complete form operation, and recording a submission result; and monitoring potential errors in the whole process, performing classified processing, recording detailed log information, and obtaining an optimized filling record. According to the method, the automation degree and accuracy of form processing are remarkably improved, the manual operation burden is reduced, the business process efficiency is optimized, and the method is suitable for complex form filling scenes.

A method for automatic detection and remediation of security posture in web-applications using large vision models is fulfilled in the ongoing description by (a) initiating a headless browser as an agent to access an administrative section of a web-application, (b) enabling a pre-trained large vision model to navigate through a web user-interface of the web-application using a state transition graph, (c) determining subsequent navigation actions of the navigated web-user interface using screenshots of the navigated web-user interface with the large vision model, (d) detecting and analyzing a final state of navigation sequence of the administrative section to extract security attributes, (e) monitoring and collecting data associated with security posture of the web-application based on the security attributes, and (f) initiating automated corrective actions through a security posture remediation module upon identifying a security issue in the web-application.

The invention relates to the technical field of data backup and storage, and discloses a universal chatting record backup and storage method and system.The universal chatting record backup and storage system comprises a multi-source data acquisition unit, an information storage unit, an operation recording unit and a supplementary acquisition unit. One-stop efficient acquisition is realized, and the data processing efficiency is improved; classified storage and structured processing are carried out according to data characteristics, original data storage and efficient query analysis are considered, and data structure changes are flexibly coped with; the operation recording unit completely records the whole operation process by using a screen recording and Hash algorithm, ensures that the operation record is tamper-proof, and provides authenticity guarantee for the chat record as evidence in scenes such as commercial disputes and lawsuit; the supplementary acquisition unit breaks through the limitation of a data interface by means of technologies such as screenshot and optical character recognition, realizes complete acquisition of chat records of a complex communication platform, and enhances the universality and adaptability of the system.

Provided is a system and method for dynamically adjusting a descriptive document with instructions for a user interface based on configuration settings and software version and / or release for a particular user and document data stored in a technical data object. The process ensures that the content of the descriptive document matches what a user will see on a screen when viewing the user interface. In one example, the method may include retrieving user interface configuration settings generating a simulated user interface that includes a plurality of user interface elements and formatting the user interface elements based on the user interface configuration settings, composing a window with text from a document and screenshot images of one or more formatted user interface elements from the simulated user interface corresponding to the text, and rendering the window via a display of a computing device.

A system and method automatically evaluate participant online testing and automatically provide feedback regarding test results of a participant. Automatic evaluation includes processing video frames, audio, timestamped screenshots, and initial test results of the participant answering multiple test questions and transcribing audio. The system and method utilize optical character recognition on the screenshots to generate a set of text for each screenshot and compare the set of text from each timestamped screenshot to determine which screenshots are associated with each test question. The system and method further determine time periods for each question, segment the transcribed audio, the screenshots, and the video frames by question and select a screenshot and a video frame for each test question from the segmented screenshots and video frames. The system and method utilize a first AI tool and a second AI tool to determine whether the participant demonstrated mastery for each test question.

The invention relates to a display device and a circle selection screenshot method. The display equipment comprises a display and a controller, and the controller is configured to perform screen capture on a display interface of the display in response to a received preset pressing instruction to obtain a screen image, control the display to display the screen image under the condition that a dynamic picture is displayed on the current interface, perform screen capture in the screen image according to a circled area, and display the screen image according to the circled area. And when the current interface displays the static picture, deleting the screen image, performing screenshot in the static picture according to the circled region to obtain the region image, and performing image recognition according to the character in the region image to display the object information of the character in the region image. By the adoption of the method, when the dynamic picture is displayed on the displayer, it can be guaranteed that the picture seen by a user when circling drawing is completed is consistent with the circling display interface, and therefore the accuracy of circling screenshot is guaranteed.

A machine learning model for extracting information from web pages is prepared. The preparation includes generating respective representations of a first set of web pages, including embeddings from screenshots and bounding boxes of the web pages for multi-phase training of the model. In a first phase of training of the model, multiple loss functions associated with respective prediction tasks are optimized jointly, including a markup language element prediction task and a prediction of overlap between bounding boxes and screenshot subdivisions. In a second phase of training, using output of a hidden layer of the model (whose parameters were learned in the first phase) as input, a loss function is optimized to achieve a target web information extraction objective. The trained version of the model is stored.

A text extraction method and a related device are provided. The method includes performing screen capture on a user interface of a first application, to obtain a screenshot of the first application, displaying the screenshot at a target layer, where the target layer is above a layer at which the user interface is located, and performing character recognition on the screenshot. When a character selection operation for the screenshot displayed at the target layer is detected, the method includes highlighting a selected character at the target layer, and when a drag operation for the selected character is detected, dragging the selected character to a second application.

The invention relates to the field of computers, and particularly provides an automatic translation method and system, based on OCR (Optical Character Recognition), firstly, a user selects a screen as a screen capture area, recognizes the capture area, translates into a target character, displays a translation result, monitors the selected area of the screen in the background, and translates the target character into the target character. Judging whether the content of the text area changes or not. Compared with the prior art, the OCR technology can be used, the translation area is selected as the black box model, and software can be simply Chinese when foreign language software source codes do not exist. A screen capture technology is used, monitoring and dynamic translation of the selected area can be started only by selecting the area once, and screenshot translation does not need to be performed all the time.

To provide an information processing device which acquires a screenshot posted by a user, so as to offer complementary information for complementing a content of the screenshot to viewers of the screenshot.SOLUTION: An information processing device includes an acquisition section, a generation section, and a provision section. The acquisition section acquires a screenshot posted by a user. The generation section generates complementary information for complementing a content of the screenshot on the basis of information on the screenshot and attribute information on viewers of the screenshot. The provision section provides the complementary information together with the screenshot to the viewers.SELECTED DRAWING: Figure 1

Aspects of the disclosure provide for mechanisms for dynamically generating at least one page using one or more screenshots. A method of the disclosure includes receiving a screenshot, identifying a plurality of fragments of the screenshot, wherein each fragment corresponds to one of: structural user interface (UI) elements or content of the screenshot, generating, for each fragment of the plurality of fragments, an entity, wherein the entity is a pre-defined structure representation of the fragment, and generating, based on the entities associated with the plurality of fragments, at least one page associated with the screenshot.

A system may identify editable elements on a graphical page displayed on a client device and display graphical indications showing the editable elements on the graphical page. In some instances, the system may receive a first user input selecting an element of the indicated editable elements. The system may receive a second user input modifying the first element and may modify code of the graphical page based on the second user input to change the appearance of the graphical page. In some instances, the system may receive a third user input requesting an image capture of the changed appearance of the graphical page. For example, the system may automatically remove the graphical indications and capture the graphical page with its changed appearance.

The invention provides a knowledge base construction method and device, equipment, a storage medium and a program product, and the method comprises the steps: carrying out the screen capture processing of a first graphical user interface of an application program in the operation process of the application program, and obtaining a screen capture image; acquiring class information corresponding to the first graphical user interface; and constructing a corresponding relationship between the screenshot image and the class information, and carrying out storage processing on the corresponding relationship to obtain a knowledge base of the application program. Based on the knowledge base of the application program constructed by the invention, for any type of information of the application program, a user can obtain the screenshot image corresponding to the type of information, so that the user is helped to understand the function effect realized by the type of information.

The invention discloses a server in-band data obtaining method, device and equipment and a storage medium, and relates to the technical field of servers, and the method comprises the steps: communicating with a management server in batches by means of a substrate management controller, obtaining screenshots in batches by means of a prefabricated guidable optical disc system, automatically extracting text information through a pre-trained optical character recognition model, and obtaining the in-band data of the server through the pre-trained optical character recognition model. And then the in-band data is automatically screened and output, so that the problem that batch processing cannot be performed due to the fact that manual single operation and recording are depended traditionally is solved, and efficient and automatic acquisition of the in-band data of the large-scale server cluster is realized.