Generic GUI Visualization Encoding for Client Applications
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies for capturing and reporting information about runtime sessions of client applications have limitations, such as incomplete reporting, limited scope of information, and privacy and bandwidth concerns associated with screen recordings or screenshots.
Innovation Solution
A method implemented by a client device to encode a generic visualization of a client application's GUI during runtime sessions, involving identifying GUI elements, generating a dataset with data structures encoding generic representations of these elements, and transmitting this dataset to a back-end platform for reproduction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If screen recordings or screenshots are used to capture GUI information, then complete visual information is obtained, but privacy and bandwidth concerns arise
Solution Approach 1:
The patent extracts only the essential visual information from the complete screen recording or screenshot. Instead of transmitting the entire image data, it identifies and extracts key GUI elements (buttons, text fields, icons) and their properties (position, size, type), thereby obtaining complete visual information about the GUI structure while eliminating privacy concerns by not capturing sensitive content.
Solution Approach 2:
The patent creates a simplified copy or representation of the GUI rather than using the original screen capture. It generates a dataset that encodes the GUI structure using standardized data structures that represent GUI elements abstractly, preserving the visual information needed for analysis while avoiding the transmission of actual image data that could contain sensitive information.
2Loss of information
If screen recordings or screenshots are used to capture GUI information, then complete visual information is obtained, but bandwidth consumption increases
Solution Approach 1:
The patent extracts only the essential visual information from the complete screen recording or screenshot. Instead of transmitting the entire image data, it identifies and extracts key GUI elements (buttons, text fields, icons) and their properties (position, size, type), thereby obtaining complete visual information about the GUI structure while significantly reducing bandwidth consumption by transmitting only structured data about GUI elements rather than raw image data.
3Reliability
If detailed GUI information is captured and transmitted, then effective evaluation and diagnosis is enabled, but data transmission volume increases
Solution Approach 1:
The patent segments the GUI into discrete, identifiable elements (buttons, text fields, icons, etc.) and represents each element using a standardized data structure. This segmentation allows for effective evaluation and diagnosis by enabling precise identification and analysis of specific GUI components and their interactions, while reducing data transmission volume by transmitting only the essential properties of each segmented element rather than the entire GUI as a single large image.
Data Source
AI summary
A client device installed with a client application having a graphical user interface (GUI) may be configured to (i) at a given time during a runtime session of the client application, identify a set of GUI elements within the GUI of the client application that is to be represented in a generic visualization of the GUI, (ii) generate a dataset that encodes the generic visualization of the GUI, wherein the generated dataset comprises a respective data structure for each GUI element in the identified set that encodes information for rendering a generic representation of the GUI element, and (iii) after generating the dataset that encodes the generic visualization of the GUI, transmit the generated dataset that encodes the generic visualization of the GUI to a back-end platform associated with a provider of the client application.