GUI Model Generation from Screen Images
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The development of graphic user interfaces (GUIs) for applications is labor-intensive and time-consuming, requiring manual effort from developers who need to understand design documents, programming languages, and specific platforms, making it tedious, costly, and requiring specialized expertise.
Innovation Solution
A method using machine learning-based techniques to analyze GUI screen images, detect and classify UI components, and generate a language- and platform-independent GUI model, which can be used to automatically produce code for implementing the GUI across different platforms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If manual coding is used to develop GUI from design documents, then developers can implement custom functionalities and appearances, but the process becomes labor-intensive and time-consuming
Solution Approach 1:
The patent uses optical character recognition (OCR) to create digital copies of text from design document images, and uses image processing to extract and replicate UI component layouts. This copying approach automates the translation from design visuals to coded GUI, dramatically reducing manual effort and development time while maintaining fidelity to the original design intent
Solution Approach 2:
The patent introduces an intermediary processing system that acts as a mediator between design documents and final GUI implementation. This intermediary automatically performs text extraction, UI component detection, and code generation, eliminating the need for developers to manually interpret design documents and reducing both labor intensity and development time
2Adaptability or versatility
If developers manually write code to implement GUI, then custom application requirements can be met, but substantial manual effort and specialized expertise are required
Solution Approach 1:
The system copies UI component structures and text content from design documents directly into the generated code, preserving the original design's customization characteristics. This automated copying maintains adaptability to custom requirements while eliminating the complexity of manual coding and interpretation
Solution Approach 2:
The system performs self-service by automatically analyzing design documents, extracting UI specifications, and generating implementation code without requiring developer intervention for manual coding tasks. This reduces both the effort required and the complexity of the development process while maintaining customization capability
3Loss of information
If text content is extracted from GUI screen images, then text information can be recognized and used, but text detection accuracy may be affected by text color and background variations
Solution Approach 1:
The patent applies local quality adjustments by analyzing specific regions of the image and adapting processing parameters to local characteristics. Different text regions with varying colors and backgrounds receive customized processing approaches, improving extraction accuracy while managing complexity through localized rather than global processing
Solution Approach 2:
The system dynamically changes processing parameters such as threshold values and color space conversions based on the detected text and background characteristics. This adaptive parameter adjustment improves text extraction accuracy across varied visual conditions without requiring overly complex fixed-processing systems
4Extent of automation
If machine learning-based classifier is used to identify UI components, then automated component detection can be achieved, but training data preparation and model configuration are required
Solution Approach 1:
The patent performs preliminary actions by pre-training the machine learning classifier with curated UI component data and pre-configuring the detection system before actual GUI analysis. This upfront preparation enables automated detection to function effectively without requiring complex configuration during deployment, reducing operational complexity while maintaining high automation
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques are disclosed for automating GUI development from a GUI screen image that includes text information and one or more graphic user interface components. The GUI screen image is analyzed to extract text information and to identify the UI components included in the GUI screen. One or more text regions in the GUI screen image are detected and are replaced with placeholders. Images of one or more graphic user interface components in the GUI screen are extracted from the GUI screen image and are classified using a machine learning-based classifier. A GUI model is generated for the GUI based upon the classification results, locations of the one or more text regions, and locations of the one or more graphic user interface components. The generated model can then be used to generate one or more implementations (e.g., executable code) of the GUI, possibly for various platforms in different programming languages.