UI Layout Image Conversion for Automated Code Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual generation of user interfaces is time and resource intensive, often requiring multiple separate machine learning models and manual intervention, and fails to intelligently incorporate brand style and contextual information.
Innovation Solution
A single model pipeline using a generative adversarial network (GAN) automates UI design by understanding layout, color, and text properties, synthesizing missing values, and generating multiple designs from a single set of inputs, including a residual neural network (ResNet) to convert pixel layouts to text-based representations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple separate machine learning models are used for UI generation, then the system can handle different aspects of UI design (layout, color, text), but the device complexity and time requirements increase significantly
Solution Approach 1:
The patent merges multiple separate machine learning models into a single unified model that can generate UI layouts, colors, and text simultaneously. This consolidation maintains the ability to handle different UI aspects while reducing system complexity and eliminating the need for multiple separate model executions.
Solution Approach 2:
The unified model is designed with multi-functionality to perform various UI generation tasks that previously required separate specialized models. It can process different input types and generate diverse UI components in a single execution, making the system more versatile without increasing complexity.
2Reliability
If manual intervention is used in UI generation, then quality control and brand style incorporation are maintained, but time and resource requirements increase
Solution Approach 1:
The system incorporates self-service mechanisms where the unified model automatically adjusts and refines its own outputs based on brand style guidelines and contextual information. This self-adjustment capability maintains quality control and reliability while eliminating the need for manual intervention and reducing time requirements.
3Ease of manufacture
If existing UI code with large amounts of HTML tags and handwritten CSS is used, then the user interface can be implemented, but the complexity of updates and maintenance increases
Solution Approach 1:
The patent replaces manual mechanical coding processes with an automated unified model that generates UI code automatically. This substitution eliminates the need for developers to manually write and maintain complex HTML and CSS, reducing code complexity while maintaining ease of implementation through automated generation.
Data Source
AI summary
Techniques are disclosed for automatically converting a layout image to a text-based representation. In the disclosed techniques, a server computer system receives a layout image that includes a plurality of portions representing a plurality of user interface (UI) elements included in a UI design. The server computer system transforms, via executed of a trained residual neural network (ResNet), the layout image to a text-based representation of the layout image that specifies coordinates of bounding regions of the plurality of UI elements included in the UI design, where the text-based representation is usable to generate program code executable to render the UI design. The disclosed techniques may advantageously automate one or more portions of a UI design process and, as a result save time and computing resources via the execution of an image to text-based conversion ResNet machine learning model.