UI Code Synthesis from Presentation Graphical Features

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

There is a gap between UI designers and developers due to different skills, tools, and priorities, leading to challenges in translating designed graphical user interfaces into functional code, especially in applications with limited display areas like mobile devices.

Innovation Solution

A method and system that extracts graphical features from presentation documents using machine-learning algorithms, matches them with candidate features in a library, and synthesizes user interface code through an API, bridging the gap between designers and developers by automatically generating code consistent with the graphical layout.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If manual translation from design to code is used, then developers can implement UI functionality, but productivity is reduced due to time-consuming manual coding and communication overhead

Engineering Contradiction:

ImproveUI development speedVSAvoidTime for manual coding and communication

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system enables self-service by automatically generating UI code from design documents without requiring manual developer intervention. The code synthesis engine extracts graphical features from presentation documents and automatically generates corresponding UI code, allowing the system to serve itself rather than requiring human developers to manually translate designs to code.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent introduces an intermediary code synthesis engine that acts as a mediator between design documents and functional UI code. This intermediary automatically translates graphical features from presentation documents into executable UI code, eliminating the need for direct manual translation by developers and reducing communication overhead between designers and developers.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If automated code generation is implemented, then productivity improves, but accuracy of translating design intent into code may deteriorate

Engineering Contradiction:

ImproveUI development speedVSAvoidAccuracy of design-to-code translation

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The system implements feedback mechanisms where the code synthesis engine continuously refines its translation accuracy by learning from the matching process between extracted graphical features and candidate UI elements. The feedback loop allows the system to improve its understanding of design intent over time, ensuring that automated code generation maintains high fidelity to the original design.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent replaces manual mechanical translation processes with automated machine learning-based feature extraction and matching. Instead of developers manually translating design documents to code, the system uses computer vision and pattern recognition algorithms to automatically identify and translate graphical features, improving both speed and consistency of the translation process.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Reliability

If detailed manual coding is performed, then UI functionality can be precisely implemented, but the complexity of the development process increases

Engineering Contradiction:

ImproveUI functionality implementationVSAvoidDevelopment process complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system extracts only the essential graphical features from design documents that are necessary for code generation, separating the critical information needed for UI implementation from unnecessary design details. This extraction process simplifies the development process by focusing only on the features that directly translate to functional code, reducing overall process complexity while maintaining implementation reliability.

Inventive Principle:

Principle #2Taking out (Extraction)

4Ease of operation

If designers and developers work with different tools and methodologies, then each can optimize their workflow, but communication and translation between them becomes difficult

Engineering Contradiction:

ImproveDesigner and developer workflow efficiencyVSAvoidDesign intent communication

Core Design Contradiction:

Ease of operationVSLoss of information

Solution Approach 1:

The patent creates a universal code synthesis engine that can process various types of presentation documents and design formats, making the translation process independent of specific design tools or methodologies. This universal approach allows designers to use their preferred tools while developers receive standardized, accurately translated code, preventing information loss during translation between different workflows.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS10678517B1User interface synthesis based upon extracted presentation document graphical features

Publication Date: 2020.06.09 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A presentation document specifying a graphical layout of a user interface is received. A processor extracts a first user interface graphical feature from the presentation document. The extracted first user interface graphical feature is matched to one or more first candidate user interface graphical features to determine a first selected candidate user interface graphical feature. A user interface code representation of the user interface is synthesized based upon the first selected candidate user interface graphical feature.