Machine Learning UI Code Reuse from Design Documents

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current user interface (UI) development processes are inefficient as developers spend significant time and effort recreating UI elements from design files, especially when unfamiliar with specific elements, and existing visual drag/drop tools provide only generic skeleton code without logic implementation.

Innovation Solution

A system utilizing machine learning to extract and encapsulate complete source code for UI elements from past designs, leveraging cognitive machine learning and visual recognition to match design elements with corresponding source code, providing more comprehensive code for re-use across applications.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If developers manually recreate UI elements from design files, then UI implementation can be completed, but development time and effort increase significantly

Engineering Contradiction:

ImproveUI development efficiencyVSAvoidtime spent recreating UI elements

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system automatically copies UI element definitions from design files (such as Sketch, Figma, or Adobe XD) into executable code. The machine learning model extracts visual information from design file images or XML structures and generates corresponding UI implementation code, eliminating the need for developers to manually recreate each UI element from scratch.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent replaces the manual mechanical process of UI development with an automated machine learning system. Instead of developers manually analyzing design files and writing code, the system uses computer vision and natural language processing to automatically interpret design specifications and generate implementation code, substituting human cognitive and manual labor with automated intelligence.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If visual drag/drop tools are used to create UI elements, then code generation is accelerated, but only generic skeleton code without logic implementation is provided

Engineering Contradiction:

Improvecode generation speedVSAvoidcompleteness of code logic

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system performs preliminary extraction and analysis of complete UI element specifications from design files before code generation. By pre-processing the design files to extract not just visual structure but also embedded logic, styling, and behavioral specifications, the system ensures that the generated code includes complete logic implementation rather than requiring subsequent manual completion.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces an intermediary machine learning model that bridges design files and code generation. This intermediary layer extracts comprehensive information from design files including visual appearance, layout specifications, and implicit logic, then translates this into complete, ready-to-use code. This intermediary process ensures both speed and completeness by automating the translation from design to implementation.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Adaptability or versatility

If developers are unfamiliar with specific UI elements, then design flexibility is maintained, but development complexity and difficulty increase

Engineering Contradiction:

Improvedesign element flexibilityVSAvoiddevelopment complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The system enables self-service code generation by automatically interpreting design files and generating implementation code without requiring developer expertise in specific UI elements. The machine learning model serves itself by extracting necessary information from design specifications and producing complete, functional code, eliminating the need for developers to manually research or implement complex UI element logic.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS11048485B2User interface code re-use based upon machine learning of design documents

Publication Date: 2021.06.29 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Facilitating code re-use includes receiving a first design document. The first user design document specifies a graphical layout of a first set of user interface elements of a user interface of a first application. First source code associated with the design document is received. The first source code is configured to implement the first set of user interface elements of the user interface. A processor extracts the first set of user interface elements from the first user interface design document. A first portion of the first source code corresponding to a first user interface element of the first set of user interface elements is identified. The first portion of the first source code is encapsulated into a module to enable re-use of the portion of the source code in a second application.