Automated UI Element Relocation via Computer Vision

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

User interfaces often become outdated or poorly developed, requiring tedious manual updates, which can disorient users and impede productivity due to drastic changes in design and functionality.

Innovation Solution

An automated UI update system that analyzes current UI components, maps them to new design specifications, generates stencils, and adjusts presentation aspects to create a neutral and appealing design, using machine learning algorithms for alignment and spacing, and implements a migration plan to smoothly transition users to new UIs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If manual updates are performed to refresh user interfaces according to new design standards, then the user interface can be updated to match new branding and design standards, but the update process becomes tedious and time-consuming

Engineering Contradiction:

Improveuser interface adaptability to new design standardsVSAvoidtime required for manual updates

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system automatically detects UI elements, determines their new positions based on design standards, and updates their locations without human intervention. The computer system performs the entire update process autonomously, from analyzing the current UI state to generating updated code, eliminating the need for manual updating while maintaining adaptability to new design standards.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual mechanical updating processes with an automated computer-based system that uses image processing and machine learning algorithms. Instead of developers manually inspecting and rewriting code, the system automatically analyzes UI screenshots, identifies elements, and generates updated code, substituting human labor with computational processes.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Adaptability or versatility

If drastic user interface changes are implemented to refresh the design, then the user interface can match new branding standards, but user productivity and engagement are impeded due to disorientation

Engineering Contradiction:

Improveuser interface design freshnessVSAvoiduser productivity and engagement

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The system selectively updates only those UI elements that need to change according to new design standards, while preserving the locations and arrangements of other elements. By applying changes locally rather than globally, the system maintains design freshness where needed while preserving familiar layouts elsewhere, reducing user disorientation and maintaining productivity.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent implements partial updates by identifying and modifying only the specific UI elements that require updates based on design standards, rather than redesigning the entire interface. This selective approach applies just enough change to achieve design goals while minimizing disruption to user workflows and familiar interactions.

Inventive Principle:

Principle #16Partial or excessive action

3Productivity

If automated systems are used to update user interfaces, then the update process becomes efficient and less time-consuming, but the system complexity increases

Engineering Contradiction:

Improveupdate process efficiencyVSAvoidautomation system complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system uses screenshot images as an intermediary representation of the UI state, allowing automated analysis without directly parsing complex code structures. By converting the UI to an image format and using image processing techniques, the system simplifies the automation process while maintaining efficiency, avoiding the need for complex code analysis and manipulation infrastructure.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Adaptability or versatility

If hand-written HTML tags and CSS are used in user interfaces, then design flexibility is maintained, but the code becomes poorly written and difficult to update

Engineering Contradiction:

Improvedesign flexibilityVSAvoidcode maintainability

Core Design Contradiction:

Adaptability or versatilityVSEase of manufacture

Solution Approach 1:

The system creates updated versions of UI code by generating new HTML and CSS based on analyzed screenshots and design standards, rather than manually editing existing poorly-written code. This copying approach produces clean, standardized code that maintains design flexibility while improving maintainability, as the generated code follows consistent patterns and modern best practices.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS11475205B2Automatically locating elements in user interfaces

Publication Date: 2022.10.18 SALESFORCE INC

AI summary

Techniques are disclosed relating to determining locations of child elements in a user interface (UI), automatically updating the locations of the child elements in the UI, and generating an updated UI with the updated locations. A computer system may receive information specifying content of the UI and render an image of the UI based on the received information. Computer vision algorithms may be applied to the rendered of the UI to determine locations of elements in the UI. The locations of the elements may be updated to position the elements in selected locations relative to each other. The updated locations of the elements as determined by the computer vision algorithms may be implemented to generate updating information specifying content for the updated UI.