LLM-Based UI Snapshot Testing for Visual Design Defects

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional user interface testing methods fail to detect visual defects that require manual inspection and cannot ensure compliance with design specifications.

Innovation Solution

An AI-assisted user interface testing system utilizing a multimodal large language model to analyze user interface snapshots against design specifications, automatically generating and comparing natural language descriptions to identify and rectify visual defects.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If functional testing is used to test user interface features, then the workflow and interaction functionality can be verified, but visual defects and compliance with design specifications cannot be detected

Engineering Contradiction:

Improvetesting completenessVSAvoidvisual defect detection

Core Design Contradiction:

ReliabilityVSDifficulty of detecting and measuring

Solution Approach 1:

The patent replaces manual visual inspection with an automated AI-based system that uses image processing and computer vision to detect visual defects. The system captures screenshots of the user interface, extracts visual features using machine learning models, and automatically compares them against design specifications to identify defects such as incorrect font sizes, colors, layouts, and spacing.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent introduces an intermediary layer between the user interface and the testing system. This intermediary consists of visual feature extractors that convert UI elements into machine-readable visual representations, and comparators that automatically assess compliance with design specifications. This intermediary enables automated visual testing without requiring direct human intervention.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If manual visual inspection is performed to detect visual defects, then visual compliance with design specifications can be assessed, but testing efficiency and productivity are reduced

Engineering Contradiction:

Improvevisual compliance assessmentVSAvoidtesting efficiency

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The system enables self-service automated testing where the user interface itself is tested against its design specifications without requiring manual intervention. The automated testing framework captures the UI state, extracts visual features, compares them with specifications, and generates compliance reports independently, allowing continuous testing as part of the development pipeline.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent implements continuous automated visual testing that can run continuously during the software development lifecycle. The system can be integrated with CI/CD pipelines to automatically test visual compliance on every code change, ensuring continuous detection of visual defects without interruption to development productivity.

Inventive Principle:

Principle #20Continuity of useful action

3Productivity

If automated testing is implemented to improve productivity, then testing efficiency increases, but the ability to detect subtle visual defects may be compromised

Engineering Contradiction:

Improvetesting efficiencyVSAvoidvisual defect detection accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent segments the visual testing process into multiple independent stages: image capture, visual feature extraction, comparison with specifications, and defect identification. Each stage uses specialized machine learning models optimized for specific visual aspects such as typography, color, layout, and spacing. This segmentation allows the system to maintain high precision by applying domain-specific algorithms to each visual dimension.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system dynamically adjusts testing parameters and detection thresholds based on the complexity of the user interface and the specificity of design specifications. The machine learning models can adapt their sensitivity and detection criteria based on the visual elements being tested, ensuring accurate detection of subtle defects while maintaining high productivity through automated processing.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS20250355659A1User interface testing using large language models

Publication Date: 2025.11.20 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

A user interface testing system employs an AI-assisted description generator and an AI-assisted test engine to test various visual features of a user interface with respect to the design specification of the user interface. In an aspect, the AI-assisted test engine is given a natural language description of the implementation snapshot of the user interface and a natural language description of the visual feature being tested and determines whether or not the implemented user interface contains design defects. The AI-assisted description generator produces the natural language description of the implementation of the user interface from a snapshot of the implementation and produces the natural language description of the visual feature from a snapshot of the visual feature.