Automated UI Component Identification via DOM Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for generating UI codes from visual mockups require manual identification and annotation of UI component types, leading to low efficiency in the front-end business development process.
Innovation Solution
A component identification method and apparatus that utilize a set identification model to identify UI blocks in an image and a set classification model to determine UI component types, eliminating the need for manual annotation by mapping identified UI blocks to nodes in the DOM.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual identification and annotation of UI component types is used, then accuracy of component identification can be maintained, but productivity is low and time consumption is high
Solution Approach 1:
The system performs self-service by automatically identifying and annotating UI component types through the identification model and classification model, eliminating the need for manual annotation while maintaining accuracy. The models process the visual mockup and DOM nodes autonomously to generate component type annotations.
Solution Approach 2:
The manual mechanical process of identification and annotation is replaced by an automated system using the identification model to detect UI blocks and the classification model to determine component types. This substitution of human manual work with automated modeling significantly improves productivity while preserving measurement precision.
2Productivity
If automated identification model is used, then productivity is improved, but device complexity increases due to multiple models and processing steps
Solution Approach 1:
The automated identification system is segmented into two distinct functional modules: an identification model for detecting UI blocks and their boundaries, and a classification model for determining component types. This segmentation allows each model to specialize in a specific task, improving overall productivity while organizing complexity into manageable, independent components.
Solution Approach 2:
The identification model acts as an intermediary between the input visual mockup and the classification model. It first identifies UI blocks and extracts relevant features, then passes this processed information to the classification model. This intermediary structure manages system complexity by creating a clear workflow and reducing the direct complexity burden on any single component.
3Measurement precision
If multiple processing steps including edge detection and cropping are added, then measurement precision of UI blocks is improved, but loss of time increases due to additional processing
Solution Approach 1:
Edge detection and image cropping are performed as preliminary actions before the main identification and classification processes. By pre-processing the visual mockup to extract relevant regions and eliminate unnecessary elements, the system improves measurement precision of UI blocks while optimizing subsequent processing efficiency, thereby reducing overall time loss.
Data Source
AI summary
The present disclosure provides a component identification method which includes: inputting a first image into a set identification model to obtain at least one UI block in the first image outputted by the set identification model, wherein the first image is determined on the basis of a first visual mockup, the set identification model is used for identifying the at least one UI block in an inputted image, and each UI block at least comprises an image region obtained by UI component rendering; determining, among the nodes of a document object model DOM corresponding to the first visual mockup, a first node corresponding to each UI block in the at least one UI block; inputting an image corresponding to each first node into a set classification model to obtain a UI component type corresponding to the first node outputted by the set classification model.