Automated Screen Image Transition Diagram Generation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for verifying screen image transitions in computer systems require significant manual intervention and labor, even when using automated systems, due to the potential for bugs in analysis programs.

Innovation Solution

A device and method that automatically generate screen image transition information by acquiring and analyzing screen image data, extracting buttons, selecting operation targets, simulating button operations, recording transitions, and generating diagrams without manual intervention, using a system comprising a controller, robot, and camera to automate the process.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If manual clicking or tapping each operation target element is performed to verify screen image transition, then verification accuracy is improved, but labor and time consumption increase significantly

Engineering Contradiction:

Improveverification accuracyVSAvoidtime consumption

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system automatically performs screen image transition verification by having the device itself capture its own screen images, extract button information, and generate transition diagrams without requiring external manual intervention. The subject device serves its own verification needs through automated image acquisition and analysis.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces manual mechanical operations (clicking, tapping, recording) with automated image processing and analysis. Image recognition algorithms automatically identify buttons and their positions, substituting human manual operations with computational processes that are both faster and equally accurate.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If automated analysis program is used to generate screen image transition diagram, then productivity is improved, but reliability deteriorates due to potential bugs in analysis program

Engineering Contradiction:

Improveautomation efficiencyVSAvoidverification reliability

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The system captures actual screen images from the display and uses these visual copies as the basis for analysis. By working with direct image copies rather than program source code or documentation, the system verifies the actual rendered output, ensuring that the transition diagram reflects the true screen behavior.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent introduces image data as an intermediary between the screen display and the analysis process. Instead of directly analyzing program code or manually observing screens, the system uses captured image data as a mediator that bridges the gap, allowing automated analysis while maintaining accuracy through visual verification.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Measurement precision

If manual recording of screen image transition is performed, then measurement precision is improved, but device complexity increases due to manual intervention requirements

Engineering Contradiction:

Improvetransition recording accuracyVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent combines multiple functions into the subject device itself: screen image capture, button information extraction, transition detection, and diagram generation are all integrated within the same device. This merging eliminates the need for separate manual recording tools and personnel, reducing overall system complexity while maintaining precision.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS12049015B2Screen image transition information generation device, screen image transition information generation method, screen image transition information generation program, and screen image transition information generation system

Publication Date: 2024.07.30 JAPAN NOVEL

AI summary

A device to verify a screen image transition without manual intervention. The device includes: a screen image data acquisition unit which acquires image data of the screen image; a screen image transition button extraction unit which analyzes image data of the screen image and extracts a button in the screen; an operation target button selection unit which selects a button to be an operation target; a button operation unit which causes the device to generate a signal when the selected button is operated; a screen image transition recording unit which records screen image transition data in a recording medium, the data including image data of the screen images before and after the screen image transition caused by the operation, and data related to the button operated during the transition; and a screen image transition diagram generation unit which generates a screen image transition diagram using the screen image transition data.