Text-Based UI Control via Synchronous Terminal Protocol

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing techniques for programmatically controlling computer programs with text-based user interfaces are error-prone and ineffective, as they rely on image processing methods that fail to accurately interpret and control text-based UIs.

Innovation Solution

A method and system that access a string specifying the content of a text-based UI screen, generate a 2D representation, and identify controllable UI elements by processing text, allowing for direct control of the computer program via a synchronous terminal protocol without the need for terminal emulators or image recognition.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If image processing methods are used to control text-based UIs, then the system can interface with graphical interfaces, but the accuracy and reliability of interpretation deteriorates

Engineering Contradiction:

Improveinterface capabilityVSAvoidinterpretation accuracy

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent replaces image processing methods (optical/mechanical approach) with direct text processing through synchronous terminal protocol (digital/programmatic approach). Instead of capturing and analyzing UI screenshots as images, the system communicates directly with the text-based UI through protocol-level text commands, substituting a imprecise mechanical sensing method with a precise digital communication method.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The synchronous terminal protocol acts as an intermediary layer between the controlling system and the text-based UI. Rather than directly processing UI elements through image recognition, the protocol mediates communication by translating control actions into text-based commands that the UI can natively understand and execute, improving both accuracy and reliability.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If terminal emulators are used to interface with text-based UIs, then compatibility is improved, but system complexity and resource consumption increase

Engineering Contradiction:

ImprovecompatibilityVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent extracts the essential text-based communication functionality from the complex terminal emulator environment and implements it directly through synchronous terminal protocol. By removing the unnecessary emulator layer and its associated complexity, the system retains compatibility with text-based UIs while significantly reducing system complexity and resource consumption.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

Instead of using a terminal emulator to simulate a text-based UI environment (adding complexity), the patent inverts the approach by using the synchronous terminal protocol to directly communicate with the text-based UI at its native protocol level. This eliminates the need for emulation while maintaining full compatibility.

Inventive Principle:

Principle #13The other way round (Inversion)

3Ease of operation

If image recognition is used to identify UI elements, then visual interface control is enabled, but processing time and computational resources increase

Engineering Contradiction:

Improvecontrol capabilityVSAvoidprocessing time

Core Design Contradiction:

Ease of operationVSLoss of time

Solution Approach 1:

The patent replaces image recognition processing (computationally intensive visual analysis) with direct text-based command processing through synchronous terminal protocol. By substituting the mechanical image analysis process with native text command execution, the system maintains full control capability while dramatically reducing processing time and computational resource requirements.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

4Productivity

If synchronous terminal protocol is used for direct communication, then communication efficiency is improved, but compatibility with graphical interfaces is reduced

Engineering Contradiction:

Improvecommunication efficiencyVSAvoidinterface compatibility

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The synchronous terminal protocol is designed to be universally applicable to text-based UIs while maintaining efficient communication. The protocol handles multiple text-based interface types (mainframes, legacy systems, terminal applications) through a unified communication framework, achieving both high communication efficiency and broad compatibility with text-based interfaces.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11610052B2Techniques for automated control of computer programs through text-based user interfaces

Publication Date: 2023.03.21 SOROCO PTE LTD

AI summary

Techniques for programmatically controlling computer programs that are configured to be controlled via text-based user interfaces. The techniques include using at least one computer hardware processor to perform: accessing a string specifying content of a user interface (UI) screen of a computer program configured to be controlled via a text-based user interface; generating, based on the string, a two-dimensional (2D) representation of the UI screen; identifying a first controllable UI element in the UI screen at least in part by processing text in the 2D representation of the UI screen; and controlling the computer program by causing, via the first controllable UI element, the computer program to perform at least one action in furtherance of a task.