Display Control Device for Real-Time Annotation Overlay

Solution Overview

Problem

Conventional annotation systems face high processing loads and long calculation times, especially when dealing with large application target images or significant screen changes, leading to slow screen update processing and unnatural motion during frame-by-frame playback, even with high-performance PCs.

Innovation Solution

A display control device and method that includes a detection unit for monitoring screen changes and a display control unit executing provisional and definitive update processing, where provisional updates use previous frame matching results to reduce the search area and speed up overlay display calculations, while definitive updates ensure accurate icon placement using a larger search range.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If image template matching is performed for all rules on each frame, then annotation accuracy is maintained, but processing time increases and real-time update capability is lost

Engineering Contradiction:

Improveannotation accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs image template matching in advance for all rules to generate matching result information, storing these results for subsequent reuse. This preliminary action allows the system to avoid redundant matching calculations in real-time frame updates, significantly reducing processing time while maintaining annotation accuracy through the use of pre-computed matching data.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adjusts the matching strategy based on screen change detection. When screen changes are detected, the system selectively updates only the necessary matching results rather than re-computing all matches. This dynamic approach optimizes the balance between maintaining accuracy and reducing processing time by adapting the matching scope to actual screen changes.

Inventive Principle:

Principle #15Dynamics

2Adaptability or versatility

If image template matching is performed on large application target images, then comprehensive annotation coverage is achieved, but processing load increases and screen update speed decreases

Engineering Contradiction:

Improveannotation coverageVSAvoidscreen update speed

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The system performs image template matching in advance for all rules to generate matching result information, storing these results for subsequent reuse. This preliminary action allows the system to avoid redundant matching calculations in real-time frame updates, significantly reducing processing time while maintaining annotation accuracy through the use of pre-computed matching data.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system applies different processing strategies to different regions of the screen based on detected changes. When only partial screen changes occur, the system updates matching results only for the changed regions rather than re-processing the entire large image. This local quality approach maintains comprehensive annotation coverage while reducing the processing load to only necessary areas.

Inventive Principle:

Principle #3Local quality

3Area of stationary object

If frame-by-frame processing is used for large windows or scrolled web browsers, then complete screen coverage is achieved, but unnatural motion and delayed updates occur

Engineering Contradiction:

Improvescreen coverage areaVSAvoidupdate speed

Core Design Contradiction:

Area of stationary objectVSSpeed

Solution Approach 1:

The system performs image template matching in advance for all rules to generate matching result information, storing these results for subsequent reuse. This preliminary action allows the system to avoid redundant matching calculations in real-time frame updates, significantly reducing processing time while maintaining annotation accuracy through the use of pre-computed matching data.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adjusts the matching strategy based on screen change detection. When screen changes are detected, the system selectively updates only the necessary matching results rather than re-computing all matches. This dynamic approach optimizes the balance between maintaining accuracy and reducing processing time by adapting the matching scope to actual screen changes.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS10147400B2Display control device, display control method, and display control program

Publication Date: 2018.12.04 NIPPON TELEGRAPH & TELEPHONE CORP

AI summary

A display control device (10) detects change of a screen, and executes, depending on the detected change of the screen, any one or both of provisional update processing that generates a strip-shaped rectangular region extended in a vertical direction or a horizontal direction based on the position of an icon displayed on the screen, searches the screen for a region similar to a specified image template by using the rectangular region as a search range, and performs overlay display of an icon in the similar region, and definitive update processing that searches for a region similar to the image template by using a region larger than a search range of the provisional update processing as a search range, and performs overlay display of an icon at a position corresponding to the similar region.