On-Screen Text Detection for Interactive Video GUIs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video processing systems fail to effectively interact with users by detecting and utilizing on-screen text within live and on-demand video content, limiting the ability to provide enhanced user experiences through contextual and accurate text recognition.
Innovation Solution
A system and method that analyzes video data to detect on-screen text, performs optical character recognition, and generates graphical user interfaces (GUI) elements based on the detected text, allowing users to interact with the video content by selecting and performing operations related to the recognized text, with context-based dictionary selection to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If on-screen text detection is implemented in video content, then user interaction capability is improved, but system complexity increases
Solution Approach 1:
The system segments the video processing task into distinct modules: video data reception, on-screen text detection, optical character recognition, and GUI element generation. Each module handles a specific aspect of the processing pipeline, making the overall complex system manageable and maintainable while enabling sophisticated user interaction capabilities.
Solution Approach 2:
The patent introduces an intermediary processing layer that detects on-screen text and converts it into structured data before generating GUI elements. This intermediary layer acts as a mediator between the raw video content and the user interface, transforming visual information into interactive elements without requiring direct complex interaction between video processing and UI generation systems.
2Measurement precision
If optical character recognition is performed on on-screen text, then text recognition accuracy is improved, but processing time increases
Solution Approach 1:
The system performs preliminary detection of on-screen text regions before applying optical character recognition. By pre-identifying and isolating text areas in the video frames, the system prepares the data in advance for OCR processing, reducing the actual recognition time while maintaining high accuracy through focused processing on detected text regions rather than entire video frames.
3Measurement precision
If context-based dictionary selection is used, then text recognition accuracy is improved, but computational resources increase
Solution Approach 1:
The system applies context-based dictionary selection locally to specific video segments or frames where on-screen text is detected, rather than applying it uniformly to all video processing. This localized approach uses computational resources only when and where text recognition is needed, improving accuracy for detected text while minimizing overall resource consumption by avoiding unnecessary processing in non-text areas.
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
Systems, methods, and devices for an interactive viewing experience by detecting on-screen data are disclosed. One or more frames of video data are analyzed to detect regions in the visual video content that contain text. A character recognition operation can be performed on the regions to generate textual data. Based on the textual data and the regions, a graphical user interface (GUI) definition to can be generated. The GUI definition can be used to generate a corresponding GUI superimposed onto the visual video content to present users with controls and functionality with which to interact with the text or enhance the video content. Context metadata can be determined from external sources or by analyzing the continuity of audio and visual aspects of the video data. The context metadata can then be used to improve the character recognition or inform the generation of the GUI.