Screen Image Table Recognition for Accurate RPA Data Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robotic Process Automation (RPA) systems face difficulties in accurately extracting tabular data from diverse table renderings in screen images due to variations in table structures and layouts, which conventional methods like OCR and machine learning algorithms struggle to address effectively.
Innovation Solution
An image processing technique that recognizes rows, columns, text, and controls within screen images to determine table structures, enabling automated extraction and construction of a computer-readable electronic table.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional OCR processing is used to recognize text in screen images, then text can be obtained, but the table structure and content cannot be accurately recognized
Solution Approach 1:
The patent segments the table recognition task into multiple components: detecting table boundaries, identifying rows and columns, extracting cell content, and reconstructing the table structure. This segmentation allows each component to be processed separately with specialized algorithms, improving overall accuracy while preserving structural information that conventional OCR loses.
2Difficulty of detecting and measuring
If machine learning algorithms like CNN are used to detect tabular structure, then structure detection capability is improved, but content and control location for each cell cannot be adequately located
Solution Approach 1:
The patent introduces an intermediary step between structure detection and content extraction. After detecting the table structure, the system identifies specific regions of interest (cells) and then extracts content from those regions. This intermediary approach allows the system to first understand the overall structure and then precisely locate content within each cell, solving the problem of inadequate content location.
3Extent of automation
If RPA systems use visual screenshots to access tabular data, then automation capability is enabled, but the variety of table renderings makes detection very complicated
Solution Approach 1:
The patent creates a universal table detection framework that can handle multiple table rendering styles and formats. The system identifies common structural elements (boundaries, rows, columns, cells) that appear across different table types, allowing a single detection algorithm to work universally across diverse table renderings without requiring separate specialized algorithms for each format.
4Ease of manufacture
If RPA systems are not integrated with existing software applications, then integration difficulties are eliminated, but difficulties in acquiring tabular data from screens increase
Solution Approach 1:
The patent uses screen image copying as the basis for data extraction. Instead of requiring integration with application programming interfaces, the system captures visual screenshots of the screen and processes these images to extract tabular data. This copying approach allows RPA systems to access data from any application without integration, eliminating the integration difficulties while providing an alternative method for data acquisition.
Data Source
AI summary
Embodiments disclosed herein concern improved techniques for recognition of tabularized data within an image and construction of a computer-readable electronic table representing the recognized tabularized data. The image can be a screen image produced by a software application program operating on a computing device. The tabular data, such as pertaining to a table, can be recognized by recognizing rows, columns, text, controls and lines.