Table Pre-processing for NLP Using Interactive Element Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Natural Language Processing (NLP) technologies face challenges in accurately pre-processing tables within documents, as they lack user interaction and visualization tools for inspecting and editing the classification of table elements, which affects the accuracy of NLP outputs.
Innovation Solution
A graphical user interface (GUI) is developed to allow users to select and modify the classification of table elements, including main, conditional, and value elements, enabling users to optimize the NLP engine's processing by providing a modified set of element classifications and lexical connectors for improved understanding of table data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated table flattening is performed without user interaction, then processing speed is improved, but classification accuracy deteriorates
Solution Approach 1:
The system performs preliminary automated classification of table elements into main, conditional, and value elements before user review. This preliminary action enables fast initial processing while allowing subsequent refinement, thus resolving the contradiction between processing speed and classification accuracy.
Solution Approach 2:
The system implements a feedback mechanism where users can review and correct the automated classification results through a graphical interface. The corrected classifications are then fed back into the system to improve future processing accuracy, maintaining both speed and precision.
2Measurement precision
If user interaction is added for editing table element classifications, then classification accuracy is improved, but device complexity increases
Solution Approach 1:
A graphical user interface acts as an intermediary between the automated processing system and the user. This intermediary presents classification results in an intuitive visual format and captures user corrections, enabling accurate classification review without requiring complex system changes.
Solution Approach 2:
The system performs self-service by automatically generating the initial classification and preparing it for review. This self-service capability reduces the burden on users and simplifies the overall system architecture while still enabling accuracy improvement through user feedback.
3Productivity
If table flattening is performed without visualization tools, then processing efficiency is improved, but understanding of table data deteriorates
Solution Approach 1:
The system segments the table data into distinct classification categories (main elements, conditional elements, value elements) and presents them separately in the graphical interface. This segmentation enables efficient processing of each category while providing users with clear visual understanding of the data structure and relationships.
Data Source
AI summary
Provided are a computer program product, system, and method for pre-processing a table in a document for natural language processing (NLP). A graphical user interface (GUI) provides a representation of table items in a table in a document including a set of a main element comprising an entity whose value is to be extracted, a conditional element that refines the entity, and a value element comprising a value for the entity. Graphical controls are rendered in the GUI to enable a user to select an element from the table to be the main element, conditional element, and value element. The set of the main element, conditional element, and value element are updated with the user selected element to form a modified set. The modified set of the main element, conditional element, and the value element are provided to an NLP engine to perform natural language processing.