NLU Model Training with HTML Tag Structure
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing natural language understanding (NLU) models trained on text data from HTML documents lack the ability to utilize structural information from HTML documents, leading to degraded analysis performance when analyzing HTML documents.
Innovation Solution
An apparatus and method for processing natural language that includes a collection module for gathering documents with tags, a parsing module for extracting text and tag-related information, and a preprocessing module for generating tokens and token position information, which are then used to train an NLU model that incorporates tag attribute and structural position information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If only text information is extracted from HTML documents for training NLU models, then the training process is simplified, but structural information is lost leading to degraded analysis performance
Solution Approach 1:
The patent segments the HTML document processing into multiple components: text extraction, tag information extraction, and structured data generation. The parsing module divides the document into text segments and corresponding tag segments, creating structured training data that preserves both content and structural relationships without overwhelming the training process
Solution Approach 2:
The patent introduces an intermediary structured data format that bridges raw HTML and training data. The parsing module generates intermediate representations including text segments, tag segments, and relationship indicators that maintain structural information while being suitable for NLU model training
2Reliability
If structural information from HTML tags is preserved in training data, then analysis performance improves, but data processing complexity increases
Solution Approach 1:
The patent segments structural information into discrete, manageable components: depth levels, tag types, and position relationships. Each text segment is paired with corresponding tag segments that encode structural properties in a standardized format, making complex structural information processable without overwhelming complexity
Solution Approach 2:
The patent transforms structural information into parameterized representations: depth levels as numerical values, tag types as categorical labels, and position relationships as structured descriptors. This parameterization converts complex structural data into a format suitable for machine learning processing
Data Source
AI summary
An apparatus for processing natural language according to an embodiment includes a collection module that collects documents having tags, a parsing module that extracts text from the collected documents and extracts tag-related information on the tag surrounding each extracted text, and a preprocessing module that generates tokens of a preset unit by tokenizing each extracted text, generates token position information for each token in full text of the document, and sets the token and the token position information as training data in matching with the tag-related information.