NLP Text Normalization via Properties Files
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Natural language processing systems face challenges in interacting with diverse text formats from various data sources, as existing technologies struggle to convert and normalize text into a shared format for effective processing.
Innovation Solution
A system and computer program product that utilize properties files and extension classes to identify and convert text from different formats into a normalized format, assigning text portions to formatting elements and storing them as NLP objects, allowing for uniform processing across diverse data sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If text from multiple different data sources with different formats is ingested directly into the NLP system, then the system can access diverse information sources, but the system cannot effectively process and analyze the text due to format inconsistencies
Solution Approach 1:
The patent introduces an intermediary component (text normalization service) that sits between the diverse text sources and the NLP processing system. This service receives text in various formats from different sources, normalizes it to a common format using properties files and extension classes, and then provides the standardized text to the NLP system, thereby enabling both diverse source access and effective processing
Solution Approach 2:
The patent segments the text processing system into distinct modular components: format detectors that identify source-specific formats, properties files that define format characteristics, extension classes that implement format-specific parsing logic, and a normalization service that coordinates these components. This segmentation allows each component to handle specific format requirements independently while working together to achieve overall normalization
2Manufacturing precision
If a custom processing approach is developed for each data source format, then each format can be processed accurately, but the system complexity increases significantly
Solution Approach 1:
The patent creates a universal text normalization service that can handle multiple different text formats through a single unified interface. The service uses properties files and extension classes to adapt to different formats without requiring separate processing pipelines, thereby maintaining processing accuracy while reducing system complexity through multi-functionality
Solution Approach 2:
The patent uses parameter changes to adapt the processing approach based on the input format. Properties files store format-specific parameters (delimiters, encoding, structure), and the system dynamically adjusts its processing behavior by loading and applying the appropriate parameters for each data source, eliminating the need for hard-coded format-specific logic
3Ease of operation
If text is normalized through a systematic conversion process, then text from different sources can be processed uniformly, but the processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary action by pre-defining format characteristics in properties files and pre-compiling extension classes for common formats. This preparation work is done beforehand, allowing the normalization service to quickly identify and apply the appropriate processing rules without performing extensive analysis during actual text processing, thereby reducing processing time while maintaining uniformity
Data Source
AI summary
The data store for a natural-language computing system may include information that originates from a plurality of different data sources—e.g., journals, websites, magazines, reference books, and the like. In one embodiment, the information or text from the data sources are converted into a single, shared format and stored as objects in a data store. In order to ingest the different documents with their respective formats, a natural language processing system may perform preprocessing to change the different formats into a normalized format. When a new text document is received, the text may be correlated to a particular properties file which includes instructions specifying how the preprocessor should interpret the received text. Based on these instructions, a preprocessor identifies relevant portions of the text document and assigns these portions to formatting elements in the normalized format. The text may then be stored in the objects based on this assignment.