Context-Sensitive Text Segmentation Using User-Defined Rules
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Text segmentation in natural languages, particularly in languages like Korean and Japanese, is challenging due to the absence of fixed word delimiters, requiring context understanding and extensive language-specific knowledge, which is difficult and costly to obtain.
Innovation Solution
A computer-implemented system and method for context-sensitive text segmentation using user-defined rules that apply pattern matching to segment textual data into meaningful elements, allowing for segmentation without delimiter characters and integrating data from disparate sources for standardization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional text segmentation methods are used for languages like Korean and Japanese, then segmentation can be performed, but extensive language-specific knowledge and large corpora are required which are difficult and costly to obtain
Solution Approach 1:
The patent introduces user-defined rules as an intermediary mechanism between the text segmentation system and language-specific knowledge. Instead of requiring the system to possess extensive language corpora and knowledge bases, users provide segmentation rules that act as mediators, enabling accurate segmentation without needing complex language-specific data structures.
Solution Approach 2:
The system enables users to define their own segmentation rules based on their specific needs and data characteristics. This self-service approach allows users to provide the necessary language-specific knowledge directly without requiring the system to pre-store extensive corpora or knowledge bases, reducing system complexity while maintaining segmentation accuracy.
2Ease of operation
If context-sensitive segmentation rules are applied to segment text without delimiter characters, then segmentation can be achieved in languages like Korean, but the system must handle complex pattern matching without fixed word boundaries
Solution Approach 1:
The patent segments the text processing task into multiple stages: first applying user-defined segmentation rules to divide the text into potential words or phrases, then passing these segments to a morphological analyzer for further processing. This segmentation approach simplifies the overall operation by breaking down the complex task of segmenting text without delimiters into manageable steps.
Solution Approach 2:
The system dynamically adjusts the segmentation process based on user-defined rules and context-sensitive patterns. Rather than using fixed segmentation methods, the system adapts its behavior according to the specific language characteristics and user requirements, making the operation more flexible while managing complexity through configurable parameters.
3Measurement precision
If extensive language-specific corpora are used for text segmentation, then segmentation accuracy improves, but the cost and difficulty of obtaining such corpora increases
Solution Approach 1:
User-defined rules serve as an intermediary that replaces the need for extensive pre-existing language corpora. Instead of requiring the system to acquire and store large amounts of language-specific data, users provide the necessary segmentation patterns as rules, achieving precise segmentation without the complexity of corpus acquisition and maintenance.
Solution Approach 2:
The system allows users to create custom segmentation rules that copy and adapt segmentation patterns from their specific domain or language requirements. Rather than relying on pre-existing corpora that may not match specific needs, users can copy relevant patterns and modify them to fit their exact requirements, reducing the cost and difficulty of acquiring suitable training data.
Data Source
AI summary
Computer-implemented methods and systems are provided for text segmentation of textual data. Rules are accessed that define how the input stream is to be segmented into textual data elements through pattern matching. The one or more rules are applied to the input stream to determine the textual data elements in the input stream which are then provided as output.