Learning-Based Parser Rule Generation for Data Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing information collection systems face challenges in efficiently extracting target information from vast amounts of data in various formats, requiring developers to create parser programs for each service and reprogram them with specification changes, leading to increased costs and inefficiencies.
Innovation Solution
An information collection system that uses learning data to generate parser rules for extracting target character strings, which are then used to efficiently collect specific information from data across different services, reducing the need for manual programming and improving adaptability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a parser program is developed for each service to extract information from Web contents, then the extraction accuracy for service-specific formats is improved, but the device complexity and programming cost increase significantly
Solution Approach 1:
The patent creates a universal parser program that can extract information from multiple service types (SNS, blogs, news sites, etc.) using a single unified structure. The parser uses service identification based on URL patterns and applies appropriate extraction rules dynamically, eliminating the need to develop separate parser programs for each service while maintaining high extraction accuracy for service-specific formats
Solution Approach 2:
The patent changes the approach from creating multiple fixed parser programs to using a single flexible parser that adapts its extraction parameters based on service identification. The system stores extraction rules as configurable parameters that can be adjusted based on the identified service type, allowing the same parser structure to handle diverse formats by changing its operational parameters rather than its fundamental structure
2Measurement precision
If parser programs are reprogrammed every time content specifications change, then the extraction accuracy is maintained, but the productivity and response time deteriorate
Solution Approach 1:
The patent implements a dynamic extraction rule storage that allows extraction specifications to be modified without reprogramming the parser program itself. The system stores extraction rules in a configurable format that can be updated to reflect changes in service content specifications, enabling the parser to adapt to new formats by loading updated rules rather than requiring program rewrites
Solution Approach 2:
The parser program automatically identifies the service type based on URL patterns and selects the appropriate extraction rules from the storage without requiring manual intervention or reprogramming. This self-service capability allows the system to maintain high extraction accuracy even when content specifications change, as the updated rules are automatically applied based on service identification
3Reliability
If manual development of parser programs is performed for each service, then the extraction reliability is improved, but the loss of time and automation level worsen
Solution Approach 1:
The patent implements preliminary service identification based on URL patterns before the actual information extraction process. By pre-configuring service identification rules and extraction rule sets for multiple services, the system automatically determines the appropriate extraction approach without manual intervention, maintaining high reliability while eliminating the time-consuming manual development process for each new service
Solution Approach 2:
The unified parser structure with service identification capability provides reliable extraction across multiple service types through a single program. The system maintains extraction reliability by implementing service-specific extraction rules within a universal framework, eliminating the need for manual development of separate parsers for each service while preserving the reliability benefits of service-optimized extraction logic
Data Source
AI summary
An information collection system for efficiently collecting target information from an enormous amount of contents in a variety of formats is provided. The information collection system 100 includes a learning unit 110 and an extraction unit 130. The learning unit 110 generates, by using learning data, a parser rule for extracting a target character string from data, the target character string being a character string including specific information. The extraction unit 130 extracts the target character string from data by using the parser rule.