Data Generation System for Language Resource Creation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for generating language resources, such as dictionaries, are time-consuming and labor-intensive due to the need to determine whether extracted words should be registered, especially when dealing with compound words in text data across multiple documents.

Innovation Solution

A data generation system that divides sentences into tokens, extracts candidate tokens for combination based on token characteristics, and generates language resources by combining these tokens, using both pair extraction and clustering methods to efficiently identify and register unique expressions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If determination is performed for many extracted words to decide whether to register them in the dictionary, then the accuracy of the language resource is improved, but the time and workload required for generation increases significantly

Engineering Contradiction:

Improveaccuracy of language resourceVSAvoidtime and workload for generation

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent segments the dictionary generation process into distinct phases: automatic extraction of candidate words from text, clustering of these candidates based on co-occurrence patterns, and selective verification only for clustered candidates. This segmentation allows the system to process large volumes of text efficiently while maintaining accuracy through targeted human review of only the most promising candidates.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary automatic processing of word extraction and clustering before human verification. By pre-processing and organizing candidate words into clusters based on their co-occurrence patterns in the text, the system reduces the burden on human operators who only need to verify pre-organized candidates rather than evaluating all extracted words from scratch.

Inventive Principle:

Principle #10Preliminary action

2Quantity of substance

If all extracted words are processed individually for registration determination, then the completeness of the language resource is improved, but the complexity and workload of the generation process increases

Engineering Contradiction:

Improvecompleteness of language resourceVSAvoidcomplexity of generation process

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent merges multiple candidate words into clusters based on their co-occurrence patterns in the text. Words that frequently appear together are grouped into the same cluster, allowing the system to process them collectively. This merging reduces the total number of individual verification tasks while ensuring that related terms are captured together in the language resource.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The clustering mechanism serves multiple functions simultaneously: it groups related candidates, identifies patterns in the text, reduces verification workload, and organizes data for efficient processing. This multi-functional approach handles both large quantities of candidates and process complexity through a single unified mechanism.

Inventive Principle:

Principle #6Universality (Multi-functionality)

3Measurement precision

If manual verification is performed for every extracted word, then the accuracy of word registration is improved, but the productivity of the language resource generation decreases

Engineering Contradiction:

Improveaccuracy of word registrationVSAvoidproductivity of generation

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The system applies partial verification by performing manual checking only on clustered candidate words rather than all extracted words. The clustering process filters and prioritizes candidates based on their co-occurrence patterns, so verification is applied excessively to the right subset (clustered candidates) while skipping unnecessary verification of low-priority candidates, thus maintaining accuracy where needed while improving overall productivity.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20230376688A1Data generation system, data generation method, and recording medium

Publication Date: 2023.11.23 NEC CORP

AI summary

A data generation system includes a division unit, an extraction unit, an output unit, an input unit, and a generation unit. The division unit divides each of the plurality of sentences into tokens. The extraction unit extracts a sentence including candidates for a token to be combined as a word from a plurality of sentences based on characteristics of tokens included in the plurality of sentences. The output unit outputs a sentence including the candidates for the token to be combined. The input unit receives, as an input, an instruction to combine the candidates for the token to be combined included in the sentence output by the output unit. The generation unit generates a language resource including a word obtained by combining the candidates for the token to be combined based on the input.