Character String Division With Shared Vocabulary Embeddings

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing character string dividing systems face inefficiencies in vocabulary management, particularly in handling new vocabularies without expanding data size or increasing memory costs.

Innovation Solution

A method and apparatus that generate a new vocabulary group by reusing identifiers from an existing vocabulary group, associating embedding vectors without change for common vocabularies, and adding new identifiers for non-common vocabularies, thereby reducing data expansion and memory costs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a new vocabulary group is generated by adding new vocabularies to an existing model, then the model can support new languages or vocabularies, but the data size and memory costs increase

Engineering Contradiction:

Improvelanguage support capabilityVSAvoiddata size

Core Design Contradiction:

Adaptability or versatilityVSQuantity of substance

Solution Approach 1:

The patent makes the vocabulary group structure universal by introducing identifiers as a common interface between existing and new vocabularies. The identifier system allows the same data structure to serve multiple languages and vocabulary sets without duplication, enabling one vocabulary group to fulfill multiple language support functions while maintaining compact storage.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

Instead of copying entire vocabulary entries when adding new languages, the patent copies only the identifier mappings. The actual vocabulary data is shared through references to the existing vocabulary group, reducing data duplication from megabytes to minimal identifier tables.

Inventive Principle:

Principle #26Copying

2Adaptability or versatility

If a new vocabulary group is generated by adding new vocabularies to an existing model, then the model can support new languages or vocabularies, but the memory costs increase

Engineering Contradiction:

Improvelanguage support capabilityVSAvoidmemory cost

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The identifier-based vocabulary group structure serves multiple languages and vocabulary sets simultaneously, making the memory structure universal. This eliminates redundant storage of identical vocabulary data across different language models, significantly reducing overall memory consumption while maintaining support for multiple languages.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent merges multiple vocabulary groups into a unified structure where shared vocabularies are stored once and referenced by multiple language models through identifier mappings. This consolidation reduces memory usage by eliminating duplication while maintaining the ability to support multiple languages through the combined vocabulary group.

Inventive Principle:

Principle #5Merging (Combining)

3Adaptability or versatility

If vocabulary identifiers are changed when generating a new vocabulary group, then new vocabularies can be added, but the number of identifiers in the language model increases

Engineering Contradiction:

Improvevocabulary coverageVSAvoididentifier quantity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The identifier system is designed to be universal across vocabulary groups, where a single identifier can represent the same vocabulary item across multiple languages. This prevents identifier proliferation by making the identifier space reusable and shared, rather than creating separate identifier sets for each language or vocabulary group.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

Instead of creating new identifiers for vocabularies that already exist in the existing vocabulary group, the patent copies the existing identifier mappings. New identifiers are only created for truly new vocabularies, minimizing identifier quantity while maximizing vocabulary coverage through shared references.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS20250278566A1Character string dividing apparatus, vocabulary group generating method, and storage medium

Publication Date: 2025.09.04 RICOH CO LTD

AI summary

A character string dividing apparatus includes processing circuitry. The processing circuitry generates a new vocabulary group based on an existing vocabulary group including a plurality of vocabularies each being associated with an identifier. The processing circuitry acquires a character string and divides the character string into a plurality of vocabularies based on at least one of the existing vocabulary group and the new vocabulary group. In generating the new vocabulary group, the processing circuitry determines whether a vocabulary of a plurality of vocabularies included in the new vocabulary group is commonly included in the existing and the new vocabulary group, and associates an embedding vector associated with the vocabulary included in the existing vocabulary group with the vocabulary included in the new vocabulary group without changing the embedding vector based on a determination indicating that the vocabulary is commonly included in the existing and the new vocabulary group.