Multi-node Word Segmentation System for Large Vocabulary Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search engine tokenizers face high costs and memory limitations when handling large vocabularies, particularly in industries with detailed categorization, such as electronic components, leading to system unavailability and memory leaks.
Innovation Solution
A multi-node word segmentation system comprising a word distribution proxy server, client proxy server, and multiple word segmentation node servers, where words are distributed based on preset rules, such as word attribute information, to efficiently manage and search large vocabularies across multiple independent or virtual servers, with a standby server for reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single memory type is used to store the tokenizer dictionary, then the system structure is simple, but the memory cost rises linearly when vocabulary reaches millions or tens of millions of levels
Solution Approach 1:
The patent divides the single large memory into multiple distributed memory nodes (first memory, second memory, third memory, etc.), each storing a portion of the vocabulary. This segmentation allows the system to handle millions or tens of millions of vocabulary items without requiring a single large memory, thus controlling memory cost while maintaining scalability.
2Quantity of substance
If memory size is expanded to handle large vocabularies, then vocabulary capacity increases, but the system may cause unpredictable memory leaks and system collapse
Solution Approach 1:
By distributing vocabulary across multiple independent memory nodes, the system avoids the memory leak and collapse issues associated with single large memory expansions. Each memory node operates independently within safe memory limits, preventing system-wide failures.
Solution Approach 2:
The tokenizer is introduced as an intermediary component that manages the distributed memory nodes. It receives segmentation information, determines which memory node contains the required vocabulary item, and coordinates the retrieval process, thereby managing large vocabulary capacity while maintaining system stability.
3Adaptability or versatility
If third-party vocabulary is added to supplement semantics, then semantic coverage improves, but the vocabulary size increases leading to higher memory costs
Solution Approach 1:
The distributed memory architecture allows third-party vocabulary to be added across multiple memory nodes rather than requiring expansion of a single memory. This enables semantic coverage to improve while memory cost per node remains controlled, as the total vocabulary is distributed across the network.
4Device complexity
If a single tokenizer handles all vocabulary, then the system is simple to manage, but retrieval efficiency decreases with large vocabulary sizes
Solution Approach 1:
The tokenizer functionality is divided across multiple tokenizer instances associated with different memory nodes. Each tokenizer handles a specific portion of the vocabulary, enabling parallel retrieval operations that maintain high speed even as total vocabulary size increases to millions or tens of millions of items.
Data Source
AI summary
The invention relates to a multi-node word segmentation system and method for keyword search. A word distribution proxy server in that system is communication connection with each word segmentation node server respectively, receive a large number of words, and distributes each word to a corresponding word segmentation node server according to a preset word segmentation rule; the client proxy server is communication connection with each word segmentation node server respectively, receives terms to be searched uploaded by a user terminal, and distributes the terms to be searched to the corresponding word segmentation node servers according to a preset word segmentation rule; the segmentation node server inquires the words matched with the terms to be searched, and returns the words obtained by inquiry to the client proxy server; and that client proxy server sends the query words to a corresponding user terminal.