Multilingual Text Indexing via Frozen Neural Network Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for indexing and searching multilingual texts are inefficient, often relying on multiple indices or pivot languages, which complicate the process and do not effectively leverage language-agnostic representations.
Innovation Solution
A computer-implemented method using a transformer-type neural network to transform texts into latent vectors, freezing certain layers to remove meaningless words, and grouping vectors to create unique identifiers for indexing and searching across multiple languages, allowing for language-agnostic information creation and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple indices or pivot languages are used for multilingual text indexing, then language coverage is improved, but system complexity increases
Solution Approach 1:
The patent applies universality by creating a single multilingual index structure that can handle multiple languages simultaneously through language-agnostic vector representations. Instead of maintaining separate indices for each language or requiring pivot language translations, the system uses a unified indexing mechanism that directly processes and stores semantic vectors from texts in any language, making the index universally applicable across language boundaries.
2Measurement precision
If all layers of the neural network are trained, then language-specific nuances are preserved, but processing speed decreases
Solution Approach 1:
The patent applies the extraction principle by selectively training only the first NK layers of the transformer neural network while freezing the last K layers. This extracts and utilizes only the necessary language-agnostic semantic features from the earlier layers, removing the need to process and train all layers. The frozen later layers, which would capture language-specific nuances, are bypassed to achieve faster processing while maintaining sufficient semantic representation quality for multilingual search.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention relates to a method for indexing a plurality of texts to be indexed, the method comprising the following steps: obtaining the plurality of texts to be indexed; transforming, by a neural network, the texts to be indexed into one or more vectors in a latent space, the neural network being learned with training texts by freezing the last K layers of the neural network to remove words without specific meaning; determining for each vector for each text to be indexed at least one unique identifier of a nearest group; creating agnostic information for the texts to be indexed, the agnostic information comprising the determined unique identifiers, and associating the agnostic information with the respective texts to be indexed; indexing the agnostic information.