Latent Semantic Analysis Using Morpheme-by-Document Matrix Factorization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The computational linguistics and information retrieval communities have remained separate, with insights from computational linguistics being underutilized in information retrieval, particularly in areas like morphology, translation, and semantics, limiting the effectiveness of standard information retrieval methods.
Innovation Solution
The application of latent semantic analysis and computational linguistics principles to information retrieval systems, using techniques such as pointwise mutual-information weighting, morpheme-by-document matrices, and multilingual parallel corpora to improve text retrieval by accounting for linguistic relationships and semantic meanings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional information retrieval methods are used, then the system is simple to implement, but retrieval accuracy and relevance are limited
Solution Approach 1:
The patent introduces computational linguistics components as intermediaries between the query and document matching process. Specifically, morphological analyzers, semantic parsers, and linguistic knowledge bases act as mediators that transform conventional text into linguistically enriched representations, thereby improving retrieval accuracy while adding structured complexity to the system architecture
Solution Approach 2:
The patent combines conventional information retrieval techniques with computational linguistics methods to create a hybrid system. This composite approach integrates statistical matching with linguistic analysis, merging the simplicity of traditional IR with the precision of computational linguistics to achieve both improved accuracy and manageable complexity
2Reliability
If computational linguistics principles are integrated into information retrieval, then linguistic relationships and semantic meanings are better captured, but the system complexity increases
Solution Approach 1:
The patent segments the information retrieval process into distinct linguistic analysis stages: morphological analysis, syntactic parsing, semantic interpretation, and pragmatic evaluation. Each stage processes specific linguistic features independently, allowing the system to capture complex linguistic relationships while managing overall complexity through modular organization
Solution Approach 2:
The patent develops universal linguistic components that can be applied across different languages and domains. The morphological analyzer, semantic parser, and linguistic knowledge base are designed to handle multiple linguistic phenomena and language types, reducing the need for separate specialized systems and thereby controlling complexity while maintaining high retrieval relevance
Data Source
AI summary
A technique for information retrieval includes parsing a corpus to identify a number of wordform instances within each document of the corpus. A weighted morpheme-by-document matrix is generated based at least in part on the number of wordform instances within each document of the corpus and based at least in part on a weighting function. The weighted morpheme-by-document matrix separately enumerates instances of stems and affixes. Additionally or alternatively, a term-by-term alignment matrix may be generated based at least in part on the number of wordform instances within each document of the corpus. At least one lower rank approximation matrix is generated by factorizing the weighted morpheme-by-document matrix and/or the term-by-term alignment matrix.