Semantic Distance Calculation Using Latent Dirichlet Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rapid increase in data production from unstructured natural language makes it difficult to understand and classify large amounts of information, with previous methods failing to effectively calculate semantic distances between subjects to determine their relevance and cluster them accurately.
Innovation Solution
A computer-implemented method using latent Dirichlet allocation to extract subjects from documents, generate classification lists, calculate semantic distances, and create a distance matrix, while applying natural language processing techniques to normalize relevance values and exclude irrelevant subjects, thereby determining the semantic proximity of subjects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If latent Dirichlet allocation is used to extract subjects from documents, then subject classification accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent segments the document processing into multiple stages: first extracting subjects using LDA, then generating classification lists, calculating semantic distances, and creating distance matrices. This segmentation allows each component to be optimized independently and reduces overall computational burden by processing data in manageable chunks rather than monolithically.
Solution Approach 2:
The patent performs preliminary subject extraction and classification using LDA before computing semantic distances. By pre-processing the documents to identify and rank subjects beforehand, the system reduces computational complexity during the distance calculation phase, as the foundation of subject-relation data is already established.
2Measurement precision
If semantic distance calculation is performed for all subject pairs, then subject clustering accuracy is improved, but processing time increases
Solution Approach 1:
The patent calculates semantic distances locally for subject pairs that are relevant to the query or document set, rather than uniformly for all possible pairs. By determining which subject pairs need distance calculation based on their relevance to the input data, the system reduces processing time while maintaining clustering accuracy for the most important relationships.
Solution Approach 2:
The patent computes semantic distances for a subset of subject pairs that are most likely to be relevant, rather than exhaustively calculating for all pairs. This partial action approach maintains sufficient clustering accuracy by focusing computational resources on the most promising subject relationships, thereby reducing overall processing time.
3Measurement precision
If relevance threshold filtering is applied to subjects, then information retrieval precision is improved, but the number of retrieved subjects decreases
Solution Approach 1:
The patent dynamically adjusts relevance thresholds based on the specific query and document set characteristics. By changing the threshold parameter adaptively rather than using a fixed value, the system maintains high retrieval precision while ensuring sufficient quantity of relevant subjects are returned. The threshold is optimized for each specific retrieval task rather than applied uniformly.
Data Source
AI summary
Embodiments provide a system and method for semantic distance calculation. The method can involve receiving a plurality of documents having a set of subjects extracted through the use of latent dirichlet allocation; for each document in the plurality of documents, generating a classification list comprising a ranking of the one or more subjects based on the relevance of each subject to the document; for each classification list, calculating the semantic distance between each subject present on the classification list; aggregating the plurality of classification lists; and creating a distance matrix containing the relative semantic distances between each member of the set of subjects.