Synonym Generation via Co-occurrence and Closeness Scores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for identifying synonyms in information retrieval systems are inadequate, as they are often language-restricted, costly, or inaccurate, particularly for rare words and languages, and fail to effectively expand user queries to retrieve relevant documents.
Innovation Solution
A system that automatically generates synonyms by determining co-occurrence frequencies and closeness scores of words in documents, considering correlations and word-form rules, to identify potential synonyms and generate altered queries for improved search results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If pre-constructed synonym information from a thesaurus or structured lexical database is used, then synonym identification is straightforward, but the system becomes expensive to construct and restricted to one language
Solution Approach 1:
The system automatically generates synonym information by analyzing query substitution patterns from actual user behavior, eliminating the need for manual thesaurus construction and enabling automatic adaptation to multiple languages and domains without human intervention
Solution Approach 2:
The system pre-processes query sessions to collect and analyze substitution patterns, building a synonym database from actual usage data before it is needed for search expansion, allowing the system to be ready to handle diverse languages and domains immediately
2Reliability
If query data is used to identify synonyms through term substitution frequency, then the system adapts to actual usage, but there is not enough query data for rare words and rare languages
Solution Approach 1:
The system uses a unified statistical model that works across all languages and word frequencies, applying the same substitution pattern analysis methodology universally whether dealing with common English words or rare foreign terms, eliminating the need for language-specific or frequency-specific processing
3Device complexity
If stemming relationships are used to identify synonyms, then the system is computationally simple, but accuracy is poor as many words with the same stem have different meanings
Solution Approach 1:
The system uses actual query substitution feedback from user behavior to validate and refine synonym identification, comparing predicted synonyms against real-world replacement patterns to ensure accuracy while maintaining computational efficiency through statistical modeling
4Reliability
If user queries are expanded to include synonyms, then search relevance improves, but the query processing complexity increases
Solution Approach 1:
The system applies synonym expansion selectively based on query characteristics, applying the full synonym substitution process only when beneficial while using simpler methods for straightforward queries, avoiding unnecessary processing complexity while maintaining relevance improvement where needed
Data Source
AI summary
One embodiment of the present invention provides a system that automatically generates synonyms for words from documents. During operation, this system determines co-occurrence frequencies for pairs of words in the documents. The system also determines closeness scores for pairs of words in the documents, wherein a closeness score indicates whether a pair of words are located so close to each other that the words are likely to occur in the same sentence or phrase. Finally, the system determines whether pairs of words are synonyms based on the determined co-occurrence frequencies and the determined closeness scores. While making this determination, the system can additionally consider correlations between words in a title or an anchor of a document and words in the document as well as word-form scores for pairs of words in the documents.