Semantic Embedding Deduplication for Browse Shelf Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems fail to effectively deduplicate query to assortment pages, leading to overlap with browse pages and inefficiencies in search engine crawling, as conventional models are insensitive to semantic scope differences.
Innovation Solution
A system employing a semantic embedding generation machine learning model and query classification model to generate shelf and keyword embeddings, determining similarity scores, and filtering out keywords with high similarity to existing browse shelves, thereby preventing duplicate topic page generation and enabling crawling by search engines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If new topic pages are generated based on keyword queries, then website visibility and traffic are improved, but overlap with existing browse pages occurs and resources are wasted
Solution Approach 1:
The system performs preliminary deduplication analysis before generating topic pages by comparing keyword embeddings with existing browse shelf embeddings. This preliminary check prevents duplicate page generation in advance, avoiding resource waste while maintaining website visibility and traffic benefits.
2Speed
If conventional keyword classification models are used, then processing speed is maintained, but semantic scope differences are not detected leading to deduplication failures
Solution Approach 1:
The system transforms the keyword and browse shelf names into embedding vectors using a pre-trained language model, changing the parameter representation from raw text to dense vector space. This enables precise semantic similarity detection through cosine similarity calculation while maintaining efficient processing speed.
3Measurement precision
If semantic embedding models are used for deduplication, then deduplication accuracy is improved, but computational complexity and processing time increase
Solution Approach 1:
The semantic embedding model is pre-trained offline before deployment. During runtime, only embedding generation and cosine similarity calculation are performed, which are computationally efficient operations. This preliminary preparation reduces online computational complexity while maintaining high deduplication accuracy.
Solution Approach 2:
The system uses cosine similarity to compare embeddings, which is a computationally efficient operation compared to full semantic analysis. This copying approach of using simple mathematical operations on pre-generated embeddings reduces computational complexity while preserving deduplication accuracy.
Data Source
AI summary
A method including generating, using a semantic embedding generation machine learning model, one or more respective shelf embedding vector representations for each of one or more browse shelves based on a respective shelf name for the each of the one or more browse shelves. The method also can include obtaining a keyword. The method additionally can include generating, using the semantic embedding generation machine learning model, a keyword embedding vector representation based on the keyword. The method further can include determining a respective similarity score between the keyword embedding vector representation and each of the one or more respective shelf embedding vector representations for each of the one or more browse shelves. The method additionally can include determining whether any of the respective similarity scores for the one or more respective shelf embedding vector representations across the one or more browse shelves exceed a predetermined threshold value. When any of the respective similarity scores for the one or more respective shelf embedding vector representations across the one or more browse shelves exceeds a predetermined threshold value, the method further can include filtering out the keyword. When none of the respective similarity scores for the one or more respective shelf embedding vector representations across the one or more browse shelves exceed a predetermined threshold value, the method additionally can include generating a new topic page using the keyword. Other embodiments are described.