Product Entity Extraction from User Content Using Scoring and Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems fail to accurately identify products mentioned in user-generated content, such as comments and social media posts, due to colloquial terms, lack of explicit product references, and context-dependent information.
Innovation Solution
A system and method that utilize a combination of machine learning algorithms, kgram rules, user rules, URL rules, and a suffix-tree-like data structure to extract product entities from user-generated content by identifying product types, attributes, and candidate products, and applying pruning and scoring mechanisms to select the most likely product.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional search engines or analytic software are used to identify products in user-generated content, then the system is simple to operate, but the product identification accuracy deteriorates due to colloquial terms and implicit references
Solution Approach 1:
The system segments the product identification process into multiple independent modules: product type detection, attribute detection, candidate product identification, pruning, and scoring. Each module handles a specific aspect of the analysis, allowing the complex task to be divided into manageable components that can be processed sequentially
Solution Approach 2:
The system introduces intermediary data structures including a product type taxonomy, attribute dictionaries, and candidate product lists that bridge the gap between raw user-generated content and final product identification. These intermediaries facilitate the transformation of unstructured text into structured product information
2Measurement precision
If the system uses multiple machine learning algorithms and rule-based approaches to improve product identification accuracy, then measurement precision improves, but the computing resources and time required increase
Solution Approach 1:
The system performs preliminary actions by pre-building product type taxonomies, attribute dictionaries, and candidate product lists before processing user-generated content. This pre-computation reduces the computational burden during actual product identification, as the system can leverage these pre-organized data structures rather than creating them from scratch for each analysis
Solution Approach 2:
The system applies partial action by using pruning mechanisms that eliminate candidate products at intermediate stages based on confidence thresholds and contextual relevance. This allows the system to process only the most promising candidates through the full analysis pipeline, reducing overall computational requirements while maintaining high identification accuracy
3Productivity
If the system processes only explicit product references, then the processing speed is fast, but the quantity of identified products decreases due to missed implicit references
Solution Approach 1:
The system dynamically adjusts its analysis depth and candidate generation strategies based on the characteristics of the user-generated content being processed. For content with clear product references, the system can use faster, more direct identification methods, while for content with implicit references, it activates more comprehensive analysis routines to ensure product detection
Data Source
AI summary
Systems and methods are disclosed herein for extracting products referenced in a document. A document is analyzed to identify a product type that is referenced in the document. Attributes are extracted from the document. A set of candidate products are identified corresponding to the extracted attributes. A score is calculated for the candidate products and the products are further selected or filtered based on the score, whitelist rules, and blacklist rules in order to identify one or more inferred products referenced by the document. The whitelist and blacklist rules may take as inputs a domain, a user identifier, and keywords included in the document. A set of sufficient attributes may be identified for each product type. Selection of a candidate product may be based at least in part on the document including all of the attributes in the set of sufficient attributes.