Map-Reduce Rule Engine for Product Attribute Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The variability in product attribute data provided by third-party sellers on marketplaces like Walmart.com makes it challenging to normalize and compare products effectively, requiring a systematic approach to standardize attributes across different formats and schemas.
Innovation Solution
A method and system for normalizing product attributes using a rule engine and Map-Reduce architecture, which processes large datasets to standardize attributes according to a canonical schema, involving the definition of normalization rules and their application across clusters of non-normalized attributes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If normalization rules are applied to standardize product attributes, then attribute consistency and searchability are improved, but processing time and computational complexity increase
Solution Approach 1:
The normalization process is divided into multiple passes, with each pass handling specific aspects of attribute standardization. This segmentation allows the system to process attributes in manageable chunks, improving overall efficiency while maintaining comprehensive normalization coverage.
Solution Approach 2:
Normalization rules are pre-defined and stored in a rule engine before processing begins. This preliminary preparation allows the system to quickly apply standardization transformations without performing complex analysis during the main processing operation, reducing processing time while ensuring consistent attribute normalization.
2Measurement precision
If comprehensive normalization rules are implemented to handle all attribute variations, then data quality improves, but system complexity increases
Solution Approach 1:
The rule engine is designed as a universal system that can handle multiple attribute types and normalization scenarios through a single unified architecture. This multi-functional approach allows comprehensive data quality improvement without proportionally increasing system complexity, as the same core mechanisms apply across different attribute domains.
Solution Approach 2:
A rule engine acts as an intermediary layer between the raw attribute data and the canonical schema. This intermediary component encapsulates the complexity of normalization logic, presenting a simplified interface that maintains high data quality while hiding the underlying computational complexity from the rest of the system.
3Ease of operation
If attribute normalization is performed on all product records, then search and comparison effectiveness improve, but processing throughput decreases
Solution Approach 1:
The normalization process operates in periodic batches rather than continuously processing all records at once. This batch processing approach allows the system to maintain high throughput by processing manageable quantities of attributes in parallel, while still achieving comprehensive normalization for search and comparison effectiveness.
Solution Approach 2:
The system processes attributes across multiple dimensions simultaneously using parallel computing architectures. By distributing the normalization workload across multiple processing units and leveraging map-reduce patterns, the system achieves both high throughput and comprehensive normalization coverage.
Data Source
AI summary
Product records having attributes according to various schema are normalized such that the attributes thereof conform to a canonical schema. Attributes for large numbers of product records are normalized according to a map-reduce framework in which mappers are defined but no reducers. Rules are implemented by a rule engine that is instantiated one time per VM of the map-reduce framework. Likewise, each rule may be implemented by only one object instance per rule engine instance. Generation of rules by analysts may be facilitated by defining an attribute hierarchy. A rule defined for a parent attribute may be presented as a default rule for a child attribute and either accepted or revised. Attributes may be clusters and proposed rules generated. Proposed rules may then be accepted or rejected by an analyst.