Attribute Tuple Normalization Using Clustering and Large Language Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing online concierge systems face challenges in automatically normalizing attribute tuples for items from multiple sources, leading to inefficiencies and errors in data processing and comparison, as conventional methods are time-consuming and impractical for large datasets.
Innovation Solution
An unsupervised clustering algorithm combined with a large language model is used to group and normalize attribute tuples, enabling automatic normalization and mapping to canonical forms, thereby facilitating efficient data management and comparison.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual human curation is used to normalize attribute tuples, then normalization accuracy can be maintained, but the process becomes time-consuming and infeasible for large datasets
Solution Approach 1:
The patent introduces an intermediary normalization module that acts as an automatic translation layer between diverse attribute tuple formats and a standardized schema. This module uses machine learning models and natural language processing to bridge the gap between different data sources without requiring manual human curation, thereby maintaining accuracy while dramatically reducing normalization time for large datasets
Solution Approach 2:
The patent replaces the mechanical human curation process with an automated computational system. The normalization module uses algorithms, machine learning models, and text processing techniques to perform attribute normalization automatically, substituting human manual work with automated mechanical processes that can handle large-scale data efficiently
2Reliability
If conventional manual methods are used for attribute normalization, then data quality can be ensured, but the process is not feasible at the scale required by online concierge systems with many items
Solution Approach 1:
The normalization module is designed to be self-sufficient, automatically processing attribute tuples from multiple data sources without requiring external human intervention. The system uses built-in machine learning models and normalization rules to autonomously handle data standardization, enabling it to serve itself at scale while maintaining data quality through automated validation and error handling mechanisms
Solution Approach 2:
The patent transforms the normalization process by changing key parameters from manual operation to automated computational processing. This includes adjusting the scale of operation from individual tuple processing to batch processing of large datasets, modifying the time parameter from hours/days to minutes, and changing the resource allocation from human labor to computational power, thereby achieving both high throughput and maintained data quality
Data Source
AI summary
A computer system uses clustering and a large language model (LLM) to normalize attribute tuples for items stored in a database of an online system. The online system collects attribute tuples, each attribute tuple comprising an attribute type and an attribute value for an item. The online system initially clusters the attribute tuples into a first plurality of clusters. The online system generates prompts for input into the LLM, each prompt including a subset of attribute tuples grouped into a respective cluster of the first plurality. Based on the prompts, the LLM generates a second plurality of clusters, each cluster including one or more attribute tuples that have a common attribute type and a common attribute value. The online system maps each attribute tuple to a respective normalized attribute tuple associated with each cluster. The online system rewrites each attribute tuple in the database to a corresponding normalized attribute tuple.