LLM Ensemble Attribute Extraction for Noisy Catalog Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for attribute extraction from network catalogs require high-quality datasets and struggle with noisy and unstructured data, leading to inaccurate results.
Innovation Solution
An ensemble of large language models (LLMs) is used to generate preliminary attribute labels, which are combined using a weighted approach to produce a final label, leveraging structured latent variable models for optimal results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional extraction processes are used to extract attributes from catalog data, then the system requires high quality datasets for training and extraction, but actual catalog datasets are noisy and unstructured resulting in incorrect attribute extraction
Solution Approach 1:
The patent combines multiple large language models into an ensemble system where each model processes the input data independently and their predictions are aggregated through weighted combination. This merging of multiple models compensates for individual model weaknesses and handles noisy/unstructured data more robustly, resolving the contradiction between extraction accuracy and data quality requirements
Solution Approach 2:
The system dynamically adjusts the weights assigned to each LLM in the ensemble based on their performance characteristics and the specific attributes being extracted. By changing the parameter weights adaptively, the system optimizes extraction accuracy for different data conditions without requiring uniformly high-quality training datasets for all models
2Measurement precision
If an ensemble of multiple large language models is used to generate preliminary attribute labels, then the attribute extraction accuracy improves, but the system complexity increases
Solution Approach 1:
The patent segments the attribute extraction task by assigning different LLMs to handle different aspects or types of attribute prediction. Each model focuses on specific patterns in the data, and their outputs are combined to form the final attribution labels. This segmentation reduces the effective complexity each individual model must handle while maintaining high overall accuracy
3Reliability
If weighted combination of preliminary attribute labels is used to generate final labels, then the extraction reliability improves, but the processing time increases
Solution Approach 1:
The system implements a staged approach where LLMs generate preliminary attribute labels that capture the most important information, and the weighted combination process focuses on refining these partial results rather than generating complete labels from scratch. This partial action approach achieves reliable extraction with reduced processing time compared to full re-generation
Data Source
AI summary
Systems and methods of attribute extraction and labelling are disclosed. An input dataset is received and a plurality of preliminary attribute labels are generated for at least a first attribute of a first element in the input dataset. Each preliminary attribute label in the plurality of preliminary attribute labels is generated by one of a plurality of large language models (LLM). A final attribute label for the first attribute is generated based on a weighted combination of the plurality of preliminary attribute labels for the first attribute and a data structure representative of the first element is updated to include the final attribute label for the first attribute.