Graph-Based Data Partitioning for Product Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated product classification systems face inefficiencies and inaccuracies due to random or manual partitioning of data, leading to overfitting and poor generalization, as they often rely on shared or similar data for training and testing, resulting in unreliable predictions for new data.
Innovation Solution
A computer-implemented method that dynamically partitions data into training, testing, and validation sets by representing samples as nodes on a graph, measuring relative similarity distances, and grouping them to minimize overlap, ensuring disjoint sets for improved model generalization and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If random splitting of data is used for training and testing, then the process is simple and fast, but the model suffers from overfitting and lacks generalization ability
Solution Approach 1:
The patent segments the dataset into multiple disjoint subsets based on feature similarity, creating distinct training, validation, and testing sets that do not share identical or highly similar samples. This segmentation prevents the model from memorizing specific samples while maintaining adequate representation of the data distribution.
Solution Approach 2:
The patent changes the partitioning parameter from random assignment to similarity-based assignment. By computing similarity metrics between data samples and assigning them to different sets based on their similarity relationships, the method transforms the partitioning strategy to eliminate overlap while preserving data characteristics.
2Measurement precision
If identical or highly similar components are shared between training and testing datasets, then the model achieves high accuracy on training data, but it relies on memorization rather than true learning
Solution Approach 1:
The patent extracts highly similar samples from the training dataset and places them into separate validation or testing sets. By taking out these duplicate or near-duplicate samples, the method ensures that the model cannot rely on memorization of specific training examples, forcing it to learn generalizable features instead.
Solution Approach 2:
The patent introduces similarity computation and clustering algorithms as intermediary processes between data collection and model training. These intermediaries analyze the relationships between samples and systematically separate similar instances across different datasets, preventing direct memorization while maintaining balanced representation.
3Reliability
If the model is trained on a diverse training dataset, then it improves generalization ability, but the data partitioning process becomes more complex and time-consuming
Solution Approach 1:
The patent performs preliminary computation of similarity metrics and clustering assignments before the actual model training begins. By pre-processing the data to establish similarity relationships and assign samples to appropriate sets in advance, the method eliminates the need for complex iterative adjustments during training, reducing overall computational complexity.
Solution Approach 2:
The patent replaces manual or heuristic data partitioning methods with automated similarity-based assignment algorithms. This substitution uses computational mathematics (similarity metrics, clustering) to systematically divide the dataset, replacing complex manual judgment processes with objective, repeatable computational procedures.
Data Source
AI summary
A data partitioning system receives an input dataset for e-commerce products, each sample containing attributes and associated values for each product including at least an image; represents each sample as a node on a graph to provide a graph of nodes for the dataset; measures a relative similarity distance between each pair of nodes based on comparing at least image values for the attributes; determines for each pair of nodes whether they are related if the similarity distance between them is below a defined threshold, and if related, generate an edge between them on the graph; group the connected nodes into a first or a second group such that the grouped nodes have no edges connecting them to nodes in the other group and have a shortest relative similarity distance with each other. The groups are used as training dataset and testing data sets for a supervised machine learning classifier.