Extreme Classification Using Graph Convolution and Label Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional classification systems face challenges in providing accurate recommendations when the number of candidate classes is extremely large, such as millions, as they often lack sufficient performance in both training and prediction stages.
Innovation Solution
The development of an extreme classification system that generates a set of classifiers based on a joint graph of documents and labels, using graph convolution and label attention to create multi-dimensional vector representations, allowing for the annotation of documents with the most relevant subset of labels from an extremely large set.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of candidate classes is increased to millions to improve recommendation accuracy, then the accuracy of recommendations is improved, but the performance in training and classification deteriorates
Solution Approach 1:
The patent segments the extremely large set of candidate classes into multiple clusters or groups. Instead of treating all million+ classes uniformly, the system divides them into manageable segments that can be processed separately, enabling efficient training while maintaining high recommendation accuracy across the entire large-scale class space.
Solution Approach 2:
The patent introduces intermediary structures such as cluster centers, prototype vectors, or intermediate representation layers that mediate between the input features and the extremely large number of candidate classes. These intermediaries reduce the direct computational burden while preserving the ability to accurately classify across millions of classes.
2Measurement precision
If the number of candidate classes is increased to millions to improve recommendation accuracy, then the accuracy of recommendations is improved, but the system complexity increases
Solution Approach 1:
The patent segments the extremely large set of candidate classes into multiple clusters or groups. Instead of treating all million+ classes uniformly, the system divides them into manageable segments that can be processed separately, enabling efficient training while maintaining high recommendation accuracy across the entire large-scale class space.
Solution Approach 2:
The patent transforms the problem from directly handling millions of discrete classes into a continuous or lower-dimensional space through techniques such as embedding layers, prototype-based clustering, or hierarchical organization. This dimensional transformation reduces system complexity while preserving the ability to distinguish among extremely fine-grained classes.
3Device complexity
If traditional classification systems are used with thousands of candidate classes, then the system complexity is manageable, but the recommendation accuracy deteriorates due to insufficient candidate classes
Solution Approach 1:
The patent employs dynamic mechanisms such as adaptive clustering, online learning, or dynamic prototype selection that allow the system to efficiently handle extremely large numbers of candidate classes. These dynamic approaches enable the system to adapt to the scale of the problem rather than being constrained by static architectural limitations.
Solution Approach 2:
The patent changes key parameters of the classification system, such as introducing continuous class representations, modifying the loss function for extreme classification, or adjusting the dimensionality of feature vectors. These parameter changes enable the system to achieve high accuracy with millions of classes while maintaining manageable complexity through mathematical transformations.
Data Source
AI summary
Systems and methods are provided for learning classifiers for annotating a document with predicted labels under extreme classification where there are over a million labels. The learning includes receiving a joint graph including documents and labels as nodes. Multi-dimensional vector representations of a document (i.e., document representations) are generated based on graph convolution of the joint graph. Each document representation varies an extent of reliance on neighboring nodes to accommodate context. The document representations are feature-transformed using a residual layer. Per-label document representations are generated from the transformed document representations based on neighboring label attention. A classifier is trained for each of over a million labels based on joint learning using training data and the per-label document representation. The trained classifier performs highly efficiently as compared to other classifiers trained using disjoint graphs of documents and labels.