Multi-decoder neural architecture for label partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing encoder-decoder models face challenges in multi-label classification due to the large number of labels, which leads to increased learning effort, prolonged training times, and decreased inference accuracy, as well as numerical instabilities such as excessive rounding error, underflow, and overflow.
Innovation Solution
The proposed solution involves a novel multi-decoder architecture that uses coarse-grained data categorization to partition the training corpus into semantic partitions, each with its own dedicated decoder. This approach minimizes the number of labels each decoder needs to predict, avoiding the need for additional preprocessing models and reducing training latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single decoder is used to handle all labels in multi-label classification, then the model can predict all labels, but the training time is prolonged and inference accuracy decreases due to the large number of labels
Solution Approach 1:
The patent divides the large label space into multiple coarse-grained semantic partitions, with each partition containing a subset of labels. Instead of using one decoder to handle all labels, multiple decoders are created, each responsible for predicting labels in a specific partition. This segmentation reduces the number of labels each decoder must process, thereby improving inference accuracy while reducing training time compared to a single decoder handling the entire label space.
2Productivity
If dimensionality reduction preprocessing is applied to reduce the number of labels, then the classification problem becomes more manageable, but additional training latency is introduced that is not concurrent to classifier training
Solution Approach 1:
The patent merges the data partitioning operation with the decoder architecture design. Rather than treating partitioning as a separate preprocessing step that requires additional training, the partitioning is integrated into the model structure itself through the multi-decoder architecture. This allows the classifier training to proceed concurrently with the partitioning logic already embedded in the architecture, eliminating additional training latency while managing the large label space effectively.
3Reliability
If a large number of labels are processed simultaneously, then comprehensive classification coverage is achieved, but numerical instabilities such as excessive rounding error, underflow, and overflow occur
Solution Approach 1:
The patent segments the large label space into multiple smaller partitions, with each partition containing a manageable subset of labels. Each decoder processes only the labels in its assigned partition, which significantly reduces the numerical computation burden compared to processing all labels simultaneously. This segmentation maintains comprehensive classification coverage across all labels while improving numerical stability by avoiding the excessive rounding errors, underflow, and overflow that occur when processing large numbers of labels in a single computation.
Data Source
AI summary
A pretraining computer generates a neural encoder and multiple partition decoders (PDs) for respective partitions of training inputs (TIs) in a training corpus. A training batch is generated that contains a mix of TIs from multiple partitions. For each TI in the batch, the neural encoder infers an encoding and, based on the partition of the TI, exactly one PD is used to decode the encoding, for which an individual loss is measured. The individual loss is combined into a batch loss that is based on the entire batch, and combined into a partition loss that is based on TIs only in the partition of the exactly one PD. After measuring losses for the batch, the batch loss is backpropagated into the neural encoder without backpropagating the batch loss into any PD. Into each PD is backpropagated a respective partition loss that is based on TIs only in the decoder's partition.