Dual-Modality Image Classification for Multi-Label and Zero-Shot Recognition

The patent merges visual embedding and textual embedding into a unified dual-modality classification framework. The classifier simultaneously processes both visual features (from image encoders) and textual features (from text encoders) through integrated transformer decoder layers, enabling the system to achieve high accuracy in multi-label and zero-shot scenarios by combining complementary information from both modalities rather than treating them separately.

The dual-modality classifier serves multiple functions within a single unified architecture: it performs single-label classification, multi-label classification, and zero-shot classification using the same core components (visual encoder, text encoder, transformer decoder). This universal design allows the system to handle diverse classification tasks without requiring separate specialized models, thereby improving accuracy across all scenarios while managing complexity through code reuse.

The computational process is segmented into distinct stages: visual embedding extraction (via image encoder), textual embedding extraction (via text encoder), correlation calculation between modalities, and progressive refinement through transformer decoder layers. Each stage processes specific information independently before integrating results, which manages computational complexity by breaking down the dual-modality fusion into manageable steps rather than attempting simultaneous processing of all data.

The system performs preliminary extraction of visual embeddings and textual embeddings before performing the complex correlation analysis and fusion operations. By pre-processing and organizing the data into structured embedding representations, the system reduces the computational burden during the actual fusion stage, as the preliminary structures guide subsequent correlation calculations and refinement operations more efficiently.