The invention discloses a chart visual question-answering method based on dynamic routing and low-rank mixing and a related device, and relates to the technical field of visual question-answering, and the method comprises the steps: obtaining
original data containing to-be-processed charts and text questions, and generating corresponding answers through the
processing of a pre-trained dual-channel fusion model. In the model, a visual
encoder is provided to extract semantic features of complex
layout of the chart, and a text
encoder accurately converts text problem vectors, so that the
feature matching degree of the chart and the problem is improved; the chart mixed connection structure dynamically fuses visual features of the
data graph and the data table, and the cross-type chart adaptability is enhanced; according to the chart low-rank mixed structure, related vectors are weighted and fused through a LoRA routing network, low-rank
adaptation and dynamic fusion are achieved in combination with output of a large
language model layer, training parameters are reduced, and the zero sample question and answer ability is improved; the two-stage training strategy reduces the loss of the
video memory. According to the method, the universality, accuracy and zero sample performance of chart visual
questions and answers are effectively improved, and the training cost is reduced.