MAC Unit Visual Reasoning for Graphical Illustrations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems fail to accurately reason over visual percepts from statistical charts, particularly for visually impaired individuals and automated systems, as they lack the ability to understand and analyze information from graphical illustrations effectively, relying on dataset biases and memorization rather than true reasoning capabilities.
Innovation Solution
The implementation of a processor-based method using a Memory, Attention, and Composition (MAC) unit, which is an end-to-end differentiable neural network, to perform attention-based reasoning operations on graphical illustrations, generating a concatenated vector to predict textual answers and probability distributions, leveraging pre-trained Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) for feature extraction and question embeddings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning is used to improve visual perception accuracy, then perception accuracy is improved, but the ability to reason over visual percepts deteriorates
Solution Approach 1:
The system segments the visual reasoning task into multiple sub-questions that are processed sequentially. The MAC unit divides complex questions about graphical illustrations into smaller reasoning steps, allowing the system to maintain high perception accuracy while building reasoning capabilities through step-by-step analysis of visual percepts.
Solution Approach 2:
The patent introduces an intermediary reasoning layer between visual perception and final answers. The MAC unit acts as a mediator that takes visual features from deep learning models and natural language questions, then performs iterative attention-based reasoning to generate answers. This intermediary layer enables reasoning over visual percepts without compromising the accuracy of the underlying perception system.
2Adaptability or versatility
If attention-based reasoning operations are performed iteratively on sub-questions, then reasoning capability is improved, but computational complexity deteriorates
Solution Approach 1:
The iterative attention-based reasoning process is segmented into discrete time steps, each handling a specific sub-question. By breaking down complex reasoning into smaller temporal segments, the system manages computational complexity while maintaining strong reasoning capabilities across multiple processing stages.
Solution Approach 2:
The system employs dynamic attention mechanisms that adaptively allocate computational resources across different sub-questions and visual objects. The attention weights are dynamically adjusted at each time step based on the current reasoning state, allowing the system to focus computational effort where most needed rather than uniformly processing all elements.
3Measurement precision
If MAC unit with recurrent MAC cells is used for end-to-end differentiable reasoning, then reasoning accuracy is improved, but system complexity deteriorates
Solution Approach 1:
The patent merges multiple functions into the MAC unit: memory storage, attention computation, and composition of visual-language representations are combined in a single recurrent architecture. This unified approach achieves high reasoning accuracy through end-to-end differentiability while managing system complexity by integrating rather than separating these critical components.
Solution Approach 2:
The system replaces traditional mechanical or rule-based reasoning approaches with a differentiable neural network-based MAC unit. This substitution enables gradient-based optimization for improved reasoning accuracy while the modular recurrent cell structure helps manage the inherent complexity of the neural network approach.
Data Source
AI summary
This disclosure relates generally to intelligent visual reasoning over graphical illustrations using a MAC unit. Prior arts use visual attention to map particular words in a question to specific areas in an image to memorize the corresponding answers, thereby resulting in a limited capability to answer questions of a specific type. The present disclosure incorporates the MAC unit to enable reasoning capabilities and accordingly attend to an area in the image to find the answer. The present disclosure therefore allows generalizing over a possible set of questions with varying complexities so that an unseen question can also be answered correctly based on the reasoning methods that it has learned. The system and method of the present disclosure can be used for understanding of visual information when processing documents like business reports, research papers, consensus reports etc. containing charts and reduce the time spent in manual analysis.