Machine Operation Visual Question Answering via Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine operation systems lack effective visual question answering capabilities to accurately interpret digital images and make informed decisions based on sensor data, particularly in environments requiring immediate actions like traffic scenarios.
Innovation Solution
A computer-implemented method and device that utilize an object detector and classifier to analyze digital images from sensors, providing structured representations of questions to an answer set programming program, determining scores and thresholds to generate rules and constraints, and using an answer set solver to determine actions for machine operation, such as stopping in response to traffic signs or pedestrian attributes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If visual question answering capabilities are added to machine operation systems, then decision-making accuracy is improved, but device complexity increases
Solution Approach 1:
The system segments the visual question answering task into distinct functional modules: an object detector for identifying objects in digital images, a classifier for categorizing detected objects, and an answer set solver for determining actions based on classified objects. This segmentation allows each module to specialize in a specific aspect of the task, improving overall decision-making accuracy while managing system complexity through modular architecture.
Solution Approach 2:
The patent introduces structured representations of questions as an intermediary layer between the visual input and the decision-making process. These structured representations translate natural language questions into a format that the answer set solver can process, acting as a mediator that bridges the gap between visual question answering capabilities and machine operation commands, thereby improving reliability without directly increasing core system complexity.
2Measurement precision
If multiple classes and scores are analyzed for object classification, then measurement precision is improved, but loss of time increases
Solution Approach 1:
The system performs preliminary classification by analyzing multiple scores for different object classes simultaneously. The answer set solver evaluates multiple possible classifications in advance and selects the most appropriate one based on the structured question representation. This preliminary action allows the system to maintain high measurement precision through multi-class analysis while reducing processing time by preparing classification options beforehand rather than sequentially.
Data Source
AI summary
A device and method for operating a machine. The method comprises providing a digital image, providing a structured representation of a question, predicting with an object detector and depending on at least a part of the digital image an area of the digital image wherein an object is depicted in the digital image, predicting with a classifier and depending on at least a part of the digital image within the area a first score indicating a likelihood that the object is of a first class and a second score indicating a likelihood that the object is of a second class, providing at least one attribute value for the first class, adding to an answer set programming program a first rule comprising the at least one attribute value of the first class and/or a first constraint comprising the at least one attribute value of the first class.