Robot Instruction Learning for Ambiguous Object Grasping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robots struggle to accurately interpret ambiguous language instructions and select appropriate objects in complex environments, such as factories, where objects are stacked randomly and instructions are unclear, leading to difficulties in planning grasping trajectories.
Innovation Solution
A learning device that includes a model definition unit, learning unit, weight storage unit, and inference unit to predict the intention behind ambiguous instructions and identify the intended object, using machine learning techniques like convolutional neural networks and recurrent neural networks to generate operation commands for the robot.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a robot uses general image recognition methods to recognize objects, then it can recognize objects with learned labels, but it cannot recognize objects with unlearned labels or abstract instructions
Solution Approach 1:
The patent introduces a reference point as an intermediary element that mediates between the ambiguous instruction and the target object. The reference point serves as a bridge that the robot can identify using general image recognition, and then use to locate the actual target object even when the instruction is ambiguous or the object label is unlearned. This resolves the contradiction by allowing the robot to handle unlearned objects while maintaining recognition accuracy through the intermediary reference point.
2Ease of operation
If a robot follows one-to-one instruction mapping, then it can execute clear instructions, but it cannot handle ambiguous instructions with multiple possible objects
Solution Approach 1:
The patent segments the instruction interpretation process into multiple stages: first identifying reference points, then using those reference points to locate target objects. This segmentation allows the robot to break down complex ambiguous instructions into manageable steps, improving ease of operation while controlling device complexity through structured processing.
Solution Approach 2:
The reference point acts as an intermediary that simplifies the handling of ambiguous instructions. Instead of directly mapping ambiguous instructions to multiple possible objects, the robot first identifies the reference point and then uses it to determine the target object, reducing instruction interpretation complexity.
3Loss of information
If a robot lacks reference points for spatial relationships, then it can process simple instructions, but it cannot determine instructed objects using spatial descriptors like 'that' or 'object on the right'
Solution Approach 1:
The patent enables the robot to self-determine reference points based on the current situation and past experience without external assistance. The robot autonomously identifies suitable reference points from the environment that can serve as spatial anchors, thereby recovering spatial context information while avoiding the difficulty of manual reference point detection or configuration.
Data Source
AI summary
A learning device comprises: a learning unit that, using a model read from a model definition unit in accordance with a desired action of a robot, learns measurement information, captured images, and instruction information; a weight storage unit that stores a weight for a model learned by the learning unit; and an inference unit that, when instruction information is newly input, gives a weight read from the weight storage unit to a model read from the model definition unit, predicts an intention of the instruction information by applying, to the model, newly measured measurement information, a newly captured image, and newly input instruction information to the robot, infers a desired action of the robot, and an object which is the target of the desired action, and predicts a drive command for causing the robot to perform the desired action according to the intention of the predicted instruction information.