Mobile Manipulation Teaching via Image Transform Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current robotic systems struggle to perform tasks when the starting point or orientation of objects differs from the programmed or taught task, as they are specific to a fixed orientation and starting point, limiting their adaptability in diverse and unstructured environments.
Innovation Solution
A method and system that utilize virtual reality to teach robotic devices parameterized behaviors, allowing them to map task images to teaching images and update parameters based on relative transforms, enabling the robotic device to perform tasks in various environments and orientations by capturing and matching pixel descriptors and dense neural network mappings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional programming methods are used to teach robotic devices specific tasks, then the robot can perform the taught task accurately, but it cannot adapt to variations in starting points, orientations, or environmental configurations
Solution Approach 1:
The system captures images of the teaching environment and creates a digital representation (task image) that can be mapped to the real-world task environment. This copying approach allows the robot to learn from visual representations rather than requiring complex programming for each scenario, enabling adaptation to different starting points and orientations through image mapping and descriptor matching.
Solution Approach 2:
The system extracts descriptors from images and computes relative transforms between teaching and task environments by comparing these descriptors. By changing the parameters (descriptors) extracted from visual data rather than hardcoding task parameters, the system achieves adaptability to environmental variations while keeping the programming approach relatively simple.
2Reliability
If the robotic device is programmed for specific orientations and starting points, then task execution is precise for those conditions, but the robot fails to perform the same task when environmental conditions differ
Solution Approach 1:
The system dynamically adapts to different environmental conditions by computing relative transforms between the teaching environment and the actual task environment. Instead of using fixed programmed coordinates, the robot dynamically calculates the transformation needed to map taught actions to the current environment based on descriptor matching, enabling reliable task execution across varying conditions.
Solution Approach 2:
The system uses feedback from image descriptors to determine the relative transform between teaching and task environments. By continuously comparing extracted descriptors and computing the transformation that best aligns them, the system receives feedback about environmental differences and adjusts its action parameters accordingly, ensuring reliable task execution in diverse settings.
3Ease of manufacture
If conventional task programming is used, then the robot can execute the specific taught task, but it requires extensive reprogramming for each new task or environmental configuration
Solution Approach 1:
The system captures images of the teaching environment and creates a reusable digital representation. Once the teaching image is captured and descriptors are extracted, this visual representation can be applied to multiple different task environments without requiring reprogramming. The copied image and its descriptors serve as a template that can be mapped to various real-world scenarios, significantly reducing teaching time.
Solution Approach 2:
The system performs preliminary action by capturing the teaching environment image and extracting descriptors in advance. This preliminary capture of the task specification in image form allows the system to quickly adapt to different environments by computing transforms rather than requiring time-consuming reprogramming for each new task or configuration.
Data Source
AI summary
A method for performing a task by a robotic device includes mapping a group of task image pixel descriptors associated with a first group of pixels in a task image of a task environment to a group of teaching image pixel descriptors associated with a second group of pixels in a teaching image based on positioning the robotic device within the task environment. The method also includes determining a relative transform between the task image and the teaching image based on mapping the plurality of task image pixel descriptors. The relative transform indicates a change in one or more of points of 3D space between the task image and the teaching image. The method also includes performing the task associated with the set of parameterized behaviors based on updating one or more parameters of a set of parameterized behaviors associated with the teaching image based on determining the relative transform.