Natural-Language Robot Assembly Control for Precision Skill Switching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generalist robotic policies struggle to achieve high-precision tasks in industrial assembly due to the need for intricate factors like contact engagement, friction handling, and refined motor skills, which require additional sensory data for enhanced precision.
Innovation Solution
A globally robust and locally precise control policy using Large Language Models (LLMs) that dynamically switch between specialized skills, integrating sensory data for precise robotic operations, and a centralized controller that directs robots based on natural language prompts and image inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a generalist policy is used to control robots in various environments, then the robot can adapt to diverse tasks, but the precision and accuracy required for intricate industrial assembly tasks (such as insertion, contact engagement, friction handling) are insufficient
Solution Approach 1:
The control system is segmented into a global control policy (LLM-based) that handles high-level decision-making and task planning, and multiple specialized skill models that handle specific precision tasks. Each skill model is independently trained for particular operations (e.g., insertion, grasping), allowing the system to switch to the appropriate specialized model when high precision is required, while maintaining overall adaptability through the global policy.
Solution Approach 2:
The LLM-based global control policy acts as an intermediary between the high-level task description and the low-level specialized skill models. It processes natural language instructions, determines the current state, and dynamically selects and switches between appropriate skill models, thereby coordinating between general adaptability and specific precision requirements.
2Manufacturing precision
If additional sensory data (force, torque measurements) is integrated to enhance precision, then the accuracy for intricate tasks improves, but the system complexity increases
Solution Approach 1:
Instead of uniformly integrating additional sensory data across all control operations, the system applies enhanced sensing (force, torque measurements) only locally within specific skill models that require high precision for particular tasks. This allows precision enhancement where needed while keeping the overall system simpler for tasks that do not require such complexity.
3Device complexity
If a centralized controller processes all control decisions, then the system architecture is simplified, but the computational load and processing time increase for complex assembly sequences
Solution Approach 1:
The centralized controller is segmented into a global policy component that handles high-level decision-making and multiple independent skill model components that handle specific task executions. This segmentation allows parallel processing of different skill operations, reducing overall processing time while maintaining a relatively simple centralized architecture through the use of independent, modular skill models.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented of determining actions for controlling a robot, in particular an assembly robot, comprising the steps of: Receiving a first and second input, wherein the first input (101) is a sentence describing an action which should be carried out by the robot, wherein the second input (102) is an image of a current state of an environment of the robot. Feeding the first input (100) into a first machine learning model (101) and feeding the second input (102) into a second machine learning model (103), wherein the first and second machine learning models (101,103) are configured to determine tokens for their respective inputs. Feeding the tokens into a third machine learning model (104), wherein the third machine learning model (104) outputs two outputs, wherein the first output is a switch for incorporating specialized skill networks and the second output are actions (106).