AI Agent Task Planning via Scene Graph and Instruction Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI technologies for robots are limited in autonomous action planning and collaboration, relying on pre-defined rules and lacking the ability to generate detailed task plans using visual information and human instructions, especially when multiple AI agents need to work together.
Innovation Solution
A method for task planning that involves generating a scene graph using visual information and human instructions, creating a machine instruction set with relevance information, and requesting collaboration from nearby AI agents when necessary, utilizing an AI neural network trained on images, human instructions, and machine instruction sets to estimate the number of agents required for task completion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If simple commands are used for robot control, then ease of operation is improved, but task planning capability deteriorates
Solution Approach 1:
The patent segments human instructions into multiple machine instruction sets with different levels of detail. The scene graph is divided into objects with relevance scores, allowing the system to selectively generate detailed plans only for relevant objects. This segmentation enables the robot to maintain simple operation interfaces while achieving sophisticated autonomous task planning through hierarchical instruction breakdown.
2Productivity
If detailed task plans are generated autonomously, then productivity is improved, but device complexity deteriorates
Solution Approach 1:
The patent introduces a scene graph as an intermediary data structure between human instructions and machine execution. The scene graph captures spatial relationships and object relevances, serving as a mediator that translates natural language instructions into structured machine instruction sets. This intermediary layer enables autonomous detailed task planning without requiring complex direct translation mechanisms, thereby improving productivity while managing system complexity.
3Adaptability or versatility
If collaboration between multiple AI agents is implemented, then adaptability is improved, but loss of information deteriorates
Solution Approach 1:
The patent creates a universal scene graph representation that can be shared and understood by multiple AI agents. The scene graph serves as a common language that captures the environment state, object relationships, and task requirements in a standardized format. This universal representation enables different agents to collaborate effectively on complex tasks while maintaining consistent information about the environment and task goals, thereby improving adaptability without significant information loss.
Data Source
AI summary
Disclosed herein is a method for task planning for collaboration of artificial intelligence (AI) agents. The method includes generating a scene graph using an image acquired by an AI agent and a human instruction and generating a machine instruction set for objects in the scene graph, and the scene graph includes relevance information between each of the objects in the scene graph and the human instruction.