Robot Instruction Generation From Video Demonstrations

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing robotics methods require manual programming by technical experts, making it difficult to adapt to changing environments and task variations, leading to inefficiency and limited deployment to repetitive tasks.

Innovation Solution

Utilizing generative artificial intelligence (Gen AI) techniques to generate robotic instructions from video demonstrations, enabling non-technical users to program robots by providing video examples of desired tasks, and translating them into precise robotic action plans.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Manufacturing precision

If manual programming by technical experts is used, then programming precision is maintained, but device complexity and time consumption increase

Engineering Contradiction:

Improveprogramming precisionVSAvoidtime consumption

Core Design Contradiction:

Manufacturing precisionVSLoss of time

Solution Approach 1:

The system uses video demonstrations as copies of desired robot behaviors. Instead of manually programming each action, the system captures real-world demonstrations and translates them into robotic instructions, allowing rapid replication of tasks without expert intervention

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent replaces manual mechanical programming with an automated AI-based system that processes video data and generates control instructions automatically, substituting human expert labor with an intelligent automated system

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Reliability

If manual programming by technical experts is used, then task execution reliability is maintained, but ease of operation deteriorates

Engineering Contradiction:

Improvetask execution reliabilityVSAvoidease of operation

Core Design Contradiction:

ReliabilityVSEase of operation

Solution Approach 1:

The system enables non-expert users to program robots themselves by recording demonstrations and automatically converting them into executable instructions, allowing the end-user to serve their own programming needs without external expert assistance

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The AI processing system acts as an intermediary that translates simple video demonstrations into reliable robotic instructions, bridging the gap between user-friendly input and reliable task execution

Inventive Principle:

Principle #24Intermediary (Mediator)

3Stability of the object's composition

If traditional robotics methods are used, then system stability is maintained, but adaptability to changing environments deteriorates

Engineering Contradiction:

Improvesystem stabilityVSAvoidadaptability to changing environments

Core Design Contradiction:

Stability of the object's compositionVSAdaptability or versatility

Solution Approach 1:

The system dynamically adapts to different tasks and environments by processing new video demonstrations in real-time, allowing the robot to learn and adjust its behavior based on observed actions rather than relying on pre-programmed static instructions

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS20260061622A1Method and system for generating robotic instructions

Publication Date: 2026.03.05 ACCENTURE GLOBAL SOLUTIONS LTD

AI summary

A computer-implemented method to generate robotic instructions is disclosed. The method may include receiving video data demonstrating one or more tasks and text data related to the one or more tasks. Further, the method may include encoding the video data and the text data, wherein the encoding is generated using at least one cross-attentional transformer. The method also includes receiving image data providing environmental data for at least one robotic task. Furthermore, the method may include encoding vision data corresponding to the image data. Consequently, the method may include generating robotic instructions based upon the video data, the text data and the vision data that was encoded.