Procedural Knowledge Graph for Instructional Video Planning

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing AI systems face challenges in constructing a logical sequence of action steps for instructional videos due to implicit causal constraints and variability in multiple feasible plans, despite leveraging various sources of information.

Innovation Solution

A computer-implemented method and system that utilize a Procedural Knowledge Graph (PKG) and Large Language Models (LLMs) to predict initial and final action steps, retrieve multiple procedural plans, and generate instructional videos by infusing procedural knowledge and minimizing supervision.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If extensive leveraging of various sources of information is used to construct procedure planning, then the model can access more information, but the complexity of implicit causal constraints and variability in multiple feasible plans remains formidable

Engineering Contradiction:

Improveinformation utilizationVSAvoidprocedure planning complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent introduces a Procedural Knowledge Graph (PKG) as an intermediary structure that mediates between the raw information sources and the procedure planning model. The PKG organizes and pre-processes information about procedures, actions, and their relationships, transforming the complex task of leveraging multiple information sources into a more manageable query process within the graph structure.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent segments the procedure planning task into multiple components: extracting initial and final actions from visual observations, querying the PKG for intermediate procedural steps, and synthesizing a complete procedure plan. This segmentation allows the system to handle complexity by breaking down the overall planning problem into smaller, manageable sub-tasks.

Inventive Principle:

Principle #1Segmentation

2Adaptability or versatility

If multiple feasible plans are considered to address variability, then the adaptability improves, but the difficulty of detecting and measuring the best plan increases

Engineering Contradiction:

Improveprocedure plan variabilityVSAvoidplan evaluation difficulty

Core Design Contradiction:

Adaptability or versatilityVSDifficulty of detecting and measuring

Solution Approach 1:

The patent incorporates feedback mechanisms where the procedure planning model queries the PKG for procedural steps based on predicted initial and final actions, and then uses this information to generate and evaluate multiple feasible plans. The system can measure plan quality by comparing generated plans against the structured knowledge in the PKG and visual observation data.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS20250157353A1Knowledge-enhanced procedure planning of instructional videos using knowledge graph and large language models

Publication Date: 2025.05.15 NEC LABORATORIES AMERICA INC

AI summary

Systems and methods include predicting a first action step and a last action step based on an initial visual observation and a goal visual state and retrieving multiple procedural plans from a procedural knowledge graph (PKG), trained using a set of training instructional videos, which start with the first action step and end with the last action step. A procedure plan is generated using the retrieved multiple procedural plans. An instructional video is generated based on the procedure plan.