Video Workflow Step Extraction for Query-Based Task Guidance

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing online instructional videos are not optimized for individual user circumstances and needs, making it difficult for users to follow tasks due to noise, pace mismatch, and inefficient playback controls.

Innovation Solution

A system that parses video data to identify discrete workflow steps using speech-to-text and action recognition, generates text and images for each step, and uses a large language model to present tailored content based on user queries, allowing for customizable and interactive step-by-step guidance.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If online instructional videos are used to teach complex tasks, then users can access instructional content, but the videos are not optimized for individual user circumstances and needs

Engineering Contradiction:

Improveadaptability to individual user circumstancesVSAvoiddifficulty to watch in noisy environment

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The video is segmented into discrete workflow steps that can be independently accessed and presented. Each step is extracted and presented separately with its own text description and image, allowing users to navigate through specific steps without watching the entire video, thus adapting to individual learning needs and environmental constraints.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Text descriptions and images are introduced as intermediary elements between the video content and the user. These intermediaries convey the essential information from each video step without requiring audio playback, enabling users to follow instructions in noisy environments or at their own pace.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Loss of time

If users watch instructional videos at the video's playback pace, then they can follow the content, but it is difficult to perform the task at the same pace leading to frequent pausing and resuming

Engineering Contradiction:

Improvetime lost to pausing and resumingVSAvoiddifficulty to perform task at video pace

Core Design Contradiction:

Loss of timeVSEase of operation

Solution Approach 1:

By dividing the video into discrete, independently accessible steps, users can navigate directly to the step they need without pausing and rewinding. This eliminates time loss associated with frequent pausing and resuming, as users can progress through steps at their own operational pace.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

All video steps are pre-processed into text descriptions and images before the user interacts with them. This preliminary conversion allows users to access step information without interrupting their task flow to pause or rewind the video, maintaining continuous workflow.

Inventive Principle:

Principle #10Preliminary action

3Measurement precision

If users rewind video to review prior portions, then they can re-watch content, but imprecise rewinding leads to unnecessary re-watching

Engineering Contradiction:

Improveprecision of navigation to prior contentVSAvoidtime spent on unnecessary re-watching

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The video content is segmented into discrete, clearly defined steps with unique identifiers. Users can navigate precisely to any specific step without the ambiguity of video timecodes, eliminating unnecessary re-watching and ensuring accurate positioning at the desired content location.

Inventive Principle:

Principle #1Segmentation

4Adaptability or versatility

If instructional videos are converted to text and images, then content can be accessed without audio, but the system complexity increases

Engineering Contradiction:

Improveability to access content without audioVSAvoidsystem complexity for parsing and presenting content

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The video parsing, transcription, and image extraction processes are performed in advance before user interaction. This preliminary processing converts the video into text and image formats that can be easily accessed and presented without requiring complex real-time processing during user interaction, reducing perceived system complexity.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS12488046B1Workflow content generation via video analysis

Publication Date: 2025.12.02 LENOVO UNITED STATES INC

AI summary

In one aspect, a device includes a processor system and storage accessible to the processor system. The storage includes instructions executable by the processor system to, responsive to a user query, parse data related to a source video to identify discrete steps that conform to the user query. The discrete steps are steps in a workflow indicated in the source video. Based on identifying the discrete steps, the instructions are then executable to present, on a display, text and images that indicate the discrete steps. The text and images are different from the source video itself but are derived from the source video. In one particular example, the instructions may even be executable to use a large language model (LLM) to execute retrieval-augmented generation (RAG) to present, on the display, the text and images in conformance with the user query.