Keyboard Performance Video Compositing Without Complex Sensors

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing technologies face challenges in generating videos that realistically depict a desired performer playing a desired keyboard instrument, as they require complex sensor detection and processing.

Innovation Solution

A method and system for extracting a reference portion from a performance video of a first keyboard instrument, including the performer's hands, and superimposing it on a second keyboard instrument to generate a composite video, utilizing techniques like object detection and depth estimation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If various sensors are used to detect player performance, then video generation capability is improved, but device complexity and difficulty of detection increase

Engineering Contradiction:

Improvevideo generation capabilityVSAvoidsensor detection complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent uses image processing to extract hand portions from performance videos and superimposes them onto different keyboard instruments. This copying approach allows the same performer video to be adapted to multiple instrument types without requiring complex sensor detection, thereby improving versatility while reducing device complexity

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent introduces an intermediary processing system that extracts hand portions and keyboard portions separately, then combines them to create composite videos. This intermediary approach simplifies the overall detection process by breaking down the complex task of detecting player performance into simpler, manageable steps

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If complex sensor detection is used, then performance detection accuracy is improved, but ease of operation deteriorates

Engineering Contradiction:

Improveperformance detection accuracyVSAvoidease of video generation

Core Design Contradiction:

Measurement precisionVSEase of operation

Solution Approach 1:

The patent extracts only the necessary hand portions from the performance video using image processing techniques. This extraction approach maintains detection accuracy for the critical elements (hands playing keys) while simplifying the overall operation by focusing only on relevant visual information rather than requiring complex sensor arrays

Inventive Principle:

Principle #2Taking out (Extraction)

3Measurement precision

If multiple sensors are deployed, then measurement precision is improved, but loss of time increases

Engineering Contradiction:

Improveperformance detection accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent replaces complex mechanical sensor systems with digital image processing techniques. By using computer vision algorithms to extract hand portions from videos, the system achieves measurement precision without the time-consuming data collection and processing required by multiple physical sensors

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20250363761A1Video processing method, video processing system, and non-transitory computer-readable storage medium storing video processing program

Publication Date: 2025.11.27 YAMAHA CORP

AI summary

A video processing method includes extracting, from a performance video representing a performance of a first keyboard instrument by a performer, a first reference portion that includes a hand of the performer. The video processing method further includes superimposing the first reference portion on a keyboard portion of a second keyboard instrument, thereby generating a composite video.