Facial Expression Sound Control via Near-Infrared Imaging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current sound effect control interfaces for musical instruments are inadequate as they often require musicians to use hands or feet, limiting their ability to control multiple effects simultaneously and move freely on stage, with existing solutions only allowing control of one parameter at a time and being cumbersome or aesthetically unappealing.
Innovation Solution
A process using digital image processing to recognize facial expressions and associate them with sound effects, allowing for simultaneous control of multiple sound effects through a camera capturing near-infrared images, enabling the use of facial expressions to control sound effects like fading and echo, with a database mapping expressions to sound effects and allowing for 'sound morphing' between them.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If manual control interfaces (pedals, potentiometers) are used to control sound effects, then sound effects can be controlled, but only one parameter can be controlled at a time and the musician's hands and feet are occupied
Solution Approach 1:
The patent transitions from controlling sound effects using hands and feet (traditional spatial dimensions) to using facial expressions (adding a new dimensional space for control). This allows musicians to control multiple sound effect parameters simultaneously through different facial muscles without occupying their hands or feet, resolving the contradiction between ease of operation and adaptability.
Solution Approach 2:
The system makes the face a multi-functional control interface by mapping different facial expressions (mouth opening, eyebrow movement, eye blinking) to different sound effect parameters. This universalizes the face as a control device that can simultaneously manage multiple sound effects, overcoming the limitation of single-parameter control in traditional interfaces.
2Ease of operation
If foot-controlled interfaces (pedals) are used to control sound effects, then sound effects can be controlled without using hands, but the musician cannot move around freely on stage and precision is difficult to achieve
Solution Approach 1:
The patent replaces foot-based control (limited movement) with facial expression-based control (full body freedom). By using the face as the control interface, musicians can move freely on stage while maintaining precise control over sound effects through facial muscle movements, eliminating the harmful constraint on movement freedom.
3Ease of operation
If the talk box device is used to control sound effects with the mouth, then sound effects can be controlled without using hands, but the device is aesthetically unappealing and only enables control of one type of effect
Solution Approach 1:
The system transforms the mouth and face from a single-function control device (as in the talk box) into a multi-functional interface. By detecting various facial expressions including mouth opening, eyebrow movement, and eye blinking, the system enables control over multiple types of sound effects simultaneously, while eliminating the need for the aesthetically unappealing tube device.
Solution Approach 2:
The patent extracts the control function from the physical talk box tube and relocates it to the natural facial expressions themselves. This removes the harmful aesthetic element while preserving and enhancing the control capability, allowing musicians to control sound effects using their natural facial movements without any external apparatus in their mouth.
4Adaptability or versatility
If multiple sound effects are controlled simultaneously using traditional interfaces, then more parameters can be adjusted, but the interface complexity increases and requires multiple devices
Solution Approach 1:
The patent merges multiple control functions into a single unified interface: the face. By combining detection of various facial expressions (mouth, eyebrows, eyes) into one integrated system, the patent enables simultaneous control of multiple sound effects without requiring multiple separate devices, thereby reducing overall system complexity while maintaining high adaptability.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables musicians to control a wide range of sound effects simultaneously without using their hands or feet, allowing for precise and flexible control of sound effects during performances, even in varying lighting conditions, while maintaining freedom of movement.
Implementation Method 1
The camera is sensitive to near-infrared wavelengths. The camera comprises an emission medium configured to emit a source of light, preferably of infrared wavelength ranging between approximately 700 to 3000 nanometers
Implementation Method 2
The capture module comprises a filtering medium, such as an optical filter, which preferably only allows the passage of a frequency band set around to the frequency of the emitted light
Data Source
AI summary
The object of the present invention concerns a control device (100) for a generation module (GM) of sound effects (EFA, EFB) of a musical instrument (MI), such device comprising computer software configured for: —the capture, using a digital camera (10), of at least one digital image (I) comprising at least one portion of the user's (U) face; —processing of such at least one image (I) to define expression data (D_EXi, i being a positive integer) containing information relating to facial expressions (EXa, EXb) of the user (U); —an analysis of such expression data (D_EXi) using a predefined first database (DB1) to determine a sound effect data (D_EFj, j being a positive integer) containing information relating to at least one sound effect (EFA, EFB) corresponding to the facial expression (EXa, EXb) of the user (U).