Haptic Waveform Generation Using Audio Onset Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing systems that generate haptic feedback using linear resonant actuators (LRAs) face synchronization delays and lack customization, as they rely on pre-stored haptic waveforms that are not synchronized with audio signals, resulting in unsatisfactory user experiences, especially for events like explosions in games.

Innovation Solution

The method involves converting audio signals to the frequency domain, computing gradients of frequency coefficients, summing these gradients to detect onset events, and generating haptic signals based on the sum value, which are synchronized with the audio signals by modulating a sinusoidal signal at the resonant frequency of the LRA, ensuring timely and customized haptic feedback.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of time

If pre-stored haptic waveforms are used for LRA feedback, then haptic feedback can be generated, but synchronization delay occurs and customization is lost

Engineering Contradiction:

Improvesynchronization delayVSAvoidcustomization

Core Design Contradiction:

Loss of timeVSAdaptability or versatility

Solution Approach 1:

The system dynamically generates haptic waveforms in real-time based on the audio signal characteristics rather than using fixed pre-stored waveforms. The haptic waveform generator creates customized waveforms by processing the audio signal through spectral analysis and waveform synthesis, allowing both low latency and adaptation to different audio events.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system uses the audio signal itself to generate the corresponding haptic waveform, eliminating the need for external pre-stored waveforms. The audio signal undergoes spectral analysis and waveform generation processes that are self-contained within the system, enabling real-time customization without relying on pre-defined templates.

Inventive Principle:

Principle #25Self-service

2Device complexity

If pre-stored haptic waveforms are used, then haptic feedback generation is simplified, but the feedback is not synchronized with audio signals

Engineering Contradiction:

Improvehaptic waveform storageVSAvoidsynchronization timing

Core Design Contradiction:

Device complexityVSLoss of time

Solution Approach 1:

The system replaces the mechanical approach of storing and retrieving pre-defined waveforms with a signal processing approach. Audio signals undergo spectral analysis (FFT), envelope detection, and waveform synthesis to generate haptic waveforms programmatically, substituting physical storage with computational generation.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system performs preliminary processing of the audio signal through spectral analysis and envelope detection before waveform generation. This preliminary action extracts relevant features from the audio signal that guide the synthesis of synchronized haptic waveforms, ensuring timing alignment without retrieval delays.

Inventive Principle:

Principle #10Preliminary action

3Difficulty of detecting and measuring

If narrowband filters are used to detect audio events, then event detection is achieved, but large propagation delay occurs

Engineering Contradiction:

Improveaudio event detectionVSAvoidpropagation delay

Core Design Contradiction:

Difficulty of detecting and measuringVSLoss of time

Solution Approach 1:

The system segments the audio signal into frequency components using Fast Fourier Transform (FFT) rather than using narrowband filters. This segmentation approach analyzes multiple frequency bands simultaneously through spectral analysis, detecting audio events across the entire frequency spectrum without the sequential processing delays inherent in narrowband filtering.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system transitions from the time-domain analysis used in narrowband filtering to frequency-domain analysis using FFT. By transforming the detection problem into the frequency domain, the system can identify audio events through spectral characteristics and envelope changes, achieving faster detection without the propagation delays of traditional time-domain filtering.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS11361633B2Audio to haptics waveform generation using onset detection

Publication Date: 2022.06.14 TEXAS INSTRUMENTS INC

AI summary

A method includes converting an audio signal of a current time period to a frequency domain to produce a set of frequency coefficients. For each of the frequency coefficients, the method includes computing a gradient of that frequency coefficient's magnitude relative to a magnitude of the same frequency coefficient from a previous time period. The method then includes summing the gradients computed for the set of frequency coefficients to produce a sum value, and then generating a haptic signal based on the sum value.