Video-Based Heart Rate Extraction for Mood Analysis

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current methods for determining mental states during human-computer interaction are unreliable and impractical, as they rely on subjective surveys or intrusive physiological monitoring, which often result in inaccurate and incomplete data.

Innovation Solution

A computer-implemented method that analyzes video footage to extract heart rate information, using statistical classifiers trained on blood volume pulse data, to infer mental states by correlating heart rate variability with stimuli, enabling non-invasive and objective mood measurement.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If subjective surveys are used to determine mental states, then ease of operation is improved, but measurement precision deteriorates

Engineering Contradiction:

Improveease of operationVSAvoidmeasurement precision

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The patent replaces subjective survey methods with an automated computer vision system that uses machine learning algorithms to objectively analyze facial expressions, gestures, and physiological indicators. This substitution transforms manual self-reporting into an automated optical measurement system, thereby improving measurement precision while maintaining ease of operation.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent introduces an intermediary computational layer that processes raw video data through trained machine learning models to extract mental state indicators. This intermediary system acts as a mediator between the subject's natural behavior and the measurement outcome, enabling precise objective assessment without requiring direct subject participation or subjective input.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If intrusive physiological monitoring is used to determine mental states, then measurement precision is improved, but device complexity and ease of operation worsen

Engineering Contradiction:

Improvemeasurement precisionVSAvoiddevice complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent uses a standard webcam to capture video images that serve as a copy or proxy for physiological data. Instead of requiring specialized physiological sensors, the system extracts heart rate, respiration rate, and other physiological indicators from visual information in ordinary video feeds, thereby maintaining measurement precision while eliminating device complexity.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system enables the subject to serve themselves by naturally performing daily activities while being passively monitored. The subject不需要 to wear special equipment, attach sensors, or actively participate in measurement procedures - their natural behavior during routine tasks is captured by the webcam and analyzed automatically, thus improving ease of operation without sacrificing measurement precision.

Inventive Principle:

Principle #25Self-service

3Measurement precision

If intrusive physiological monitoring is used to determine mental states, then measurement precision is improved, but ease of operation deteriorates

Engineering Contradiction:

Improvemeasurement precisionVSAvoidease of operation

Core Design Contradiction:

Measurement precisionVSEase of operation

Solution Approach 1:

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces intrusive physiological monitoring equipment with a non-intrusive computer vision system. By substituting specialized sensors with a standard webcam and using algorithmic analysis of visual data, the system maintains measurement precision while eliminating the operational burden and discomfort associated with wearing and managing physiological monitoring devices.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Applied Scientific Principles

This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.

Function Achieved in This Case

This method provides accurate and objective mental state analysis, allowing for improved media presentation optimization and user experience evaluation, while being non-intrusive and easily integrable with existing computer systems.

Implementation Method 1

analyzing the video to determine heart rate information... separating pixels from the video of the individual, into at least a green pixel temporal intensity trace... recognizing a pulse, from the video of the individual, using the statistical classifier

Methodology Applied

Scientific EffectPhotoplethysmography: Absorption (EM radiation)

Data Source

PatentUS10517521B2Mental state mood analysis using heart rate collection based on video imagery

Publication Date: 2019.12.31 AFFECTIVA

AI summary

Video of one or more people is obtained and analyzed. Heart rate information is determined from the video. The heart rate information is used in mental state analysis. The heart rate information and resulting mental state analysis are correlated to stimuli, such as digital media, which is consumed or with which a person interacts. The heart rate information is used to infer mental states. The inferred mental states are used to output a mood measurement. The mental state analysis, based on the heart rate information, is used to optimize digital media or modify a digital game. Training is employed in the analysis. Machine learning is engaged to facilitate the training.