Dynamic Vision Sensing for Low-Latency Facial Speech Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing facial tracking systems face challenges in noisy environments due to dark areas inside the mouth, teeth, and fast facial structure motion, requiring high-speed cameras that consume high power and generate heat, and existing solutions do not effectively address motion blur and latency issues.
Innovation Solution
A camera sensor system combining RGB and IR images with event-driven sensors (EDS) to detect motion, using EDS information for robust tracking, especially in conditions of fast motion, and fusing with RGB camera and audio information through a classifier trained with audio/camera/events data, including recurrent neural networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If frame-based imaging sensors are used to process audio signals, then the system can capture visual information, but the temporal resolution is insufficient to detect rapid changes in acoustic waveforms
Solution Approach 1:
The patent transitions from static frame-based imaging to dynamic event-based sensing, where the sensor operates asynchronously and only outputs data when changes exceed a threshold. This dynamic operation enables the system to capture rapid temporal variations in acoustic waveforms without being constrained by fixed frame rates, achieving microsecond-level temporal resolution.
Solution Approach 2:
The patent implements periodic sampling at extremely high rates (e.g., 1-10 million samples per second) for temporal-critical parameters like acoustic waveforms, while using lower-rate sampling for less time-sensitive visual parameters. This multi-rate periodic action optimizes temporal resolution where needed while maintaining overall system efficiency.
2Loss of information
If high-resolution visual data is captured continuously, then detailed spatial information is obtained, but data transmission and processing bandwidth requirements increase significantly
Solution Approach 1:
The patent extracts only the essential and changed information from the visual scene using event-based sensing. Instead of continuously capturing and transmitting all pixel data, the sensor outputs only events where changes exceed a predefined threshold, thereby preserving critical visual information while dramatically reducing data volume and transmission energy requirements.
Solution Approach 2:
The patent applies different sampling rates and quality levels to different regions or aspects of the visual scene based on their temporal-criticality. Acoustic waveforms receive high-rate sampling for precise temporal capture, while less time-sensitive visual elements are captured at lower rates, optimizing the balance between information quality and energy consumption.
3Measurement precision
If traditional imaging sensors are used for acoustic waveform detection, then the system structure is simple, but the measurement precision of acoustic signals is insufficient
Solution Approach 1:
The patent employs a multi-functional sensor system that can detect both visual and acoustic information using the same event-based sensing platform. This universal approach allows a single sensor type to perform multiple functions (spatial visualization and temporal-acoustic waveform detection) without requiring separate specialized sensors, thereby managing complexity while achieving high measurement precision for acoustic signals.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The system provides low latency, high dynamic range, and motion blur-free tracking, enhancing face, eye, and emotion recognition by leveraging EDS information in fast motion conditions and RGB details in slow motion scenarios.
Implementation Method 1
a dynamic vision sensor comprising a plurality of sensing units arranged in a sensor matrix, each sensing unit having a photodetector and an associated change detection circuit
Data Source
Figure 1
Figure 2
Figure 3
AI summary
To track certain difficult facial features during speech such as the corners of the mouth and the teeth, a camera sensor system (212/306/308/318/320) generates RGB/IR images and the system also uses light intensity change signals from an event driven sensor (EDS) (212/306/318), as well as voice analysis using signals from a microphone (328). In this way, the camera sensor system enables improved performance tracking (equivalent to using very high-speed camera) at lower bandwidth and power consumption.