Audio-Video-Haptic Stream Encoding for Immersive Playback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current devices, such as mobile cameras, only capture audio and video streams without recording haptic data, limiting the ability to provide a comprehensive user experience.
Innovation Solution
A system and method for capturing and encoding a coordinated audio/video/haptic (AVH) stream by identifying objects and events in the video stream, generating a haptic stream based on these, and encoding it with the audio and video data in a single file or associated files.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If haptic stream is added along with audio and video stream, then user experience is enhanced, but device complexity increases
Solution Approach 1:
The system segments the haptic recording process into distinct functional modules: an objects/events identifier that generates object information from AV streams, a haptic generator that creates haptic streams based on object types, and an encoder that combines AV and haptic streams into coordinated AVH streams. This modular segmentation allows each component to be optimized independently while working together to enhance user experience without overwhelming device complexity.
Solution Approach 2:
The haptic generator is designed as a multi-functional component that can generate haptic streams for various types of objects and events detected in the AV stream. By creating a universal haptic generation mechanism that handles different object types (e.g., characters, vehicles, environmental elements), the system enhances adaptability across diverse content while using a single integrated approach rather than requiring separate specialized systems for each object type.
2Adaptability or versatility
If haptic stream is generated and encoded with AV stream, then immersive playback is achieved, but processing time increases
Solution Approach 1:
The system performs preliminary actions by generating object information and identifying objects/events in the AV stream during the recording phase. The haptic generator pre-processes this information to create haptic streams that are synchronized with the AV content. By preparing haptic data in advance during recording rather than generating it in real-time during playback, the system achieves immersive playback experience while minimizing processing time delays during actual playback.
Solution Approach 2:
The haptic generator creates haptic streams as data representations (copies) of the objects and events detected in the AV stream. Instead of requiring complex real-time physical haptic simulations during playback, the system uses pre-generated haptic data copies that can be efficiently processed and rendered. This copying approach allows immersive playback experience with reduced processing requirements during playback compared to real-time generation.
3Measurement precision
If objects/events identification is performed on AV stream, then haptic accuracy is improved, but computational load increases
Solution Approach 1:
The objects/events identifier extracts only the essential object information and characteristics from the AV stream that are necessary for haptic generation. Rather than processing and analyzing every detail of the AV content, the system identifies and extracts key object types, positions, and relevant features. This extraction approach improves haptic accuracy by focusing on critical elements while reducing computational load by ignoring unnecessary details.
Data Source
AI summary
Innovative techniques to generate a haptic stream are proposed. The proposed techniques allow haptic stream to be captured and along with audio/video stream. In so doing, a full experience—audio, video, haptics experience—may be experienced during playback.