Serialized Data Stream for Post-Capture AR Editing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial reality technologies face limitations in rendering artificial reality effects when replaying pre-recorded videos, as they lack access to real-time sensor data and computational resources needed to regenerate computed data, leading to reduced quality and increased power consumption.
Innovation Solution
A method that captures and serializes video data along with contextual data streams, including raw sensor and computed data, allowing for post-capture editing and rendering of artificial reality effects without regenerating computed data, thereby reducing power consumption and enhancing effect quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If computed data is regenerated during replay, then rendering quality is maintained, but power consumption and computational resource demands increase
Solution Approach 1:
The patent applies preliminary action by computing and storing computed data (object detection, face tracking, world tracking results) during the recording phase before replay occurs. This pre-computed data is then reused during replay to maintain rendering quality without regenerating it, thereby avoiding the high computational cost and power consumption that would otherwise be required during replay operations.
2Adaptability or versatility
If multiple data streams are captured and stored, then post-capture editing capability is enhanced, but data storage requirements increase
Solution Approach 1:
The patent segments captured data into distinct data streams including video data, raw sensor data (IMU, camera, microphone), and computed data (object detection, face tracking, world tracking). Each data stream is stored separately with associated timestamps, enabling selective access and post-capture editing of specific streams without requiring storage of all data, thus managing storage requirements while enhancing editing capability.
Solution Approach 2:
The patent adds a temporal dimension by associating timestamps with each data chunk across all data streams. This temporal indexing allows efficient retrieval and editing of specific time segments without storing redundant data, enabling post-capture editing capability while optimizing storage requirements through time-based data organization.
3Speed
If computed data is stored with video data, then rendering speed during replay improves, but data serialization complexity increases
Solution Approach 1:
The patent merges multiple data streams (video data, sensor data, computed data) into a single serialized data stream by interleaving data chunks from different streams according to their timestamps. This unified serialization approach simplifies storage and retrieval operations while enabling fast rendering during replay, as all necessary data is already organized in chronological order without requiring complex deserialization logic.
Data Source
AI summary
In one embodiment, the system may receive a serialized data stream generated by serializing data chunks including data from a video stream and contextual data streams associated with the video stream. The contextual data streams may include a first computed data stream and a sensor data stream. The system may extract the video data stream and one or more contextual data streams from the serialized data stream. The system may generate a second computed data stream based on the sensor data stream in the extracted contextual data streams. The system may compare the second computed data stream to the first computed data stream extracted from the serialized data stream to select a computed data stream based on one or more pre-determined criteria. The system may render an artificial reality effect for display with the extracted video data stream based at least in part on the selected computed data stream.