Mixed-Reality Recording Tracks for Low-Latency Editing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing technologies face challenges in efficiently editing and rendering mixed-reality recordings, which combine natural and synthetic elements, due to resource-intensive processing and complex composition of these elements in media files.

Innovation Solution

A media data format is introduced that includes separate tracks for different content types, using snapshots and deltas to represent scene models, allowing for efficient storage, streaming, and rendering by separating pre-rendering and real-time rendering operations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If mixed-reality recordings combine natural and synthetic elements in a single media file, then the recording captures the complete user experience, but resource consumption and processing complexity increase significantly

Engineering Contradiction:

Improvecompleteness of recordingVSAvoidresource consumption

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The patent divides the mixed-reality recording into separate tracks: natural element tracks and synthetic element tracks. Each track independently stores media samples for specific content types. This segmentation allows selective processing and rendering of only required elements, reducing overall resource consumption while maintaining complete scene representation.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent extracts synthetic elements from the natural scene recording and stores them in separate tracks. This extraction enables independent processing of synthetic content, allowing the system to load and render only necessary elements during playback, thereby reducing memory usage and processing power requirements.

Inventive Principle:

Principle #2Taking out (Extraction)

2Reliability

If all elements of mixed-reality scene are processed together, then complete scene is rendered, but processing latency increases

Engineering Contradiction:

Improvescene rendering completenessVSAvoidprocessing latency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent segments the rendering pipeline into separate processing streams for natural and synthetic elements. Each track can be decoded and rendered independently, allowing parallel processing that reduces overall latency while ensuring complete scene composition.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent performs pre-processing of media samples during encoding, organizing them into separate tracks with synchronized timestamps. This preliminary organization enables faster runtime processing by eliminating the need for complex real-time composition operations, reducing rendering latency.

Inventive Principle:

Principle #10Preliminary action

3Ease of operation

If natural and synthetic elements are composited together in real-time, then seamless integration is achieved, but computational complexity increases

Engineering Contradiction:

Improveintegration smoothnessVSAvoidcomposition complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent maintains separate tracks for natural and synthetic elements throughout the processing pipeline. This segmentation simplifies the compositing operation to a straightforward overlay process based on synchronized timestamps, reducing computational complexity compared to full scene graph processing.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces timestamp synchronization as an intermediary mechanism that coordinates between natural and synthetic element tracks. This mediator enables seamless integration by automatically aligning elements from different tracks without requiring complex real-time composition algorithms.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Loss of information

If complete scene hierarchy is stored in recording, then element relationships are preserved, but data size and processing overhead increase

Engineering Contradiction:

Improvescene structure preservationVSAvoiddata size

Core Design Contradiction:

Loss of informationVSQuantity of substance

Solution Approach 1:

The patent extracts essential scene hierarchy information (element relationships and composition order) and stores it as metadata in the media file header, rather than embedding complete scene graphs with every media sample. This extraction reduces data redundancy while preserving necessary structural information for correct scene assembly.

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS12505614B2Editing mixed-reality recordings

Publication Date: 2025.12.23 APPLE INC

AI summary

Improved techniques for editing and rendering a mixed-reality recording are provided, including receiving such a mixed-reality recording including parameters of the natural and/or synthetic elements, and then responsive to a user selection of a selected element of a plurality of elements in the recording, building an animation track based on the selected element including parameter(s) of the selected element. Responsive to a user modification input, modifying the parameter(s) of the selected element based on the user modification input, and storing a modified recording including the modified parameter(s).