Scalable creation and sharing of multi-sensory assets and experiences

Abstract

Some examples involve creating a sensory object-based asset that includes one or more sensory objects, corresponding asset metadata, and a sensory object timeline. The sensory object timeline may include information corresponding to a start time and duration for each sensory object of the one or more sensory objects. The asset metadata may include a control interface mapped to sensory object properties of the one or more sensory objects. The sensory object-based asset may be, or may include, a nested asset. The sensory object timeline may include one or more sensory object-based assets. Some disclosed methods involve unwrapping a nested sensory object-based asset and arranging the sensory objects of the nested sensory object-based asset into a single timeline.

Description

Dolby Ref. D24098W001SCALABLE CREATION AND SHARING OF MULTI-SENSORY ASSETS AND EXPERIENCES CROSS-REFERENCE TO RELATED APPLICATIONSThe present application claims the benefit of priority from United States Provisional Patent Application No. 63 / 720,639 (our reference: D24098USP1), filed on November 14, 2024, United States Provisional Patent Application No. 63 / 720,653 (our reference: D24101USP1), filed on November 14, 2024, United States Provisional Patent Application No. 63 / 720,664 (our reference: D24104USP1), filed on November 14, 2024, United States Provisional Patent Application No. 63 / 720,667 (our reference: D24105USP1), filed on November 14, 2024, and United States Provisional Patent Application No. 63 / 879,383 (our reference: D24098USP2), filed on September 10, 2025, each of which is incorporated herein by reference in its entirety.TECHNICAL FIELD

[0001] The present disclosure relates to creating multi-sensory (MS) experiences, which may also be referred to herein as multi-modal (MM) experiences, some of which include lightbased sensory experiences.BACKGROUND

[0002] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted as prior art by inclusion in this section.

[0003] Media content creation has generally focused on audio and video experiences. There has been limited creation of multi-sensory content due to the bespoke nature of actuation. For example, luminaires, which are also referred to herein as light fixtures or simply as lights, are used extensively as an expression of art and function for concerts. However, each installation is designed specifically for a unique set of luminaires. Creating a lighting design beyond the set of light fixtures the system was designed for is generally not feasible. Some other systems that attempt to create light experiences, also referred to herein as “lightscapes,” simply do so by extending the screen visuals algorithmically, but are not specifically authored. Improved methods, devices and systems for creating and sharing multi-sensory assets and experiences would be desirable.SUMMARYDolby Ref. D24098W001

[0004] At least some aspects of the present disclosure may be implemented via methods, such as audio processing methods. In some instances, the methods may be implemented, at least in part, by a control system such as those disclosed herein. Some methods involve interchanging and packaging sensory object -based assets. Some such methods involve receiving, by a control system, user input regarding the creation of a sensory object-based asset comprising one or more sensory objects, corresponding asset metadata, and an asset timeline. The asset timeline may include information corresponding to a start time and duration for each sensory object of the one or more sensory objects. Some such methods involve creating, by the control system, the sensory object-based asset according to the user input.

[0005] According to some examples, the user input may be received via a graphical user interface of a sensory asset creation tool implemented by the control system. In some examples, the asset metadata may include a control interface mapped to sensory object properties of the one or more sensory objects. According to some examples, the sensory object-based asset may include one or more nested sensory object-based assets. In some such examples, the asset timeline may correspond to the one or more sensory object-based assets.

[0006] Some disclosed methods involve unwrapping a nested sensory object-based asset into a single timeline. Some such methods may involve receiving, by a control system, a first sensory object-based asset comprising a second sensory object-based asset. The second sensory object-based asset may be nested within the first sensory object-based asset. In some examples, each of the first sensory object -based asset and the second sensory object-based asset may include one or more sensory objects, corresponding asset metadata, and a sensory object timeline.

[0007] Some methods may involve combining, by the control system, a first sensory object timeline of the first sensory object-based asset and a second sensory object timeline of the second sensory object-based asset to create a combined sensory object timeline that includes the one or more sensory objects of both the first and second sensory object-based assets. Some methods may involve recalculating, by the control system, timing metadata for each of the one or more sensory objects of the first sensory object-based asset or the second sensory object -based asset in reference to the combined sensory object timeline.

[0008] Some or all of the operations, functions and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more computer-readable non-transitory media. Such non-transitory media may include one or more memory devices such as those described herein, including but not limited to one orDolby Ref. D24098WG01more random access memory (RAM) devices, read-only memory (ROM) devices, etc.Accordingly, some innovative aspects of the subject matter described in this disclosure can be implemented in one or more computer-readable non-transitory media having software stored thereon.

[0009] At least some aspects of the present disclosure may be implemented via apparatus. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some implementations, an apparatus may include an interface system and a control system. The control system may include one or more general purpose single- or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof. The control system may be configured to perform some or all of the disclosed methods.

[0010] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Disclosed embodiments now be described, by way of example only, with reference to the accompanying drawings.

[0012] Figure 1 is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure.

[0013] Figure 2 shows example elements of an endpoint.

[0014] Figure 3 shows examples of actuator elements.

[0015] Figure 4 shows example elements of a system for the creation and playback of multi-sensory (MS) experiences.

[0016] Figure 5 shows example elements of another system for the creation and playback of MS experiences.

[0017] Figure 6 shows an example of a graphical user interface (GUI) that may be presented by a display device of the lightscape creation tool of Figure 5.

[0018] Figure 7 shows another example of a GUI that may be presented by a display device of the lightscape creation tool of Figure 5.Dolby Ref. D24098W001

[0019] Figure 8 shows examples of spatial primitive visualizations within a reference environment.

[0020] Figure 9 shows an example of two-dimensional (2D) visualisation relating to volumetric fields.

[0021] Figure 10 shows an example of three-dimensional (3D) visualisation relating to volumetric fields.

[0022] Figures 11 A and 1 IB provide examples of importing and using a 2D volumetric texture asset.

[0023] Figure 12 shows an example of an image that can be used during a process of sensory content creation.

[0024] Figures 13A and 13B show an example of a GUI for sensory object parameter modification.

[0025] Figures 14A and 14B show examples of two different methods for achieving substantially the same sensory object movement trajectory.

[0026] Figures 15A and 15B show example states of a state machine that may be implemented according to some implementations.

[0027] Figure 16 shows an example of a creation tool GUI that includes a visual scripting interface.

[0028] Figures 17A and 17B show examples of scheduling assets and asset modifiers on a timeline.

[0029] Figure 18 is a block diagram that illustrates an example of applying the effect intensity modifier shown in Figure 17A.

[0030] Figure 19 shows an example of a GUI that for comparing two versions of the same asset.

[0031] Figure 20 shows an example of a GUI that may be used for bed-based sensory object authoring.

[0032] Figure 21 shows an example of a GUI that may be used for ambient sensory object authoring.

[0033] Figure 22 shows an example of a GUI for associating one or more sensory objects with a game actor.

[0034] Figure 23 shows examples of linking sensory object properties and game state information.

[0035] Figure 24 shows an example of an auditioning endpoint with warnings for a content creator.Dolby Ref. D24098W001

[0036] Figure 25A shows another example of a reference environment with reference to which content can be created.

[0037] Figure 25B shows an example of an actuator domain for content creation in the reference environment of Figure 25A.

[0038] Figure 26A shows another example of a reference environment with reference to which content can be created.

[0039] Figures 26B and 26C show examples of actuator domains for content creation in the reference environment of Figure 26A.

[0040] Figure 27 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein.

[0041] Figure 28 shows an example of a GUI that includes personalization controls.

[0042] Figures 29A, 29B and 29C show three examples of zone-based personalization.

[0043] Figure 30 is a block diagram that illustrates assets being imported from, and exported to, an asset library.

[0044] Figures 31 and 32 are block diagrams that illustrate different types of asset structures.

[0045] Figure 33 is a block diagram that shows examples of mapping a control to sensory object and asset properties.

[0046] Figure 34 shows an example of sensory objects within an asset.

[0047] Figure 35 shows an example of nested assets.

[0048] Figure 36 shows asset Z of Figure 35 in a mezzanine format.

[0049] Figure 37 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein.

[0050] Figure 38 shows an example of a GUI that may be presented as part of a semantic analysis tooling process.

[0051] Figure 39 shows example blocks that may be involved in a process of augmenting previously-created assets.

[0052] Figure 40 shows example blocks that may be involved in a process of using previously-created assets to automatically generate sensory assets.

[0053] Figure 41 shows an example of auto-completion at the event level based on existing video content.

[0054] Figure 42 shows an example of auto-completion at the event level based on existing audio content.DETAILED DESCRIPTIONDolby Ref. D24098W001

[0055] Described herein are techniques related to providing multi-sensory media content. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be evident, however, to one skilled in the art that the present disclosure as defined by the claims may include some or all of the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0056] In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is mainly for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in another order), and may occur in parallel with other steps. A second step is required to follow a first step only when the first step must be completed before the second step is begun. Such a situation will be specifically pointed out when not clear from the context.

[0057] In this document, the terms “and”, “or” and “and / or” are used. Such terms are to be read as having an inclusive meaning. For example, “A and B” may mean at least the following: “both A and B”, “at least both A and B”. As another example, “A or B” may mean at least the following: “at least A”, “at least B”, “both A and B”, “at least both A and B”. As another example, “A and / or B” may mean at least the following: “A and B”, “A or B”. When an exclusive-or is intended, such will be specifically noted (e.g., “either A or B”, “at most one of A and B”).

[0058] This document describes various processing functions that are associated with structures such as blocks, elements, components, circuits, etc. In general, these structures may be implemented by one or more processors controlled by one or more computer programs.

[0059] As noted above, media content delivery has generally been focused on audio and video experiences. There has been limited delivery of multi-sensory (MS) content due to the customized nature of actuation.

[0060] This application describes methods for extending the creative palette for content creators, allowing spatial, MS experiences to be created and delivered at scale. Some such methods involve the introduction of new layers of abstraction, in order to allow authored MSDolby Ref. D24098W001experiences to be delivered to different endpoints, with different types of fixtures or actuators. As used herein, the term “endpoint” is synonymous with “playback environment” or simply “environment,” meaning an environment that includes one or more actuators that may be used to provide an MS experience. Such endpoints may include a room, such as the living room of a home, a car, a cinema, a night club or other venue, etc. Some disclosed methods involve the creation, delivery and / or rendering of object-based sensory data, which may include sensory objects and corresponding sensory metadata. This abstraction allows creative intent to be implemented in an object-based format that does not require prior knowledge of the specific controller actuation, thereby enabling greater flexibility and scalability of fixtures and actuators across endpoints. An MS experience provided via objectbased sensory data may be referred to herein as a “flexibly-scaled MS experience.”

[0061] This disclosure outlines systems and methods for scalable creation and sharing of multi-sensory assets and experiences. Some aspects of this disclosure provide tooling required to create new content types such as generalized discrete spatial objects, bed-based content & volumetric field and texture parameterization and generation. In some disclosed examples, the tooling also supports the creation, sharing and management of assets that may be imported (nested) into other assets or experiences and the parameterization of interactive APIs in the content.

[0062] This disclosure also describes schemas that facilitate storage and interchange of the assets that comprise multi-stream multi-sensory and potentially interactive experiences. Also described are methods to interpret these schemas so that they can be represented in a human readable and subsequently editable form.

[0063] This disclosure also describes methods and systems that enable a user to personalise and control a flexibly rendered multi-sensory experience in multiple zones across an endpoint.

[0064] This disclosure also describes methods and systems used to assist in the creation of multisensory experience content. Assistive systems may include trained Al networks, classical algorithms, heuristics or combinations thereof. Some disclosed systems can assist the creator by automatically generating and tagging salient event markers, by analyzing content in one or more modalities to produce initial draft content in a different modality, by extracting semantic information to aid and inform the creator, or combinations thereof.Dolby Ref. D24098W001

[0065] Figure 1 is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure. As with other figures provided herein, the types and numbers of elements shown in Figure 1 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, the apparatus 101 may be, or may include, a device that is configured for performing at least some of the methods disclosed herein, such as a smart audio device, a laptop computer, a cellular telephone, a tablet device, a smart home hub, etc. In some such implementations the apparatus 101 may be, or may include, a server that is configured for performing at least some of the methods disclosed herein.

[0066] In this example, the apparatus 101 includes at least an interface system 105 and a control system 110. In some implementations, the control system 110 may be configured for performing, at least in part, the methods disclosed herein. The control system 110 may, in some implementations, be configured for receiving, via the interface system 105, receiving, by a control system, a content bitstream including encoded object-based sensory metadata, The encoded object-based sensory metadata may correspond to sensory effects such as lighting, haptics, airflow, one or more positional actuators, or combinations thereof, to be provided by a plurality of sensory actuators in an environment. In some implementations, the control system 110 may be configured for extracting object-based sensory metadata from the content bitstream and for providing the object-based sensory metadata to a sensory Tenderer.

[0067] According to some examples, the object-based sensory metadata may include sensory spatial metadata indicating at least a spatial position for rendering the object-based sensory metadata within the environment, an area for rendering the object-based sensory metadata within the environment, or combinations thereof. In some implementations, the object-based sensory metadata does not correspond to any particular sensory actuator in the environment. In some examples, the object-based sensory metadata may include abstracted sensory reproduction information allowing the sensory rcndcrcr to reproduce authored sensory effects, which also may be referred to herein as intended sensory effects, via various sensory actuator types, via various numbers of sensory actuators and from various sensory actuator positions in the environment.

[0068] In some examples, the content bitstream also may include encoded audio objects synchronized with the encoded object-based sensory metadata. The audio objects mayDolby Ref. D24098W001include audio signals and corresponding audio object metadata. In some such implementations, the control system 110 may be configured for extracting audio objects from the content bitstream and for providing the audio objects to an audio Tenderer. According to some examples, the audio objects may include audio signals and corresponding audio object metadata. The audio object metadata may include at least audio object spatial metadata indicating an audio object spatial position for rendering the audio signals within the environment.

[0069] The interface system 105 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system and / or a gesture sensor system. In some examples, the interface system 105 may include one or more interfaces between the control system 110 and a memory system, such as the optional memory system 115 shown in Figure 1. However, the control system 110 may include a memory system in some instances.

[0070] The control system 110 may, for example, include a general purpose single- or multichip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0071] In some implementations, the control system 110 may reside in more than one device. For example, a portion of the control system 110 may reside in a device within an environment (such as a laptop computer, a tablet computer, a smart audio device, etc.) and another portion of the control system 110 may reside in a device that is outside the environment, such as a server. In other examples, a portion of the control system 110 may reside in a device within an environment and another portion of the control system 110 may reside in one or more other devices of the environment.

[0072] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may, for example, reside in theDolby Ref. D24098W001optional memory system 115 shown in Figure 1 and / or in the control system 110.Accordingly, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. The software may, for example, include instructions for controlling at least one device to process audio data. The software may, for example, be executable by one or more components of a control system such as the control system 110 of Figure 1.

[0073] In some examples, the apparatus 101 may include the optional microphone system 120 shown in Figure 1. The optional microphone system 120 may include one or more microphones. In some implementations, one or more of the microphones may be part of, or associated with, another device, such as a speaker of the speaker system, a smart audio device, etc.

[0074] According to some implementations, the apparatus 101 may include the optional actuator system 125 shown in Figure 1. The optional actuator system 125 may include one or more loudspeakers, one or more haptic devices, one or more light fixtures, also referred to herein as luminaires, one or more fans or other air-moving devices, one or more display devices, including but not limited to one or more televisions, one or more positional actuators, one or more other types of devices for providing an MS experience, or combinations thereof. The term “light fixture’’ as used herein refers generally to various types of light sources, including individual light sources, groups of light sources, light strips, etc. A “light fixture’’ may be moveable, and therefore the word “fixture” in this context does not mean that a light fixture is necessarily in a fixed position in space. The term “positional actuators” as used herein refers generally to devices that are configured to change a position or orientation of a person or object, such as motion simulator seats. Loudspeakers may sometimes be referred to herein as “speakers.” In some implementations, the optional actuator system 125 may include a display system including one or more displays, such as one or more light-emitting diode (LED) displays, one or more organic light-emitting diode (OLED) displays, etc. In some examples wherein the apparatus 101 includes a display system, the optional sensor system 130 may include a touch sensor system and / or a gesture sensor system proximate one or more displays of the display system. According to some such implementations, the control system 110 may be configured for controlling the display system to present a graphical user interface (GUI), such as a GUI related to implementing one of the methods disclosed herein.Dolby Ref. D24098W001

[0075] In some implementations, the apparatus 101 may include the optional sensor system 130 shown in Figure 1. The optional sensor system 130 may include a touch sensor system, a gesture sensor system, one or more cameras, etc.

[0076] This application describes methods for creating and delivering a flexibly scaled multi-sensory (MS) immersive experience (MSIE) to different playback environments, which also may be referred to herein as endpoints. Such endpoints may include a room, such as the living room of a home, a car, a cinema, a night club or other venue, an AR / VR headset, a PC, a mobile device, etc.

[0077] Figure 2 shows example elements of an endpoint. In this example, the endpoint is a living room 1001 containing multiple actuators 008, some furniture 1010 and a person 1000 — also referred to herein as a user — who will consume a flexibly-scaled MS experience. Actuators 008 are devices capable of altering the environment 1001 that the user 1000 is in. Actuators 008 may include one or more televisions or other display devices, one or more luminaires — also referred to herein as light fixtures — one or more loudspeakers, etc.

[0078] The number of actuators 008, the arrangement of actuators 008 and the capabilities of actuators 008 in the space 1001 may vary significantly between different endpoint types. For example, the number, arrangement and capabilities of actuators 008 in a car will generally be different from the number, arrangement and capabilities of actuators 008 in a living room, a night club, etc. In many implementations, the number, arrangement and / or capabilities of actuators 008 may vary significantly between different instances of the same type, e.g., between a small living room with 2 actuators 008 and a large living room with 16 actuators 008. The present disclosure describes various method for creating and delivering flexibly-scaled MSIEs to these non-homogenous endpoints.

[0079] Figure 3 shows examples of actuator elements. In this example, the actuator is a luminaire 1100, which includes a network module 1101, a control module 1102 and a light emitter 1103. According to this example, the light emitter 1103 includes one or more lightemitting devices, such as light-emitting diodes, which are configured to emit light into an environment in which the luminaire 1100 resides. In this example, the network module 1101 is configured to provide network connectivity to one or more other devices in the space, such as a device that sends commands to control the emission of light by the luminaire 1100. According to this example, the control module 1102 is configured to receive signals via the network module 1101 and to control the light emitter 1103 accordingly.Dolby Ref. D24098W001

[0080] Other examples of actuators also may include a network module 1101 and a control module 1102, but may include other types of actuating elements. Some such actuators may include one or more loudspeakers, one or more haptic devices, one or more fans or other airmoving devices, one or more positional actuators, one or more display devices, etc.

[0081] Figure 4 shows example elements of a system for the creation and playback of multi-sensory (MS) experiences. As with other figures provided herein, the types and numbers of elements shown in Figure 4 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, system 300 may be, or may include, one or more devices configured for performing at least some of the methods disclosed herein. In some examples, system 300 may include one or more instances of the control system 110 of Figure 1 that are configured for performing at least some of the methods disclosed herein.

[0082] According to the examples in the present disclosure, creating and providing an objectbased MS Immersive Experience (MSIE) approach involves the application of a suite of technologies for creation, delivery and rendering of object-based sensory data, which may include sensory objects and corresponding sensory metadata, to the actuators 008. Some examples are described in the following paragraphs.

[0083] Object-Based Representation: In various disclosed implementations, multi-sensory (MS) effects are represented using what may be referred to herein as “sensory objects.’’ According to some such implementations, properties such as layer-type and priority may be assigned to and associated with attached to each sensory object, enabling content creators’ intent to be represented in the rendered experiences. Detailed examples of sensory object properties are described below.

[0084] In this example, system 300 includes a content creation tool 000 that is configured for designing multi-sensory (MS) immersive content and for outputting object-based sensory data 005, either separately or in conjunction with corresponding audio data 011 and / or video data 012, depending on the particular implementation. The object-based sensory data 005 may include time stamp information, as well as information indicating the type of sensory object, the sensory object properties, etc. In this example, the object-based sensory data 005 is not “channel-based” data that corresponds to one or more particular sensory actuators in a playback environment, but instead is generalized for a wide range of playback environments with a wide range of actuator types, numbers of actuators, etc. In some examples, the object-Dolby Ref. D24098W001based sensory data 005 may include object-based light data, object-based haptic data, objectbased air flow data, or object-based positional actuator data, object-based olfactory data, object-based smoke data, object based data for one or more other types of sensor effects, or combinations thereof. According to some examples, the object-based sensory data 005 may include sensory objects and corresponding sensory metadata. For example, if the objectbased sensory data 005 includes object-based light data, the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, or combinations thereof. Although the content creation tool 000 is shown providing a stream of object-based sensory data 005 to the experience player 002 in this example, in alternative examples the content creation tool 000 may produce object-based sensory data 005 that is stored for subsequent use. Examples of graphical user interfaces for a light-object-based content creation tool are described below.

[0085] MS Object Renderer: Various disclosed implementations provide a Tenderer that is configured render MS effects to actuators in a playback environment. According to this example, system 300 includes an MS renderer 001 that is configured to render object -based sensory data 005 to actuator control signals 310, based at least in part on environment and actuator data 004. In this example, the MS renderer 001 is configured to output the actuator control signals 310 to MS controllers 003, which are configured to control the actuators 008. Tn some examples, the MS renderer 001 may be configured to receive light objects and object-based lighting metadata indicating an intended lighting environment, as well as lighting information regarding a local lighting environment. The lighting information is one general type of environment and actuator data 004, and may include one or more characteristics of one or more controllable light sources in the local lighting environment. In some examples, the MS renderer 001 may be configured to determine a drive level for each of the one or more controllable light sources that approximates the intended lighting environment. Some alternative examples may include a separate renderer for each type of actuator 008, such as one renderer for light fixtures, another renderer for haptic devices, another renderer for air flow devices, etc. According to some examples, the MS renderer 001 (or one of the MS controllers 003) may be configured to output the drive level to at least one of the controllable light sources. In some implementations, the MS renderer 001 may beDolby Ref. D24098W001configured to adapt to changing conditions. Some examples of MS tenderer 001 implementations are described in more detail below.

[0086] The environment and actuator data 004 may include what are referred to herein as “room descriptors” that describe actuator locations (e.g., according to an x,y,z coordinate system or a spherical coordinate system). In some examples, the environment and actuator data 004 may indicate actuator orientation and / or placement properties (e.g., directional and north-facing, omnidirectional, occlusion information, etc.). According to some examples, the environment and actuator data 004 may indicate actuator orientation and / or placement properties according to a 3x3 matrix, in which three elements (for example, the elements of the first row) represent spatial position (x,y,z), three other elements (for example, the elements of the second row) represent orientation (roll, pitch, yaw), and three other elements (for example, the elements of the third row) indicate a scale or size (sx, sy, sz). In some examples, the environment and actuator data 004 may include device descriptors that describe the actuator properties relevant to the MS Tenderer 001, such as intensity range and color gamut of a light fixture, the air flow speed range and direction(s) for an air-moving device, etc.

[0087] In this example, system 300 includes an experience player 002 that is configured to receive object-based sensory data 005’, audio data 011’ and video data 012’, to provide the object-based sensory data 005 to the MS Tenderer 001, to provide the audio data 011 to the audio Tenderer 006 and to provide the video data 012 to the video Tenderer 007. In this example, the reference numbers for the object-based sensory data 005’, audio data 011 ’ and video data 012’ received by the experience player 002 include primes (‘), in order to suggest that the data may in some instances be encoded. Likewise, the object-based sensory data 005, audio data 011 and video data 012 output by the experience player 002 do not include primes, in order to suggest that the data may in some instances have been decoded by the experience player 002. According to some examples, the experience player 002 may be a media player, a game engine or personal computer or mobile device, or a component integrated in an television, DVD player, sound bar, set top box, or a service provider media device such as a Chromecast, Apple TV device, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive encoded object-based sensory data 005 along with encoded audio data 011 and / or encoded video data 012. In some such examples, the encoded object-based sensory data 005’ may be received as part of the same bitstream with the encoded audio data Oil’ and / or the encoded video data 012’. Some examples areDolby Ref. D24098W001described in more detail below. According to some examples, the experience player 002 may be configured to extract the object-based sensory data 005’ from the content bitstream and to provide decoded object-based sensory data 005 to the MS Tenderer 001, to provide decoded audio data 011 to the audio Tenderer 006 and to provide decoded video data 012 to the video Tenderer 007. In some examples, time stamp information in the object-based sensory data 005’ may be used — for example, by the experience player 102, the MS Tenderer 001, the audio Tenderer 106, the video Tenderer 107, or all of them — to synchronize effects relating to the object-based sensory data 005’ with the audio data 111’ and / or the video data 112’, which may also include time stamp information.

[0088] According to this example, system 300 includes MS controllers 003 that are configured to communicate with a variety of actuator types using application program interfaces (APIs) or one or more similar interfaces. Generally speaking, each actuator will require a specific type of control signal to produce the desired output from the Tenderer. According to this example, the MS controllers 003 are configured to map outputs from the MS Tenderer 001 to control signals for each actuator. For example, a Philips Hue™ light bulb receives control information in a particular format to turn the light on, with a particular saturation, brightness and hue, and a digital representation of the desired drive level.

[0089] In some examples, room descriptors also may describe the size and orientation of the playback environment itself, to establish a relative or absolute coordinate system to which all objects are positioned. For example, in a living room a display screen may be regarded as the front, in some instances the front and center, and the floor and ceiling may be regarded as the vertical bounds. In some such examples, the room descriptors also may also indicate bounds corresponding with the left, right, front, and rear, walls relative to the front position.According to some examples, the room descriptor also may be provided in terms of a matrix, such as a 3x3 matrix. This room descriptor information is useful in describing the physical dimensions of the playback environment, for example in physical units of distance such as meters. In some such examples, sensory object locations, sensory object sizes, and sensory object orientations may be described in units that are relative to the room size, for example in a range from -1 to 1. Room descriptors may also describe a preferred viewing position, in some instances according to a matrix.

[0090] The types, numbers and arrangements of the actuators 008 will generally vary according to the particular implementation. In some examples, actuators 008 may includeDolby Ref. D24098W001lights and / or light strips (also referred to herein as “luminaires”), vibrational motors, air flow generators, positional actuators, or combinations thereof.

[0091] Similarly, the types, numbers and arrangements of the loudspeakers 009 and the display devices 010 will generally vary according to the particular implementation. In the examples shown in Figure 4, audio data 011 and video data 012 arc rendered by the audio Tenderer 006 and the video Tenderer 007 to the loudspeakers 009 and display devices 010, respectively.

[0092] As noted above, according to some implementations the system 300 may include one or more instances of the control system 110 of Figure 1 that are configured for performing at least some of the methods disclosed herein. In some such examples, one instance of the control system 110 may implement the content creation tool 000 and another instance of the control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement the audio Tenderer 006, the video Tenderer 007, the multi-sensory Tenderer 001, or combinations thereof. According to some examples, an instance of the control system 110 that is configured to implement the experience player 002 may also be configured to implement the audio Tenderer 006, the video Tenderer 007, the multi-sensory Tenderer 001, or combinations thereof.Multi-Sensory Rendering Synchronization

[0093] Object-based MS rendering involves different modalities being rendered flexibly to the endpoint / playback environment. Endpoints have differing capabilities according to various factors, including but not limited to the following:• The number of actuators,• The modalities of those actuators (e.g., light fixture vs. air flow control device vs. haptic device);• The types of those actuators (e.g., a white smart light vs. a red / green / blue (RGB) or red / green / blue / white (RGBW) smart light, or a haptic vest vs. a haptic seat cushion) and• The location / layout of those actuators.

[0094] In order to render object-based sensory content to any endpoint, some processing of the object signals, e.g. intensities, colors, patterns etc., will generally need to be done. The processing of each modality's signal path should not alter the relative phase of certainDolby Ref. D24098W001features within the object signals. For example, suppose that a lightning strike is presented in both the haptics and lightscape modalities. The signal processing chain for the corresponding actuator control signals should not result in a time delay of either type of sensory object signal — haptic or light — sufficient to alter the perceived synchronization of the two modalities. The level of required synchronization may depend on various factors, such as whether the experience is interactive and what other modalities are involved in the experience. Maximum time difference values may, for example, range from approximately 10ms to 100ms, depending on the particular context.Combinations of Lights, Airflow and HapticsCar Examples

[0095] The following examples are described with reference to a car, but are also applicable to other vehicles, such as trucks, vans, etc. In some examples, there may be a user interface on the steering wheel or on a touchscreen near or in the dashboard. According to some examples, the following actuators may be present in the car:1. Individually addressable lights, spatially distributed around the car as follows: o on the dashboard;o under the footwells;o on the doors; ando in the center console.2. Individually controllable air conditioning / heating outlets distributed around the car as follows:o In the front dashboard;o Under the footwells;o In the center console facing the rear seats;o On the side pillars;o In the seats; ando Directed to the windscreens (for defogging).3. Individually controllable seals with vibro-tractile haptics; and4. Individually controllable floor mats with vibro-tactile haptics.

[0096] In this example, the modalities supported by these actuators include the following:• Lights across the individually addressable LEDs in the car, plus the indicator lights on the dash and steering wheel;Dolby Ref. D24098W001• Air flow via the controllable air conditioning vents;• Haptics, including:o Steering wheel: tactile vibration feedback;o Dash touchscreen: tactile vibration feedback and texture rendering; and o Scats: tactile vibrations and movement.

[0097] In one example, a live music stream is being rendered to four users sitting in the front seats. In this example, the MS Tenderer 001 attempts to optimize the experience for multiple viewing positions. During the build-up before the artist has taken the stage and the previous acts have finished, the content contains:• Interlude music;• Low intensity lighting; and• Haptic content representing the moshing of the crowd.

[0098] In addition to the rendered audio and video stream, the light content contains ambient light objects that are moving slowly around the scene. These may be rendered using one of the ambient layer methods disclosed herein, for example such that there is no spatial priority given to any user’s perspective. In some examples, the haptic content may be spatially concentrated in the lower time-frequency spectrum and may be rendered only by the vibro-tactile motors in the floor mats.

[0099] According to this example, pyrotechnic events during the music stream correspond to multi-sensory-sensory content including:• Light objects that spatially correspond to the location of the pyrotechnics at the event; and• Haptic objects to reinforce the dynamism of the pyrotechnics via a shockwave effect.

[0100] In this example, the MS Tenderer 001 renders both the light objects and the haptic objects spatially. Light objects may, for example, be rendered in the car such that each person in the car perceives the light objects to come from the left if the pyrotechnics content is located at the left of the scene. In this example, only lights on the left of the car are actuated. Haptics may be rendered across both the seats and floor mats in a way that conveys directionality to each user individually.Dolby Ref. D24098W001

[0101] At the end of the concert the pyrotechnics are present in the audio content and both pyrotechnics and confetti are present in the video content. In addition to rendering light objects and haptic objects corresponding to the pyrotechnics as above, the effect of the confetti firing may be rendered using the airflow modality. For example, the individually controllable air flow vents of the HVAC system may be pulsed.Living Room Examples

[0102] In this implementation, in addition to an audio / visual (AV) system that includes multiple loudspeakers and a television, the following actuators and related controls are available in the living room:• A haptics vest that the user — also referred to as a player — is wearing;• Haptics shakers mounted to the seat in which the player is sitting;• A (haptics) controllable smart watch;• Smart lights spatially distributed around the room;• A wireless controller; and• An addressable air-flow bar (AFB), which includes an array of individually controllable fans directed to the user (similar to IIVAC vents in the front dashboard of a car).

[0103] In this example, the user is playing a first person shooter game and the game contains a scene in which a destructive hurricane moves through the level. As it does so, in-game objects are thrown around and some hit the player. Haptics objects rendered by the MS Tenderer 001 cause a shockwave effect to be provided through all of the haptics devices that the user can perceive. The actuator control signals sent to each device may be optimized according to the intensity of the impact of the in-game objects, the direction(s) of the impact and the capabilities and location of each actuator (as described earlier).

[0104] At a time before the user is struck by an in-game object, the multi-sensory content contains a haptic object corresponding to a non-spatial rumble, one or more airflow objects corresponding to directional airflow; and one or more light objects corresponding to lightning. The MS Tenderer 001 renders the non-spatial rumble to the haptics devices. The actuator control signals sent to each haptics device may be rendered such that the ensemble of actuator control signals across the haptics array is congruent in perceived onset time, intensity and frequency. In some examples, the frequency content of the actuator controlDolby Ref. D24098W001signals sent to the smart watch may be low -pass filtered, so that they are congruent with the frequency-limited capability of the vest, which is proximate to the watch. The MS tenderer 001 may render the one or more airflow objects to actuator control signals for the AFB such that the air flow in the room is congruent with the location and look direction of the player in the game, as well as the hurricane direction itself. Lightning may be rendered across all modalities as (1) a white flash across lights that are located in suitable locations, e.g., in or on the ceiling; and (2) an impulsive rumble in the user’s wearable haptics and seat shaker.

[0105] When the user is struck by an in-game object, a directional shockwave may be rendered to the haptics devices. In some examples, a corresponding airflow impulse may be rendered. According to some examples, a damage take effect, indicating the amount of damage caused to the player by being struck by the in-game object, may be rendered by the lights.

[0106] In some such examples, signals may be rendered spatially to the haptics devices such that a perceived shockwave moves across the player’ s body and the room. The MS Tenderer 001 may provide such effects according to actuator location information indicating the haptics devices locations relative to one another. The MS Tenderer 001 may provide the shockwave vector and position according to the actuator location information in addition to actuator capability information. According to some examples, a non-directional air flow impulse may be rendered, e.g., all the air vents of the AFB may be turned up briefly to reinforce the haptic modality. In some examples, at the same time, a red vignette may be rendered to the light strip surrounding the TV, indicating to the player that the player took damage in the game.

[0107] Figure 5 shows example elements of another system for the creation and playback of MS experiences. As with other figures provided herein, the types and numbers of elements shown in Figure 5 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, system 500 may be, or may include, one or more devices configured for performing at least some of the methods disclosed herein. In some examples, system 500 may include one or more instances of the control system 110 of Figure 1 that are configured for performing at least some of the methods disclosed herein.

[0108] According to this example, the system shown in Figure 5 is an instance of the system shown in Figure 4. In this example, the system shown in Figure 5 is a “lightscape”Dolby Ref. D24098W001embodiment in which vision (video), audio and light effects are combined to create the MS experience.

[0109] In this example, system 500 includes a lightscape creation tool 100, which is an instance of the content creation tool 000 that is described with reference to Figure 4. The lightscapc creation tool 100 is configured for designing and outputting object-based light data 505’ , either separately or in conjunction with corresponding audio data 111’ and / or video data 112’, depending on the particular implementation. The object-based light data 505’ may include time stamp information, as well as information indicating light object properties, etc. In some instances, the time stamp information may be used to synchronize effects relating to the object-based light data 505’ with the audio data 111’ and / or the video data 112’, which also may include time stamp information.

[0110] In this example, the object-based light data 505’ includes light objects and corresponding light metadata. For example, the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, or combinations thereof. Although the content creation tool 100 is shown providing a stream of object-based light data 505’ to the experience player 102 in this example, in alternative examples the content creation tool 100 may produce object-based light data 505’ that is stored for subsequent use. Examples of graphical user interfaces for a light-object-based content creation tool are described below.

[0111] In this example, system 500 includes an experience player 102 that is configured to receive object-based light data 505’, audio data 111’ and video data 112’, to provide the object-based light data 505 to the lightscape Tenderer 501, to provide the audio data 111 to the audio Tenderer 106 and to provide the video data 112 to the video Tenderer 107.According to some examples, the experience player 102 may be a media player, a game engine or personal computer or mobile device, or a component integrated in an television, DVD player, sound bar, set top box, or a service provider media device such as a Chromecast, Apple TV device, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive encoded object-based light data 505’ along with encoded audio data 111’ and / or encoded video data 112’. In some such examples, the encoded object-based light data 505’ may be received as part of the same bitstream with theDolby Ref. D24098W001encoded audio data 111’ and / or the encoded video data 112’. Some examples are described in more detail below. According to some examples, the experience player 102 may be configured to extract the object-based light data 505 from the content bitstream and to provide decoded object -based light data 505 to the lightscape Tenderer 501, to provide decoded audio data 111 to the audio Tenderer 106 and to provide decoded video data 112 to the video Tenderer 107. In some examples, the experience player 002 may be configured to allow control of configurable parameters in the lightscape Tenderer 501, such as immersion intensity. Some examples are described below.

[0112] In some examples, room descriptors of the environment and light fixture data 104 may describe the size and orientation of the playback environment itself, to establish a relative or absolute coordinate system to which all objects are positioned. For example, in a living room a display screen may be regarded as the front, in some instances the front and center, and the floor and ceiling may be regarded as the vertical bounds. In some such examples, the room descriptors also may also indicate bounds corresponding with the left, right, front, and rear, walls relative to the front position. According to some examples, the room descriptor also may be provided in terms of a matrix, such as a 3x3 matrix. This room descriptor information is useful in describing the physical dimensions of the playback environment, for example in physical units of distance such as meters. In some such examples, sensory object locations, sensory object sizes, and sensory object orientations may be described in units that are relative to the room size, for example in a range from -1 to 1. Room descriptors may also describe a preferred viewing position, in some instances according to a matrix.

[0113] According to this example, system 500 includes a lightscape Tenderer 501 that is configured to render object-based light data 505 to light fixture control signals 515, based at least in part on environment and actuator data 104. In this example, the lightscape Tenderer 501 is configured to output the light fixture control signals 515 to light controllers 103, which arc configured to control the light fixtures 108. The light fixtures 108 may include individual controllable light sources, groups of controllable light sources (such as controllable light strips), or combinations thereof. In some examples, the lightscape Tenderer 501 may be configured to manage various types of light object metadata layers, examples of which are provided herein. According to some examples, the lightscape Tenderer 501 may be configured to render actuator signals for light fixtures based, at least in part, on the perspective of a viewer. If the viewer is in a living room, that includes a television (TV)Dolby Ref. D24098W001screen, the lightscape tenderer 501 may, in some examples, be configured to render the actuator signals relative to the TV screen. However, in virtual reality (VR) use cases, the lightscape tenderer 501 may be configured to render the actuator signals relative to the position and orientation of the user’s head. In some examples, the lightscape tenderer 501 may receive input from the playback environment — such as light sensor data corresponding to ambient light, camera data corresponding to a person’ s location or orientation, etc. — to augment the render.

[0114] In some examples, the lightscape tenderer 501 is configured to receive object-based light data 505 that includes light objects and object-based lighting metadata indicating an intended lighting environment, as well as environment and light fixture data 104 corresponding to light fixtures 108 and other features of a local playback environment, which may include, but are not limited to, reflective surfaces, windows, non-controllable light sources, light-occluding features, etc. In this example, the local playback environment includes one or more loudspeakers 109 and one or more display devices 510.

[0115] According to some examples, the lightscape Tenderer 501 is configured to calculate how to excite various controllable light fixtures 108 based at least in part on the object-based light data 505 and the environment and light fixture data 104. The environment and light fixture data 104 may, for example, indicate the geometric locations of the light fixtures 108 in the environment, light fixture type information, etc. In some examples, the lightscape Tenderer 501 may configured to determine which light fixtures will be actuated based, at least in part, on the position metadata and size metadata associated with each light object, e.g., by determining which light fixtures are within a volume of a playback environment corresponding to the light object’s position and size at a particular time indicated by light object time stamp information. In this example, the lightscape Tenderer 501 is configured to send light fixture control signals 515 to the light controller 103 based on the environment and light fixture data 104 and the object-based light data 505. The light fixture control signals 515 may be sent via one or more of various transmission mechanisms, application program interfaces (APIs) and protocols. The protocols may, for example, include Hue API, LIFX API, DMX, Wi-Fi, Zigbee, Matter, Thread, Bluetooth Mesh, or other protocols.

[0116] In some examples, the lightscape Tenderer 501 may be configured to determine a drive level for each of the one or more controllable light sources that approximates a lighting environment intended by the author(s) of the object-based light data 505. According to someDolby Ref. D24098W001examples, the lightscape Tenderer 501 may be configured to output the drive level to at least one of the controllable light sources.

[0117] According to some examples, the lightscape Tenderer 501 may be configured to collapse one or more parts of the lighting fixture map according to the content metadata, user input (choosing a mode), limitations and / or configuration of the light fixtures, other factors, or combinations thereof. For example, the lightscape Tenderer 501 may be configured to render the same control signals to two or more different lights of a playback environment. In some such examples, two or more lights may be located close to one another. For example, two or more lights may be different lights of the same actuator, e.g., may be different bulbs within the same lamp. Rather than compute a very slightly different control signal for each light bulb, the lightscape Tenderer 501 may be configured to reduce the computational overhead, increase rendering speed, etc., by render the same control signals to two or more different, but closely-spaced, lights.

[0118] In some examples, the lightscape Tenderer 501 may be configured to spatially upmix the object-based light data 505. For example, if the object-based light data 505 was produced for a single plane, such as a horizontal plane, in some instances the lightscape Tenderer 501 may be configured to project light objects of the object-based light data 505 onto an upper hemispherical surface (e.g., above an actual or expected position of the user’s head) in order to enhance the experience.

[0119] According to some examples, the lightscape Tenderer 501 may be configured to apply one or more thresholds, such as one or more spatial thresholds, one or more luminosity thresholds, etc., when rendering actuator control signals to light actuators of a playback environment. Such thresholds may, in some instances, prevent some light objects from causing the activation of some light fixtures.

[0120] Light objects may be used for various purposes, such as to set the ambience of the room, to give spatial information about characters or objects, to enhance special effects, to create a greater sense of interaction and immersion, to shift viewer attention, to punctuate the content, etc. Some such purposes may be expressed, at least in part, by a content creator according to sensory object metadata types and / or properties that are generally applicable to various types of sensory objects — such as object metadata indicating a sensory object’s location and size.Dolby Ref. D24098W001

[0121] For example, the priority of sensory objects, including but not limited to light objects, may be indicated by sensory object priority metadata. In some such examples, sensory object priority metadata is taken into account when multiple sensor objects map to the same fixture(s) in a playback environment at the same time. Such priority may be indicated by light priority metadata. In some examples, priority may not need to be indicated via metadata. For example, the MS Tenderer 001 may give priority to sensory objects — including but not limited to light objects — that are moving over sensory objects that are stationary.

[0122] A light object may, depending on its location and size and the locations of light fixtures within a playback environment — potentially cause the excitation of multiple lights. In some examples, when the size of a light object encompasses multiple lights, the Tenderer may apply one or more thresholds — such as one or more spatial thresholds or one or more luminosity thresholds — to gate objects from activating some encompassed lights.Examples of Using a Lighting Map

[0123] In some implementations a lighting map, which is an instance of the of the actuator map (AM) that includes a description of lighting in a playback environment, may be provided to the lightscape Tenderer 501. In some such examples, the environment and light fixture data shown in Figure 5 may include the lighting map. According to some examples, the lighting map may be allocentric, e.g., indicating absolute spatial coordinate-based light fall-off, whereas in other examples the lighting map may be egocentric, e.g., a light projection mapped onto a sphere at an intended viewing position and orientation. In the case of a sphere, the lighting map may, in some examples, be projected onto a two-dimensional (2D) surface, e.g., in order to utilize 2D image textures in processing. In any case, the lighting map should indicate the capabilities and the lighting setup of the playback environment, such as a room. In some embodiments the lighting map may not directly relate to physical room characteristics, for example if certain user preference-based adjustments have been made.

[0124] In some examples, there may be one lighting map per light fixture, or per light, in a playback environment. According to some examples, the intensity of light indicated by the light map may be inversely correlated to the distance to the center of the light, or may be approximately (e.g., within plus or minus 5%, within plus or minus 10%, within plus or minus 1 %, within plus or minus 20%, etc.) inversely correlated to the distance to the center of the light. The intensity values of the light map may indicate the strength or impact of the light object onto the light fixture. For example, as a light object approaches a lightbulb, theDolby Ref. D24098W001lightscape Tenderer 501 may be configured to determine that the lightbulb intensity will increase as the distance between the light object and the lightbulb decreases. The lightscape Tenderer 501 may be configured to determine the rate of this transition based, at least in part, on the intensity of light indicated by the light map.

[0125] Inside this common rendering space, in some examples the lightscapc Tenderer 501 may be configured to use a dot product multiplication between a light object and the light map for each light to compute a light activation metric, e.g., as follows:

[0126] In the foregoing equation, Y represents the light activation metric, LM represents the lighting map and Obj represents the map of a light object. The light activation metric indicates the relative light intensity for the actuator control signal output by the lightscape Tenderer 501 based on the overlap between the light object and the spread of light from the light fixture. In some examples, the lightscape Tenderer 501 may use the maximum or closest distance, or other geometric metrics, from the light object to the light fixture as part of the determination of light intensity. In some implementations, instead of computing the light activation metric, the lightscape Tenderer 501 may refer to a look-up-table to determine the light activation metric.

[0127] The lightscape Tenderer 501 may repeat one of the foregoing procedures for determining the light activation metric for all light objects and all controllable lights of the playback environment. Thresholding for light objects that produce a very low impact on light fixtures may be helpful to reduce complexity. For example, if the effect of a light object would cause an activation of less than a threshold percent of light fixture activation — such as less than 10%, less than 5%, etc. — the lightscape Tenderer 501 may disregard the effect of that light object.

[0128] The lightscape Tenderer 501 may then use the resultant light activation matrix Y, along with various other properties such as the chosen panning law (either indicated by light object metadata or Tenderer configuration) or the priority of the light object, to determine which objects get rendered by which lights and how. Rendering lights-objects into light fixture control signals may involve:• Altering the luminance of a light-object as a function of the distance it is from the light fixture;Dolby Ref. D24098W001• Mixing the colors of multiple light-objects that are simultaneously (multiplexed) rendered by a single light fixture; or • Altering either of the above based on the light object priority.Dolby Ref. D24098W001RENDERING PARAMETERS

[0129] In addition to the information carried by the light object metadata, the rendering of light-objects can be a function of the settings or parameters of the lightscape Tenderer 501 itself. These may include:• Velocity priority - when this parameter is set, light objects that are moving are given a higher priority than those which are not. Having the velocity priority parameter set enhances the dynamism of the rendered scene;• Color priority - light-objects with higher saturation values will take priority;• Activation threshold - the minimum light activation, Y, that must be achieved in order to activate a light-fixture;• Accessibility - certain colors may be chosen over others to best represent the experience for colorblind users. Certain flash rates may be avoided for those with photo-sensitivities.RENDERING CONFIGURATION (MODES)

[0130] In addition to the information carried by the light object metadata, the lightscape Tenderer 501 may, in some implementations, be configured according to different modes. As used herein, the term “mode" is different from “parameter” in the sense that modes may, for example, involve completely different signal paths, whereas parameters may simply parameterize these signal paths. For example, one mode may involve the projection of all light objects onto a lighting map before determining how / what to render to the light-fixtures, while another mode may only snap the highest-priority lights to the nearest light fixtures. Modes may include;• Modes to support low light-fixture count. In these modes, the rendering parameters and the light object metadata are utilized in order to determine which subset of light-objects are to be rendered and in what manner. Here, the “manner” refers to the trade-off between the spatial, color, temporal fidelity of the most prominent light-objects in the scene;• Modes to support different content types, such as music vs. gaming;• Modes in which multiple light objects may be rendered by a single light fixture (or a single light) with color mixing;• Modes in which only a single light object can be rendered by a single light fixture (or a single light);Dolby Ref. D24098WG01• Modes in which the luminance of the light object is altered as a function of the geometric - or otherwise - distance between the light object and light fixture.

[0131] As noted above, according to some implementations the system 500 may include one or more instances of the control system 110 of Figure 1 that are configured for performing at least some of the methods disclosed herein. In some such examples, one instance of the control system 110 may implement the lightscape creation tool 100 and another instance of the control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement the audio Tenderer 006, the video Tenderer 007, the lightscape Tenderer 501, or combinations thereof. According to some examples, an instance of the control system 110 that is configured to implement the experience player 002 may also be configured to implement the audio Tenderer 006, the video Tenderer 007, the lightscape Tenderer 501, or combinations thereof.

[0132] Figure 6 shows an example of a graphical user interface (GUI) that may be presented by a display device of the lightscape creation tool of Figure 5. As with other figures provided herein, the types and numbers of elements shown in Figure 6 are merely provided by way of example. Other GUIs presented by a lightscape creation tool may include more, fewer and / or different types and numbers of elements. According to some examples, the GUI 600 may be presented on a display device according to commands from an instance of the control system 110 of Figure 1 that is configured for implementing the lightscape creation tool 100 of Figure 5.

[0133] In this example, a user may interact with the GUI 600 in order to create light objects and to assign light object properties, which may be associated with the light object as metadata. According to this example, a user is selecting properties of the light object 630. In this example, the GUI 600 shows the light object 630 in a three-dimensional space 631, the latter of which represents a playback environment. Element 634 shows a coordinate system of the three-dimensional space 631. Accordingly, in this example the light object 630 and the three-dimensional space 631 are being viewed from the upper left.

[0134] A user may interact with the GUI 600 in order to select a position and a size of the light object 630. In some examples, a user may select a position of the light object 630 by dragging the light object 630 to a desired position within the three-dimensional space 631, for example by touching a touch screen, using a cursor, etc. According to some examples, a user may select a size of the light object 630 by selecting the size of the circle (or other shape) thatDolby Ref. D24098W001is shown on the GUI 600 to indicate the outline of the light object 130. In some such examples, a user may decrease the size of the light object 630 via a two-fingered pinch of the outline of the light object 130, may increase the size of the light object 630 via a two- fingered expansion, etc.

[0135] Specifying the position and size of an MS object within an abstracted three-dimensional space, such as the three-dimensional space 631 of GUI 600, allows a content creator to generalize the position and extent of the corresponding MS effects without prior knowledge of the particular playback environment in which the MS effects will be provided. This is an advantage of the MS object-oriented approach of various disclosed implementations. For example, the GUI 600 allows a content creator to specify the position and size of the light object 630 within the three-dimensional space 631, thereby allowing the content creator to generalize the position and extent of the corresponding light effects, without prior knowledge of the particular size of any particular playback environment in which the light effects will be provided, without prior knowledge of the number, type and positions of light fixtures, etc., within the playback environment in which the light effects will be provided, etc. The light fixtures that will potentially be actuated responsive to the presence of the light object 630 at a particular time will be those within a volume of the playback environment corresponding to the position and size / extent of the light object 630.

[0136] According to this example, a user may interact with the color circle 635 of the GUI 600 in order to select the hue and color saturation of the current light object and may interact with the slider 636 in order to select the brightness of the current light object. These and other selectable properties of the light object 630 are displayed in area 632 of the GUI 600. According to this example, the properties of the light object 630 that may be selected via the GUI 600 also include intensity, diffusivity, “feathering,” whether or not the light object is hidden, saturation, priority and layer. Light object layers and priority will be described in more detail below. Generally speaking, light object layers may be used to group light objects into categories such as “ambient,” “dynamic,” etc. Light object priority may be assigned by a content creator and used by a Tenderer to determine, for example, which light object(s) will be presented when two or more light objects are simultaneously active and are simultaneously encompassing an area that includes the same light fixture.

[0137] Area 640 of the GUI 600 indicates time information corresponding to each of a plurality of light objects that are being created via the lightscape creation tool. In thisDolby Ref. D24098W001example, light objects are listed on the left side of the area 640, along a vertical axis, and time is shown along a horizontal axis. In this example, four-second time intervals are delineated by vertical lines. Here, time information for each light object is shown as isolated or connected diamond symbols or lines along a series of horizontal rows, each of which corresponds to one of the light objects indicated on the left side of the area 640. The line 633, for example, indicates that light object 3 will be displayed starting between 39 and 40 seconds and will be continuously displayed until almost 1 minute and 6 seconds. The diamond symbols to the right of the line 633 indicate that light object 3 will be displayed discontinuously for the next few seconds.

[0138] Figure 7 shows another example of a GUI that may be presented by a display device of the lightscape creation tool of Figure 5. As with other figures provided herein, the types and numbers of elements shown in Figure 7 are merely provided by way of example. Other GUIs presented by a lightscape creation tool may include more, fewer and / or different types and numbers of elements. According to some examples, the GUI 700 may be presented on a display device according to commands from an instance of the control system 110 of Figure 1 that is configured for implementing the lightscape creation tool 100 of Figure 5.

[0139] In this example, the GUI 700 represents an instant in time during which light fixtures in an actual playback environment are being controlled according to light objects that have been created by an implementation of the lightscape creation tool 100 of Figure 5. An image of the playback environment is shown in area 705 of the GUI 700. Various light fixtures 708 and a television 71 are shown in the playback environment of area 705. The particular instant in time is shown by vertical line 742 of area 740. At this time, the vertical line 742 intersects with horizontal lines 744a, 744b, 744c, and 744d, indicating that the light being provided in the corresponding light objects 1, 4, 5 and 7 are being played back. Area 732 indicates light object properties.

[0140] At the instant in time that is depicted in Figure 7, the left side of the playback environment shown in area 705 is being illuminated by blue light. This lighting effect corresponds, at least in part, to the effect of the light object 730 shown within the three-dimensional space 731 and the corresponding color selected in the “color picker” and shown in area 732.

[0141] According to this example, video data and audio data are also being played back in the audio environment, and the playback of rendered light objects is being synchronized withDolby Ref. D24098W001playback of the video data and audio data. In this example, an image of the played-back video is shown in area 710 of the GUI 700. The video may, for example, be played back by the television 715.

[0142] In some examples, a user may be able to interact with the GUI 700 in order to adjust light object properties, add or delete light objects, etc. For example, a user may cause the playback to be paused in order to adjust light object properties. In some alternative examples, a user may need to revert to a GUI such as the GUI 600 of Figure 6 in order to adjust light object properties, add or delete light objects, etc.

[0143] In the example described with reference to Figure 7, although the GUI 700 was being presented a display device corresponding to the lightscape creation tool 100 of Figure 5, light objects, audio and video were being rendered in an actual, real-world environment.Accordingly, in some implementations the example described with reference to Figure 7 may also involve at least some of the “downstream” rendering and playback functionality that can be provided by other blocks of Figure 5, including but not limited to that of the lightscape Tenderer 501, the light controller APIs 103 — which may in some instances be implemented by the same device that implements the lightscape Tenderer 501 — the light fixtures 108, the audio Tenderer 106, the loudspeakers 109, the video Tenderer 107 and the display device(s) 510. In some such examples, the processes described with reference to Figure 7 also may involve functionality of the experience player 102 of Figure 5.

[0144] In some alternative implementations, the example described with reference to Figure 7 may also involve at least some of the “downstream” rendering and playback functionality that can be provided by other blocks of Figure 4, including but not limited to that of the MS Tenderer 001, the MS controller APIs 003 — which may in some instances be implemented by the same device that implements the MS Tenderer 001 — the light fixtures 008, the audio Tenderer 006, the loudspeakers 009, the video Tenderer 007 and the display device(s) 010. In some such examples, the processes described with reference to Figure 7 also may involve functionality of the experience player 002 of Figure 4.

[0145] The playback environment of area 705 is one example of what may be referred to herein as a “reference environment” or as a “reference room,” to which a content creator may refer when evaluating authored sensory effects, including but not limited to lighting effects. In some examples, a creation tool may provide a content creator with various options for reference environments, each of which has varying capabilities. For example, a contentDolby Ref. D24098W001creator may find it useful to evaluate how sensory content would be played back in a “maximalist” reference environment, having a large number of light fixtures in multiple locations with the reference environment, as well how the same sensory content would be played back in one or more reference environments having fewer and / or different light fixtures.

[0146] Examples of Creation Tool Features Relating to Creating and Visualizing Primitives

[0147] Some sensory objects may be defined, at least in part, according to their size, shape and orientation. Some disclosed creation tools provide visualization of sensory object types and properties during the content creation process. For example, some disclosed creation tools allow a user to create and visualize sensory object types as parametric or nonparametric — also referred to herein as “arbitrarily-defined” spatial primitives. “Parametric” spatial primitives, such as spheres, ellipsoids, cylinders, rectangular prisms and bicones, can be defined by simple geometric parameters. “Arbitrarily-defined” spatial primitives may, for example, be defined by a set of vertices that may be referred to herein as a “mesh.”

[0148] Figure 8 shows examples of spatial primitive visualizations within a reference environment. In this example, the spatial primitive visualizations are shown in a reference environment 801 of a GUI window 800. The GUI window 800 may, for example, be presented on a display by a control system that is executing software for implementing a content creation tool. The spatial primitives 802 shown in the reference environment 801 represent parametric sensory objects and the spatial primitive 803 represents an arbitrarily-defined sensory object, which is in the shape of a monkey head in this example. Some disclosed creation tools provide controls (such as one or more windows, one or more virtual tools (such as one or more dials, sliders, etc.) or other features of a GUI) to define the exact geometry of parametric primitives, such as controls to specify properties such as:• Position;• Rotation; and• Size-which may be an array of dimensions, such as (a) radius for a sphere (b) length, width and depth for a rectangular prism, etc.Non-parametric primitives may be represented as arbitrary 3D meshes. In some examples, arbitrary 3D meshes may be created in one or more separate 3D modelling tools and imported into a creation tool.Dolby Ref. D24098W001

[0149] Examples of Creation Tool Features Relating to Creating and Visualizing Volumetric Fields

[0150] The term “volumetric field” as used herein refers so a field within a volume, the field being defined by functions. The functions may, for example, be parameterizable by a content creator. Volumetric fields may or may not be spatially uniform, depending on the particular implementation. In some examples, a content creator may specify in the content metadata whether a volumetric field is spatially uniform. Functions which may be parameterizable across the volumetric field include, but are not limited to, the following:• Spatially modulated basis functions, such as sinusoidal functions that are a function of the spatial position across the field;• Temporally modulated basis functions, such as sinusoidal functions that are a function of time;• Signal mixing functions;• Signal multiplication functions;• Activation functions, such as:o Non-linear transformations e.g. a = x2; oro Non-linear thresholding, e.g., a = min(x, b); and / or• Deterministic generators parameterised across the volume, e.g., pseudo-random noise.

[0151] The volumetric field may be sampled at spatial intervals, which may be either uniform — that is, sampled at regular intervals across the space of the endpoint — or non-uniform. Non-uniform sampling may, for example, be based on information in the content metadata. In one such example, information in the content metadata may indicate, e.g., sampling centred at a display screen, such as a television (TV) display screen. In some examples, the sampling may be log-space sampling. According to some examples, the volumetric field may be sampled temporally each time the Tenderer runs a process call.

[0152] Volumetric fields, volumetric textures, or both, can be used to augment and / or modulate the properties of spatial objects across an endpoint. For example, a volumetric field may be created in which the amplitude is spatially and temporally modulated. This amplitude may then be applied to any spatial objects in the endpoint and only then will the spatial and temporal modulations affect the actual actuators. One example may involve “flicker” modulating of all light objects in a room.

[0153] The output of a volumetric field can be a scalar or a vector. In some examples, the output of a volumetric field may be used to directly affect an actuator. For example, aDolby Ref. D24098W001volumetric field may produce an RGB color tuple, which may be directly applied to one or more actuators. In some examples, the output of a volumetric field may be used to modulate one or more actuators. For example, a scalar volumetric field may modulate the amplitude of the actuators in a playback environment.

[0154] Some disclosed creation tools provide uscr-sclcctablc controls relating to volumetric fields. Some creation tool controls may include GUI elements that allow a user to select parameters of a chosen function relating to a volumetric field, such as the density, frequency and / or amplitude of a sinusoidal function. Some such controls may include GUI elements that allow a user to select one or more object properties to modulate according to select volumetric field parameters.

[0155] Some disclosed creation tools provide visualisation relating to volumetric fields. Visualisation is important for content creators to evaluate effects created by selected volumetric field parameters and to determine whether the selected volumetric field parameters produce desired effects. In some examples, visualisation provided by a creation tool may not require a user to fully understand the underlying functions corresponding to selected volumetric field parameters. According to some examples, visualisation relating to the selected volumetric field parameters is in a separate window of a GUI from a window depicting a spatial representation of a corresponding sensory object.

[0156] Figure 9 shows an example of two-dimensional (2D) visualisation relating to volumetric fields. In this example, the visualizations are shown in a reference environment 901 that is displayed in a GUI window 900. The GUI window 900 may, for example, be presented on a display by a control system that is executing software for implementing a content creation tool. According to this example, the GUI window 900 is showing a two-dimensional cross-section in the x,y plane of the reference environment 901. In some examples, another portion of the GUI may allow a user to select a z value through which the x,y cross-section will be shown in the GUI window 900, an x value through which a y,z cross-section will be shown in the GUI window 900, or a y value through which an x,z crosssection will be shown in the GUI window 900. In this example, areas 905 represent high-amplitude portions of a selected volumetric field and areas 910 represent low-amplitude portions of the selected volumetric field. For example, if the selected volumetric field is being used to modulate light objects, the areas 905 would represent brighter regions of the reference environment 901 due to the effects of the selected volumetric field and areas 910Dolby Ref. D24098W001would represent darker portions of the reference environment 901 due to the effects of the selected volumetric field. If the selected volumetric field is time-varying, Figure 9 may represent an instant in time.

[0157] Figure 10 shows an example of three-dimensional (3D) visualisation relating to volumetric fields. In this example, the visualizations arc shown in a reference environment 1001 that is displayed in a GUI window 1000. The GUI window 1000 may, for example, be presented on a display by a control system that is executing software for implementing a content creation tool. According to this example, the GUI window 1000 is showing a three-dimensional lattice that represents the amplitudes of a selected volumetric field in the reference environment 1001. In this example, the amplitudes of the selected volumetric field are constant in each x,y plane of the reference environment 1001 and increase along the z axis from a minimum amplitude plane 1005 to a maximum amplitude plane 1010. If the selected volumetric field is time-varying, Figure 10 may represent an instant in time.

[0158] Examples of Creation Tool Features Relating to Creating and Visualizing Volumetric Textures

[0159] Volumetric textures are similar to volumetric fields in the sense that they provide either a scalar or vector field over a spatial coordinate domain of the endpoint. The main difference is that the field corresponding to a volumetric texture is derived from data, not by functional parameters provided by the creator. Volumetric textures may be combined with volumetric fields to create desired effects. For example, a scalar volumetric texture may be used to define some arbitrary function over the spatial domain which drives time-frequency oscillators at particular spatial locations.

[0160] An image or other data set that can be used to represent a volumetric texture may be referred to herein as a “volumetric texture asset.” A content creator may create or import a volumetric texture asset and then determine how the volumetric texture asset will be sampled to produce a volumetric texture. For example, a content creator may choose to create or import a 2D or 3D image and use this as the basis for the volumetric field. A greyscale image, for example, could produce a scalar volumetric field.

[0161] Some disclosed creation tools to support volumetric textures provide a user with the ability to import a 2D or a 3D volumetric texture asset. Alternatively, or additionally, someDolby Ref. D24098W001disclosed creation tools to support volumetric textures provide a user with the functionality to create a 2D or a 3D volumetric texture asset.

[0162] Figures 11 A and 1 IB provide examples of importing and using a 2D volumetric texture asset. According to these examples, Figure 11 A shows a 2D volumetric texture asset 1105, which is a 2D image in this example, as well as regions 1110a and 1110b within the 2D volumetric texture asset 1105, which are displayed in a GUI window 1100a. In this example, region 1110a is a purple region of the 2D volumetric texture asset 1105 and 1110b is an orange region of the 2D volumetric texture asset 1105.

[0163] Figure 11B shows the 2D volumetric texture asset 1105 mapped into a reference environment 1101 that is displayed in a GUI window 1100b. In this example, the content creator has chosen to place the 2D volumetric texture asset 1105 in an x,y plane that is parallel to the floor of the reference environment 1101. The content creator may, for example, choose to create a 3D volumetric texture for the reference environment 1101 by repeating values of the 2D volumetric texture asset 1105 across the Z axis. In other examples, the content creator may choose to create a 3D volumetric texture for the reference environment 1101 by modifying values of the 2D volumetric texture asset 1105 across the Z axis, e.g., by selecting a volumetric texture modification type (such as a function of z) from a variety of possible volumetric texture modification types (which may include a linear function of z, a sinusoidal function of z, etc.)

[0164] Examples of Creation Tool Features Relating to MS Content Based on Images and Videos

[0165] Images and videos differ in nature to volumetric textures (and fields) in the sense that images and videos are generally not evaluated or sampled across the spatial domain of an endpoint. Instead, images and videos are generally rendered to one or more actuators comprising an array of actuators that provide sufficient resolution and coverage to render an image or video. The reader will note that, as used herein, the terms “images” and “videos” are not necessarily optical images and videos. For example, an image may simply be a 2D or 3D signal with regular uniform spatial sampling. One example of a non-optical image is a haptic signal that is defined across an array of spatial points that could, for example, be rendered to an array of actuators on a vest. A video is simply an image in which the signal changes over time.Dolby Ref. D24098W001

[0166] A content creator may import an image or video into their sensory creation tooling and associate the image or video with a sensory object. In some instances, a content creator may create a sensory object corresponding with an image or video. The position and size of this sensory object can determine the bounding region in which the sensory object is able to be rendered upon a suitable array of actuators.

[0167] A content creator may also determine the way in which an image or video is rendered upon an array of actuators, for example by defining one or more of the following in content metadata corresponding to the image or video:• The method of resampling / resizing the image or video to an actuator array having a different aspect ratio than that of the asset:• Minimum capabilities of the actuator array required to render the image or video, which may include, but are not limited to:o The number of elements in the actuator arrayo The maximum spatial distance between elements of the actuator array;o The maximum spatial distance between actuator sub-arrays: and / or o The actuation capacity of the actuator array, such as the color gamut for light fixtures or the frequency range for haptic actuators.

[0168] Figure 12 shows an example of an image that can be used during a process of sensory content creation. According to this example, Figure 12 shows an image 1205 in a reference environment 1201 that is displayed in a GUI window 1200. By interacting with one or more features of the GUI window 1200 and other related GUIs, a content creator may specify details of the way in which the image 1205 is rendered, how to associate the image 1205 with a sensory object, etc. The image 1205 may, for example, be cropped, resampled and / or resized, according to input received via the GUI window 1200 from a content creator.

[0169] According to the example shown in Figure 12, the GUI window 1200 includes a properties sub- window 1210 with which a user may interact in order to import an image and specify related properties. In this example, the properties sub-window 1210 includes an image field in which a user may indicate the file name of a desired image and a resampling field in which a user may indicate a desired type of resampling for the selected image: in this case, the user has selected tri-cubic resampling. According to this example, the properties sub-window 1210 includes a “bed-mask” field, which allows a user to snap the selected image to a specific wall of the reference environment 1201.Dolby Ref. D24098W001

[0170] In this example, the properties sub- window 1210 includes four additional fields in which a user may indicate minimum actuator requirements of a playback environment that are necessary for the selected image to be reproduced in the playback environment.According to this example, the properties sub-window 1210 includes a field for specifying a minimum number of actuator elements, a field for specifying a maximum distribution of actuator elements (“Max dist elements”), a field for specifying a maximum distribution of actuator sub-arrays and a field for specifying one or more actuator capacity requirements. In the examples shown in Figure 12, a “Max dist elements” of 0.3 x means that the maximum distribution of actuator elements is 30% of the total x dimension of 1, using normalized dimensions for the playback environment. Similarly, a “Max dist elements” of 0.3 y means that the maximum distribution of actuator elements is 30% of the total y dimension of the playback environment.

[0171] As with other figures provided herein, the types and numbers of elements shown in Figure 12 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements.

[0172] Examples of Creation Tool Features Relating to Specifying Sensory Object Parameter Modifiers

[0173] In addition to hand- authoring parameter values along a timeline, a content creator may, in some instances, want to define multisensory object behaviour by specifying modifier parameters which are then applied to defined sensory object parameter values. The sensory object parameter values to be modified may, for example, include sensory object size, sensory object shape, sensory object intensity (e.g., brightness), etc. In the context of a lighting experience, other examples of light object parameters to be modulated may include color, spatial position, transparency, layering, and / or various other sensory object effect parameters.

[0174] Some examples of parameter modification may involve one or more of the following:• Defining the shape and / or length of a curve (such as an envelope);• Mapping a defined curve to a parameter as an override, offset, or scalar and then defining the curve’s starting point on a timeline;Dolby Ref. D24098W001• Applying recurring modifier curves (such as low-frequency oscillators (LFOs)) of various shapes to a parameter with a given modifier speed. In some examples, curves may reset at specified points on the timeline;• According to some examples, modifier parameters may themselves be changed over time either by curves on a timeline or by other modifiers. Examples of modifier parameters to be themselves modified may include speed, scale, and shape.

[0175] Figures 13A and 13B show an example of a GUI for sensory object parameter modification. According to this example, a single GUI 1300 is being shown. However, the GUI 1300 is being presented on two drawing sheets, corresponding to Figures 13 A and 13B, in order to show all of the details of the GUI 1300. In this example, the GUI 1300 includes the following elements:• 1301 : a window for presenting a view of a sensory object, such as a top-down view;• 1302: a window for selecting a “base position” for a sensory object before modification;• 1310: a window for selecting parameters of an intensity envelope ENV1, including a start 1303 of the intensity envelope and an end 1304 of the intensity envelope;• 1305: a window for selecting parameters relating to LFO, including oscillation frequency, phase and intensity, as well as LFO-related mappings to x and y position, sensory object size and sensory object speed;• 1306: a window for selecting parameters relating to mappings of input Musical Instrument Digital Interface (MIDI) data to sensory object velocity and intensity; • 131 : a window for selecting parameters of an envelope ENV2; and• 1307 : a window for selecting ENV2 mappings.

[0176] As with other figures provided herein, the types and numbers of elements shown in Figure 13 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements.

[0177] Figures 14A and 14B show examples of two different methods for achieving substantially the same sensory object movement trajectory. In the example shown in Figure 14A, a content creator may interact with the GUI 1400a in order to select a series of sensory object positions to define the sensory object movement trajectory 1405. In this example, the user has selected x,y,z coordinates of sensory object positions A through F for corresponding instants of time along the timeline 1410.Dolby Ref. D24098W001

[0178] According to the example shown in Figure 14B, a content creator may interact with the GUI 1400b in order to select a start time 1415 and a modulation function to define the sensory object movement trajectory 1405. In this example, the content creator has selected a linear modulation function to define the sensory object movement trajectory 1405.According to some examples, the start time 1415 may correspond with a “trigger event” that is scheduled to begin at the start time 1415. The trigger event may, for example, correspond with a movie scene change, a musical transition, a game event, etc.

[0179] According to some examples, at least some types of stream parameter modifiers may be specified by a content creator to take effect following a transition function or an interpolation function, e.g., over a time interval after the transition function or the interpolation function. The transition function or the interpolation function may, for example, correspond to a change in video and / or audio content that is being presented. For example, if the content includes gaming content, the transition function or the interpolation function may correspond to a change in the game level, an event in the game (such as a player moving from one virtual room to another), etc. If the content includes movie or television program content, the transition function or the interpolation function may correspond to a scene change. In some examples, the specification of the transition function or the interpolation function may also be specified as part of the stream parameter modifier data, either in the bit-stream or via an interactivity API.

[0180] Interactive Experiences

[0181] Interactivity adds another layer of complexity to the creative palette for creation of multi-sensory experiences by allowing additional input to feed into an experience in a nonlinear way. Such an input may be an event trigger, sensor data or modifier. In some examples, interactivity-related inputs may be defined and linked by content creators via interactivity APIs across multiple assets, across a single asset or across sensory objects within an asset. According to some examples, an interactive experience may may be implemented, at least in part, according to state machine logic that is configured to trigger various assets with particular timing and / or modifiers.

[0182] Figures 15A and 15B show example states of a state machine that may be implemented according to some implementations. In these examples, Figure 15A shows state 1 of the state machine 1500 at a first time and Figure 15B shows state 2 of the state machine 1500 at second time. According to these examples, the state machine 1500 is implemented by an instance of the control system 110 of Figure 1. In some examples, the state machineDolby Ref. D24098W0011500 may be implemented by an instance of the control system 110 that is also configured to implement the experience player 002 of Figure 4 or the experience player 102 of Figure 5.

[0183] In these examples, the state machine 1500 includes a state module 1510, a dynamic interactive API module 1515, state machine logic 1520 and assets 1525. According to these examples, the state machine 1500 is shown receiving external inputs 1505. As with other figures provided herein, the types and numbers of elements shown in Figures 15A and 15B are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. For example, although no transitions to state 1, or from state 2, are shown, some examples may involve transitions to state 1, from state 2, additional states, etc. Although Figures 15A and 15B show the dynamic interactive API module 1515 interacting with a set of assets, in other implementations the dynamic interactive API module 1515 may interact with a set of sensory objects, or with multiple sets of sensory objects.

[0184] In some examples, the state module 1510 is configured to describe how a particular experience, corresponding to a particular state (such as State 1) is set up and the assets 1525 corresponding to that state. In some such examples, the state module 1510 may include information regarding the mappings from external inputs 1505 to the assets 1525 via the dynamic interactive API module 1515. The static machine logic module 1520 may, in some examples, describe how to transition from one state to another (such as from State 1 to State 2) to provide a different experience, for example by fading out one or more of the assets 1525, fading in one or more of the assets 1525, changing the set of available assets, changing the mapping of inputs to the assets, changing the available inputs, changing the set of states that can be transitioned to etc. Each state can transition to a plurality of other states. The state machine can contain many potential states. Typical numbers range from a 2 up to hundreds.

[0185] According to some examples, external inputs 1505 to the dynamic interactive API module 1515 may include inputs corresponding to modulating the assets 1525 of a state. Such modulation-type external inputs 1505 may, in some examples, be direct user controls such as a fader. In other examples, the external inputs 1505 may be or may include, one or more sensor inputs, such as inputs from a sensor that indicates engine revolutions per minute (RPM).

[0186] In this example, the state machine logic 1520 is configured to respond to commands or instructions from the dynamic interactive API module 1515, for example commands regarding transition events. In some examples, the state machine logic 1520 includes instructions, such as software, for implementing responses to commands from the dynamicDolby Ref. D24098W001interactive API module 1515. The state machine logic 1520 may be configured to update its state, and may also start, stop or modulate the assets 1525 being provided, if so required. A change in state can result in the dynamic interactive API module 1515 changing, which is why it is referred to as a “dynamic” interactive API module 1515. According to some examples, the state machine logic 1520 may provide instructions to the dynamic interactive API module 1515, for example if a change in the dynamic interactive API module 1515 would be required for a different state. However, in some examples, the API may be the same for multiple states.

[0187] In this example, the state machine logic 1520 is configured to generate data corresponding to transition events, which are referred to herein as “transition events 1530.” The transition events 1530 may be generated by the state machine logic 1520 after the state machine logic 1520 determines whether certain logical conditions are met or “true,” e.g., based at least in part on inputs from the dynamic interactive API module 1515. The state machine logic 1520 may, for example, determine whether inputs from the dynamic interactive API module 1515 and the current state mean that the state machine 1500 should enter into another state. In some examples, the state machine logic 1520 may be configured to test multiple logical conditions at any given moment, e.g., should the state machine 1500 transition from state N into state M, O, P or K ? In other words, the state machine logic 1520 is not necessarily a linear state machine in which one state can only transition to another. The logical condition(s) tested by the state machine logic 1520 to cause a transition event 1530 may, in some examples, include a plurality of trigger events and other conditions such as user conditions (e.g., a user setting indicating “never enter a scary scene”) and state conditions (the state machine 1500 cannot move into state K from state N). A “trigger event” is when some input to the state machine 1500 is triggered. As noted above, a trigger event may also be a condition that is tested by the state machine logic 1520. There may be many types of trigger events. In some examples, a trigger event can be automatically generated by an event (e.g., asset N just finished playing), or by an elapsed time interval (e.g., it has been 10 minutes since the user made an input). A trigger event may, in some examples, be directly caused by user input (e.g., the user just selected option 3 on the menu).

[0188] The assets 1525 may be one or more effects, such as lighting effects, haptic effects, etc., that correspond to a particular state. According to the example shown in Figure 15A, while in state 1 the assets 1525 of the state machine 1500 include assets A, B and C. After a transition event 1530, the state machine 1500 enters state 2, during which the assets 1525 of the state machine 1500 include assets A, D and E. The transition event 1530 may, forDolby Ref. D24098W001example, correspond to a change in a game level, a scene change in a movie or a television program, etc. Assets B and C may, for example, be lighting and haptic effects, respectively, that correspond with state 1, whereas assets D and E may, for example, be lighting effects that correspond with state 2.

[0189] As noted above, according to some examples each bitstream, whether pre- authored or live-streamed, may be modified interactively, e.g., by using one or more APIs that are implemented by the dynamic interactive API module 1515. In some examples, each of a plurality of multi-modal player stream instances may include an instance of the state machine 1500 that is configured to maintain the state of any sensory objects that the multi-modal player stream instance has received. According to some examples, the state machine 1500 instance — or another module that is implemented by a multi-modal player stream instance — may be configured to interpolate one or more sensory object parameters from a current state to a new state over a specified time interval, for example by using a parameterized transfer function. In some examples, the state machine 1500 instance — or another module that is implemented by the multi-modal player stream instance — may be configured to modify perstream sensory object parameters in an absolute, relative (offset) or scaled manner. The ability to have an arbitrary number of configurable streams unlocks new and novel creative authoring possibilities.

[0190] Creation Tool Features Relating to Trigger-Based or Event-Based Modifiers

[0191] Some disclosed examples involve extending the trigger-based object modifier concept such that modifiers can be applied at the asset or sensory object level as a post-processing step. Creation tool implementations that provide such functionality can provide convenience and efficiency to a content creator by allowing the content creator to re-use of assets or sensory objects with only slight variations. For example, some creation tool implementations may allow the content creator to configure a multi-modal player stream instance to loop over all or part of a bitstream. For example, a bitstream may contain an instruction, metadata, etc., which signals the rcndcrcr to rewind the bitstream to a specific time (which may be an absolute time or a relative time, such a time relative to an event) and restart parsing from there. According to one such example, a sensory object or an asset may be associated with an indicator signal (a “turn” signal) of a vehicle of a game. After being triggered, the sensory object or asset may loop until the vehicle completes the turn.Dolby Ref. D24098W001

[0192] In some examples, a looping process may be controlled, in part, according to a counter (e.g., evaluate a given subsection of a bitstream N times before continuing). Some such creation tool implementations may be supported by a nested asset format such as one of those disclosed herein, e.g., via specific metadata fields. Processing could be specified as absolute (overriding previous values) or relative (cither additively offsetting or multiplicatively scaling previous values). In some examples, modifiers may also be parametrised to enable adaptive post-processing. Some looping examples include:• Fading an asset in and out by modifying its intensity; and• Modification of a single property across one, multiple or all objects in an asset such as incrementing size by 0.1 for all objects. In some examples, an interchange format may retain information about all assets down to the sensory object level so that such object-level modification is possible.

[0193] According to some examples, looping or other modifications may also be implemented according to a timeline, such that the “triggers” are points in time instead of, or in addition to, events. However, in some examples the looping or other modifications may nonetheless be linked to external input for future control. This use of modifiers across levels may be used both according to interactivity as well as for static content, in order to allow reuse of assets and / or particular sensory objects with some variation across different parts of the same content stream or across multiple projects.

[0194] Visual Scripting

[0195] Some creation tools are configured to provide content creators with one or more ways to interact with an interactive API and to describe creator intent. According to some such examples, a creation tool may provide one or more GUIs that link interactive APIs to assets, triggers and / or modifiers. Some such GUIs may have a visual scripting interface, such as a tree-based visual interface, to allow the content creator to link interactive APIs to assets, triggers and / or modifiers. As described elsewhere herein, a modifier may, in some examples, be applied at the asset level or at the sensory object level, and may also be controlled according to a timeline, triggers and / or external input.

[0196] Figure 16 shows an example of a creation tool GUI that includes a visual scripting interface. According to this example, the GUI 1600 allows a content creator to implement a state machine whereby the content creator can define an experience as a series of states. Depending on the current state, the interactive APIs may be configured to do different things,Dolby Ref. D24098W001such as trigger different assets, modulate different properties of assets by applying modifiers, etc. According to this example, the GUI 1600 represents states 1620a, 1620b, 1620c and 1620d, which are a beach state, a trail state, a transition state and a cave state in this example. In this example, each of the states 1620a, 1620b, 1620c and 1620d is shown with a corresponding one of the interactive GUI windows 1615a, 1615b 1615c and 1615d.

[0197] According to this example, a state may or may not have a corresponding asset. In this example, each of the states 1620a, 1620b and 1620d has at least one corresponding asset, and the state 1620c does not have a corresponding asset. Instead, the state 1620c is a transitional state between the beach state 1620b and the cave state 1620d.

[0198] Here, the interactive GUI windows 1615a— 1615d each include an asset name field, a modifier field, a delay field, a loop field and a trigger field with which a user, such as a content creator, may interact to specify properties of a state. A user may, for example, select an asset name from a list of available asset names in the asset name field. A user may, for example, indicate a trigger event in the trigger field and may indicate a desired time delay for that trigger event in the delay field. Delays may, for example, be selected offsets in time (e.g., 0 seconds, 1 second, 2, seconds, 5 seconds, etc.) from a selected trigger event. In some examples, a delay may be specified according to combinatorial logic (e.g. if interactive API gets hit AND the experience is at least 5 seconds AND another interactive API was hit in the last 30 seconds, apply the delay).

[0199] In the example shown in Figure 16, an asset is not necessarily unique to a state.According to this example, the states 1620a and 1620b both correspond to asset B. In this example, the modifier field of the interactive GUI window 1615a is being updated according to a color modifier 1610a, which has been indicated via external input 1605, which is an input from the playback environment such as input from a person playing a game, input from one or more sensors of the playback environment, etc. This is the only change between the properties specified in the interactive GUI window 1615a and the properties specified in the interactive GUI window 1615b: in both instances, a time delay of zero has been selected and the loop field indicates that a loop will be applied. Therefore, although the audio and video presentation for the beach state 1620a and the trail state 1620b may differ in other respects, the lightscape experience for the beach state 1620a will differ from that of the trail state 1620b only according to the color modification indicated by the color modifier 1610a.Dolby Ref. D24098W001

[0200] According to this example, the cave state 1620d has two corresponding assets, asset C and asset D. Asset C may, for example, be a dark — but not completely black — scene and asset D may be a torch, such as a torch with a red-orange flame. Assets C and D, like the other assets, may be created separately and combined and / or modified according to the content creator’s creative intent.

[0201] As with other figures provided herein, the types and numbers of elements shown in Figure 16 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. For example, similar GUIs may present interactive GUI windows having more, fewer and / or different types of fields. In some alternative examples, a modifier may be applied at the sensory object level. For example, a content creator may be able to specify that only the sky color may be different between the beach state 1620a and the trail state 1620b.

[0202] Human machine interfaces (HMIs) may also be incorporated into interactive experiences, such as having dynamic menus or displays which may be presented to a user in the playback environment and which change based on state or assets. In one example, an HM1 may state “you are now entering a cave” when the cave state 1620d is starting to be presented. Such HMIs may be linked using visual scripting such as shown in Figure 16.

[0203] Examples of Scheduling Assets and Asset Modifiers on a Timeline

[0204] Figures 17 A and 17B show examples of scheduling assets and asset modifiers on a timeline. As with other figures provided herein, the types and numbers of elements shown in Figures 17A and 17B are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements.

[0205] In these examples, the GUI 1700a and the GUI 1700b both include an asset library window 1705 that is configured for displaying available assets, an object designer window 1710 that shows the current position of sensory objects at the indicated time and a video preview window 1715 that shows the current video frame at the indicated time. Here, the time is indicated by the time indicator 1730 of the timeline windows 1725a and 1725b.

[0206] According to these examples, a user, such as a content creator, may interact with the asset library window 1705 to select an asset and may interact with the asset modifier window 1720a or 1720b to choose modification parameters for a selected asset. In these examples, timeline windows 1725a and 1725b indicate the selected starting and ending times forDolby Ref. D24098W001reproducing selected assets. In these examples, a user has selected asset A for playback at the indicated time, so the object designer window 1710 shows the current positions of the two sensory objects of asset A, sensory objects 1712a and 1712b, in a reference environment 1714 at the indicated time.

[0207] In the example shown in Figure 17A, a content creator may interact with the property field of the asset modifier window 1720a to select a property of a selected asset for modification and may interact with the envelope field of the asset modifier window 1720a to select ramp-up, steady state and ramp-down time intervals of an asset modification envelope 1722. The asset modifier window 1720a also includes an “Offset” field for indicating a time delay. In this example, a user has selected “intensity” via the property field, so the asset modification envelope 1722 indicates ramp-up, steady state and ramp-down time intervals for an intensity modification of the selected asset, which is asset A in this example. In this example, the selected intensity modification will apply to both sensory object 1712a and sensory object 1712b. In some alternative examples, a user may be able to select modifications on a per-object basis. According to this example, a user has scheduled another playback of asset A, then a playback of asset B, followed by a playback of asset C via the timeline window 1725a.

[0208] In the example shown in Figure 17B, the assets in the asset library window 1705 have modifiers defined with them, which can be accessed and adjusted with modulation data via the GUI 1700b. According to this example, the asset modifier window 1720b indicates that asset A has the following modifiers defined:• Envelopment extent, which indicates how spatially enveloping an asset will be (e.g.only in a discrete location of a playback environment vs. extending throughout the playback environment);• Effect intensity; and• Swirl, which indicates whether an asset will rotate, e.g., around a presumed user location in a playback environment.

[0209] In this example, a user has selected the effect intensity modifier 1722 of asset A. According to this example, the timeline window 1725b includes a manual input field 1730 in which a user can provide hand-drawn data that will be routed to the effect intensity modifier 1722: the manually drawn line in the manual input field 1730 indicates how effect intensity will be modulated for the current instance of asset A over the time that the cunent instance ofDolby Ref. D24098W001asset A is reproduced. In this example, “effect intensity,” etc., are not object properties but are instead labels assigned to the modifier by the asset creator.

[0210] Figure 18 is a block diagram that illustrates an example of applying the effect intensity modifier shown in Figure 17A. In this example, the diagram 1800 illustrates mappings 1805a and 1805b of the effect intensity modifier 1722 to the brightness levels 1810a and 1810b of sensory objects 1712a and 1712b, respectively, according to the manually drawn line in the manual input field 1730 of Figure 17B. In this example, the mappings 1805a and 1805b are equal and simultaneous.

[0211] In alternative examples, a creation tool may provide GUI controls for a user to cause individual mappings of the brightness levels, and potentially of other properties, of the sensory objects of an asset. The above-described GUIs, or similar GUIs, may be used to access the underlying sensory objects within an asset.

[0212] In some cases, a content creator may choose to an existing version of an asset for a particular project, so that updates to the asset are not required. However, the above-described GUIs, or similar GUIs, may also be used to update an asset permanently to create a new version of an asset instance, as opposed to specifying temporary modifications to an existing asset instance.

[0213] Some creation tools provide merging functionality both from downstream (e.g., sending edited versions of an asset back to the asset library) but also from upstream (e.g., retrieving versions of an asset, and potentially of “delta” information indicating changes from one version to another version of an asset, from the asset library). Some creation tools provide a GUI to show two versions — for example, an old and a new version — of an asset concurrently so that a content creator may use the GUI to choose which deltas to accept.

[0214] Figure 19 shows an example of a GUI that for comparing two versions of the same asset. In this example, the GUI 1900 includes window 1910a, which shows version 1 of asset A, window 1910b, which shows version 2 of asset A, as well as window 1910c, which shows a merged version of version 1 and version 2. According to this example, the GUI 1900 includes a timeline 1925 that includes hue graphs 1915 and position graphs 1920. The hue graphs 1915 show how the hues of sensory objects 1712a and 1712b change over time and the position graphs 1920 show how the x coordinates of sensory objects 1712a and 1712b change overtime.Dolby Ref. D24098W001

[0215] One may observe that at the time 1930 indicated on the timeline 1925, the hue of sensory object 1712a is different in versions 1 and 2. In addition, one may observe that the position of sensory object 1712b is different in versions 1 and 2 at the time 1930. According to this example, a user may interact with the “accept left” virtual button 1935 or the “accept right” virtual button 1940 in order to select a sensory object property from version 1 or version 2. For example, a user may click on the sensory object 1712a in the window 1910c and then click on the “accept left” virtual button 1935 to select the hue of the sensory object 1712a in version 1. Similarly, a user may click on the sensory object 1712b in the window 1910c and then click on the “accept right” virtual button 1940 to select the position of the sensory object 1712b in version 2.

[0216] As with other figures provided herein, the types and numbers of elements shown in Figure 19 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. In some instances, the GUI 1900 may show additional sensory object properties, such as y coordinates, z coordinates, intensity, saturation, etc. According to some examples, the GUI 1900 may show, in at least one window, “onion skin” overlays of sensory object property representations corresponding to two asset versions. In some examples, the GUI 1900 may show text-based diffs, e.g., similar to what is used in version control software, to represent sensory object property differences between two asset versions.

[0217] Creation Tools for Authoring Bed-Based Sensory Content

[0218] In some instances it may be desirable to create what may be referred to herein as “bed-based sensory content.” As used herein, the term “bed-based” is somewhat similar to what is commonly referred to as “channel-based,” in the sense that each “bed” of bed-based content expressly or implicitly refers to a position, direction or zone in a playback environment. For example, bed-based sensory content may include a left bed, a right bed, a front bed and a back bed, each of which is intended, by the content creator(s), to be played back in a corresponding area or zone of a playback environment. However, bed-based content is different from channel-based content. For example, bed-based content does not require that there is a direct correspondence between channels and actuators, as in there is with channel-based content. In one such example, if channel-based content included content for a left light, and the playback environment had no left light, the channel-based content for the left light would not be played back. However, if bed-based content included content for aDolby Ref. D24098W001left light, and the playback environment had no left light, the bed-based content for the left light could, at least in some instances, be flexibly rendered to one or more nearby light fixtures and played back accordingly. In some examples, it may be desirable to create objectbased sensory data combined with bed-based sensory content.

[0219] A creator may simply want to create at least some sensory content using a bed-based approach where they have limited spatial control of the sensory objects’ locations.According to some examples, a creation tool may allow a content creator to assign one or more sensory objects to beds. Such bed-based options can provide a relatively simpler creation workflow than creating location information for each individual sensory object. Beds may be arranged in any spatial configuration that is defined at creation time, e.g., according to a user’s selection from a creation tool GUI.

[0220] Figure 20 shows an example of a GUI that may be used for bed-based sensory object authoring. The GUI 2000 of Figure 20 represents a 6-bed example, in which the beds of the sensory object bed stream are located at canonical directions with respect to the reference environment 2001. In this example, the beds are located as follows:• Front• Rear• Ueft• Right• Up• Down

[0221] With regard to bed based content, a creation tool may allow the content creator to assign multiple sensory objects, potentially of different modalities, to the same bed. In this context, the “modality” of a sensory object refers to a corresponding type of sensory content, such as light, haptics, air flow, etc. Accordingly, sensory objects having different modalities will generally be played back using actuators of different modalities in a playback environment. If sensory objects assigned to the same bed have the same modalities, some creation tools may allow a content creator to assign a priority level to each of the sensory objects, e.g., by creating priority metadata for the sensory Tenderer of a playback environment. Where objects share modalities, the order in which objects are rendered may be important, according to the mixing and blending law(s) the sensory Tenderer is following.Dolby Ref. D24098W001The mixing and blending laws may, in some examples, be created as metadata by interacting with a creation tool and included in the metadata of the bed-based sensory content stream.

[0222] In some examples, a creation tool may allow the content creator to create bed-based objects that are, or include, temporal effects, such as a flicker or swelling vibration.According to some examples, a creation tool may allow the content creator to create bedbased objects that are, or include, a static piece of content (such as the color red).

[0223] According to some examples, a creation tool may allow the content creator to define bed locations. Such bed locations may, for example, be indicated by metadata of a bed-based sensory content stream. These user-defined bed locations may, for example, be selectable from a set of standard bed locations, in which case the metadata only need to include the identifier of the bed arrangement. However, in some examples, user-defined bed locations may be custom locations defined by the content creator. In such cases, the entire specification of the bed arrangement should be included in the bed-based sensory content stream, e.g., as bed location metadata.

[0224] A bed-based creation approach can simplify the sensory content creation process. Such simplification may be useful for various use cases, such situations in which a content creator needs to create sensory content quickly, and / or automatically, or situations in which limited spatial fidelity is required.

[0225] Creation Tools for Authoring Ambient Sensory Content

[0226] Some disclosed implementations involve what are referred to herein as “ambient sensory objects” which collectively may extend over part or all of an endpoint volume. Some creation tool implementations provide an ambient sensory object creation workflow. Such creation tool implementations can provide a relatively simpler creation workflow option, as compared to a spatial object based workflow or a bed-based workflow.

[0227] Figure 21 shows an example of a GUI that may be used for ambient sensory object authoring. The GUI 2100 of Figure 21 represents an ambient object sphere 2105, which entirely encompasses the reference environment 2101 in this example. By interacting with the GUI 2100 and / or related GUIs, a content creator may define one or more ambient sensory objects. In this example, a content creator has created ambient sensory objects 2110a, 2110b and 2110c, all of which are snapped / attached to the ambient object sphere 2105 in this example. In this example, the ambient sensory objects 2110a, 2110b and 2110c representDolby Ref. D24098W001ambient lighting effects. The content creator may define ambient sensory object sizes and positions, as well as data associated with the modality to which the ambient sensory objects belong (e.g., color for light objects or a vibrational effect for haptic objects).

[0228] By interacting with the GUI 2100 and / or related GUIs, a content creator may define the method(s) by which created ambient sensory objects combine to fill the ambient object sphere 2105. For example, a content creator may click and drag any of the ambient sensory objects 2110a, 2110b and 2110c in order to move them around the ambient object sphere 2105, expand or contract the size of each of the ambient sensory objects 2110a, 2110b and 2110c, modify the hue, brightness, or other properties, etc. In some implementations, by interacting with the GUI 2100 and / or related GUIs, a content creator may expose the position, size, color, of the ambient sensory objects 2110a, 2110b and 2110c to an interactive API, e.g., as described with reference to Figure 16.

[0229] Creation Tools for Authoring Game-Related Sensory Content

[0230] In the gaming context, the term “actor” refers to an interactive object or character in a game. In some instances, a content creator my choose one or more actors in a scene to correspond with sensory objects.

[0231] Figure 22 shows an example of a GUI for associating one or more sensory objects with a game actor. In this example, the GUI 2200 shows an actor 2205, which is a weapons portal of a game in this instance. The light bulb and speaker within the representation of the actor 2205 indicate that corresponding light and audio effects have been defined for the actor 2205. According to this example, the GUI 2200 includes a window 2210 with which a content creator may interact in order to define sensory object properties that correspond to the actor 2205. In this example, the window 2210 includes an object ID field, a color field, a size field, a feather field, a “falloff’ field and three “ignore” fields. The “feather” field may be used to indicate characteristics of a “feather” area or volume surrounding a sensory object within which the sensory object’s attributes will have some effect but will taper off with increasing distance from the outer surface of the sensory object. In this example, the “feather” field is used to indicate a rate of change in intensity in the feather area or volume. The falloff field may be used to indicate a range within which the defined sensory object should start having an effect on the player’ s environment. In this example, the falloff field may be used to indicate a feather distance, e.g., a thickness of the feather area or volume. According to these examples, the ignore x / y / z fields indicate whether to ignore the distanceDolby Ref. D24098W001between the sensory object position and any actuators in the x, y or z direction when determining if whether to activate those actuators. For example, when “Ignore Z” (the height axis) is checked, only sensory object / actuator distances in the x,y plane will be computed. In some examples, the window 2210 may initially be pre-populated with default values that may then be overwritten if the content creator so chooses.

[0232] Figure 23 shows examples of linking sensory object properties and game state information. In this example, the GUI 2300 includes visual scripting blocks for a game engine. According to this example, the GUI portion 2305 includes blocks that have been linked to the existing game script of the GUI portion 2305. By interacting with the GUI portion 2305, a content creator may define sensory object properties that correspond to game state information, which in this example includes information regarding the health of a player in the game. According to this example, the player’s health state can range from zero to 100. In this example, these health state values have been linked to the colors of a sensory object, ranging from a red color corresponding to zero health to a green color corresponding to 100% health.

[0233] In this example, the GUI portion 2305 includes a GUI portion 2310 for defining sensory object properties including a sensory object ID, sensory object color, sensory object position and sensory object size. According to this example, a content creator has interacted with the “color” field of GUI portion 2310, which has caused the GUI portion 2315 to appear. In this example, the GUI portion 2315 includes fields for defining a custom color, including hue, saturation, brightness and color temperature (Wb) fields.

[0234] According to some examples, a creation tool may provide one or more GUIs to allow existing 3D world environments to be used as extensions of a what is being displayed on a screen in a playback environment. In some such examples, a creation tool may provide an ambient bed that may be automatically filled with one or more of the following:• Static beds associated with zones;• Sampling material colors from objects near the player;• Sampling render pipeline from camera view and beyondDolby Ref. D24098W001Feedback to the Content Creator Providing Warnings & Guidance Related to the Capabilities of Endpoints

[0235] According to some examples, a creation tool may provide an extension to the visualizer that is used during content creation to not only show the content in different endpoints but also to provide warnings and guidance related to some typical endpoint capabilities. This allows the content creator, who may often have a highly capable reference environment, to “audition” content for less-capable playback environments. In some such examples, a content creation tool may provide warnings and / or cues to the content creator when they are creating content that is unlikely to be reproducible in some playback environments. According to some such examples, a content creation tool also may provide simulations of one or more lower-capability playback environments. Some examples of such instances include:• Color setpoints outside of the gamut for typical endpoints.• Having a small dynamic range in the created content (e.g., small hue changes) that would be lost in quantization noise of typical endpoints (e.g. 8-bit RGB LEDs)• Time-frequency content that is not reproducible by typical endpoints, e.g., having brightness modulate at a very high frequency that will not be reproducible by typical endpoints with update rates of 10Hz, 25Hz, etc. In haptics, content may have been produced with a wide frequency range which may not be reproducible by actuators with a lower frequency range.• Spatial frequency content that is not reproducible by typical endpoint, e.g., having very close or small objects (or spatial frequency of effects) that would get spatially aliased across typical endpoints with LED spacings of 1cm, 2cm, 5cm, 15cm, etc.

[0236] Figure 24 shows an example of an auditioning endpoint with warnings for a content creator. According to this example, the content creator has selected a sensory object color that will be out of the reproducible range for some playback environments. In response to this selection, a creation tool is presenting the GUI 2400 on a display device. In this example, the GUI 2400 includes a warning message 2402, indicating that the selected color is out of range and that a simulated playback of one or more selected sensory objects is being presented on a simulated playback environment 2401 (“Endpoint 1”) that is capable of reproducing a more typical and restricted color space. According to this example, the playback environment 2401 includes a television 2405 and light strips 2410a, 2410b andDolby Ref. D24098W0012410c. In this example, the GUI 2400 includes an endpoint selection window 2415 with which a content creator may interact in order to select another simulated playback environment in which to preview the selected sensory object(s).

[0237] Creative Boundaries on Personalisation

[0238] Some creation tools may provide a content creator with the option to set boundaries or limits within the content itself, for example regarding at least some immersion control / personalisation preferences presented to a user. Such boundaries or limits may, for example, involve one or more of the following:• The highest or lowest time frequency allowable for a particular effect or an alternative if the lowest is zero. For example, some such boundaries or limits may be applied to a strobe / flashing light effect. Such boundaries or limits should take into account any applicable regulations regarding light sensitivity.• Changing to another form of rendering if the space-frequency is on the lowest setting, for example providing an ambient bed alternative.• The smallest dynamic range or at least a way to audition content on this range.• The degree (whether it be shift in hue, RGB values or otherwise) to which colors can be tone-mapped.

[0239] Creation Tools for Domain-Bound Sensory Content Creation

[0240] Some disclosed creation tools allow a content creator to bound or scale sensory content to a domain that is not the entire reference environment, such that the domain over which sensory content is created does not encompass all of the actuators in the playback environment. Some examples of alternate domains include screen domains and actuator array domains. According to some such examples, the domain over which the content is created is then transferred / normalized to a new range which does not encompass all of the actuators in the playback environment.

[0241] Screen-Anchored Sensory Objects

[0242] Some disclosed creation tools allow a content creator to create sensory content with respect to a reference screen, which may be referred to as “screen- anchored sensory objects.” The ability to create screen-anchored sensory objects is particularly useful when creating a multisensory experience that is congruent with what is being displayed on a display screen.Dolby Ref. D24098W001For example, lightscape objects may be created that are spatially congruent with the location of objects shown on the display screen.

[0243] Some examples may involve a normalization process for screen domains. In some such examples, a region of the playback environment and a subset of the actuators within the playback environment may be projected to a normalized coordinate domain. Such methods can allow a content creator to create reference spatial content anchored to multiple screens in the reference environment, resulting in the Tenderer producing light content relative to multiple screens in the playback environment according to the reference spatial content. According to some examples, this process involves computing a transformation matrix T for each of two or more screen domains. In some examples, for every domain, the control system may use a spatially variant transformation matrix. Using a spatially variant transformation matrix allows the control system to produce piecewise transformations to support various playback environment configurations. Here, we use the term “spatially variant transformation matrix’’ to define a transformation matrix that changes discretely depending upon which subset of the playback environment to which the transformation matrix is being applied. Within that subset of the playback environment, the transformation matrix is not spatially variant in the traditional sense. In some alternative examples, a transformation matrix which is spatially variant may be used within each subset of the playback environment. Figure 26A — which is described in more detail below — shows an example of a reference environment that has two actuator domains, each of which is anchored to a different reference screen. These are examples of the “screen domains” referenced in this paragraph. Some examples may involve computing a transformation matrix T for each of these two screen domains. In some examples — such as when the screen domains include different-sized areas of the reference environment — for each screen domain, the control system may use a different transformation matrix.

[0244] Actuator Array Domain Rendering

[0245] Some disclosed creation tools allow a content creator to create sensory content that is similar in nature to screen-anchored sensory objects, in the sense that the spatial and temporal properties of the sensory objects may be designed relative to multiple actuators creating an “actuator array” in the playback environment, such as a light strip or a light grid. Some disclosed creation tools allow a content creator to create actuator array domain content such that the created sensory objects will be rendered on a domain that is normalized to theDolby Ref. D24098W001actuator array. In one such example, a content creator may design a sensory effect to be produced by way of a spatial sensory object traversing over a spatial domain with a particular positional trajectory. The content creator may, for example according to sensory object metadata, convey their intent for this sensory effect to be rendered on a single actuator array, for example on a single light strip. In order to create actuator array domain effects, the content creator may specify, for example according to sensory object metadata, a global position to indicate where the sensory effect is to be rendered in the playback environment.

[0246] The normalisation of positions may, in some examples, be performed as described elsewhere herein regarding the normalization of screen-anchored objects. In some examples, there may be multiple actuators mapped into an actuator array domain. According to some such examples, these multiple actuators may be arrays themselves and may create a superarray.

[0247] Figure 25A shows another example of a reference environment with reference to which content can be created. Figure 25B shows an example of an actuator domain for content creation in the reference environment of Figure 25 A. As with other figures provided herein, the types and numbers of elements shown in Figures 25A and 25B are merely provided by way of examples. Other implementations may include more, fewer and / or different types and numbers of elements.

[0248] According to this example, Figure 25A shows a “top down" view of the reference environment 10000, viewed along the z axis and perpendicular to the x and y axes. In this example, an actuator domain object 50001a, a reference screen 50005a and a reference person 50010 are within the reference environment 10000. In this example, Figure 25A shows the x and y extent of the actuator domain object 50001a within the reference environment 10000. Accordingly, the content creator has defined a reference region corresponding to the spatial extent of the actuator domain object 50001a within the reference environment 10000, intending that the actuator domain object 50001a will be presented in a corresponding playback region of a playback environment. According to this example, Figure 25A indicates a distance du from the reference screen 50005 to a reference person 50010.

[0249] In this example, Figure 25B shows an actuator domain 50000a within which a content creator may indicate the desired details of the actuator domain object 50001a of Figure 25 A. In this instance, the content creator has placed actuator domain sub-objects 50002 and 50003 within the actuator domain 50000a. The actuator domain sub-objects 50002 and 50003 may,Dolby Ref. D24098W001for example, be light objects at corresponding positions within the actuator domain 50000a at a snapshot in time corresponding to Figure 25B.

[0250] Figure 26A shows another example of a reference environment with reference to which content can be created. Figures 26B and 26C show examples of actuator domains for content creation in the reference environment of Figure 26A. As with other figures provided herein, the types and numbers of elements shown in Figures 26A-26C are merely provided by way of examples. Other implementations may include more, fewer and / or different types and numbers of elements.

[0251] According to this example, Figure 26A shows a “top down’’ view of the reference environment 26000, viewed along the z axis and perpendicular to the x and y axes. In this example, the actuator domain object 50001b includes a reference screen 50005b and the actuator domain object 50001c includes a reference screen 50005c. Accordingly, the content creator has defined two reference regions corresponding to the spatial extents of the actuator domain objects 50001b and 50001c within the reference environment 26000, intending that the actuator domain objects 50001b and 50001c will be presented in two corresponding playback regions of a playback environment.

[0252] In this example, Figure 26B shows an actuator domain 50000b within which a content creator may indicate the desired details of the actuator domain object 50001b of Figure 26A. Likewise, Figure 26C shows an actuator domain 50000c within which a content creator may indicate the desired details of the actuator domain object 50001c of Figure 26A. The examples shown in Figures 26A-26C are potentially advantageous for multi-screen playback environments, including but not limited to vehicle-type playback environments.

[0253] Graph signal content creation

[0254] In some examples, sensory objects and sensory effects can be created according to graph signal processing, which may involve analyzing the underlying structure (e.g., the layout or physical distribution) of the actuators within a playback environment in order to produce a graph in which the nodes arc the controllable actuators in the playback environment. A useful product of this process is the “shift matrix” SGe K / V / 'x Na, which may be used to propagate signals on the graph. One can define a signal, xG, on the graph comprised of the actuators in the room as xGG HlNa. The shift matrix is a mechanism which allows a control system to propagate this signal across the graph, e.g., as follows:xG[n + l] = SGxG[n]Dolby Ref. D24098W001

[0255] In the foregoing equation, n represents the current time step and n + 1 represents the next timestep, during which the graph signal will be propagated. A graph signal may be incorporated into an actuator activation matrix that is used to render sensory object signals in a playback environment.

[0256] Some disclosed creation tools allow a content creator to create, based on the underlying structure of the actuators in the endpoint, a graph upon signals may be propagated and rendered. According to some examples, a creation tool may simply provide a mechanism for rendering volumetric fields and textures in the playback environment. Some disclosed creation tools provide GUIs or other tools that allow a content creator to do one or more of the following:• Explicitly define graph signals which are resampled and applied to the graph in the Tenderer.• Define the propagation behaviour by a set of parameters describing, for example:o spatial and / or temporal frequency;o diffusivityo threshold / hold over times. In some examples, these values may go into the diagonal of the propagation matrix H_G discussed elsewhere in this disclosure.

[0257] Figure 27 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein. The blocks of method 2700, like other methods described herein, are not necessarily performed in the order indicated. In some implementation, one or more of the blocks of method 2700 may be performed concurrently. Moreover, some implementations of method 2700 may include more or fewer blocks than shown and / or described. The blocks of method 2700 may be performed by one or more devices, which may be (or may include) one or more instances of control system such as the control system 110 that is shown in Figure 1 and described above.

[0258] In this example, method 2700 involves visualizing and augmenting sensory content. According to this example, block 2705 involves displaying, via a 3D visualizer implemented by a control system, a representation of an endpoint and sensory content projected into the endpoint, wherein the sensory content comprises one or more sensory objects of a parametric or non-parametric shape. In some examples, the one or more sensory objects may include one or more light objects, one or more haptic objects, one or more air flow objects, or combinations thereof.Dolby Ref. D24098W001

[0259] According to this example, block 2710 involves augmenting the displayed sensory content, according to augmentation user input received by the control system via an interface system, using a volumetric field or a volumetric texture. If the augmentation involves a volumetric field, the volumetric field may or may not be spatially uniform, depending on the particular implementation. In some examples, method 2700 may involve creating, by the control system, at least one sensory object according to creation user input received by the control system via an interface system.

[0260] In some examples, method 2700 may involve receiving input from a content creator regarding whether a volumetric field is spatially uniform. Some such examples may involve specifying in sensory content metadata whether a volumetric field is spatially uniform.

[0261] According to some examples, the sensory content may include a first sensory asset and a second sensory asset. According to some examples, the first and second sensory assets may be nested sensory assets.

[0262] In some examples, the sensory content may include image data, video data, or lightscape content. In some examples, the sensory content may be configured for bed-based rendering.

[0263] Personalization and Control of Multisensory Experiences

[0264] This section of the disclosure describes methods and systems that enable a user to personalise and control a flexibly rendered sensory experience in multiple zones across an endpoint. Some disclosed aspects of this personalization involve mechanisms for dynamically adjusting the perceived immersion of sensory content through multi-knob personalization. This allows a user (such as a person in a playback environment) to manually adjust a multisensory experience to their preference based on a number of factors across different modalities.

[0265] For example, in the lighting context, such controls may allow a user to choose one or more of the following:• Limiting the spatial resolution of lighting effects and / or light objects;• Limiting the rate of change of fixture outputs, such as adjustment for photosensitivity;• Scaling the brightness of some or all of the content;• Overriding certain fixtures for functional (information-based) lights, e.g. in a gaming context, a user may only want their health status to show up on one particular light fixture.Dolby Ref. D24098W001

[0266] In the haptics context, such controls may allow a user to choose one or more of the following:• Limiting the intensity of vibration and / or its rate of change;• Scaling the intensity across content• Limiting the fixtures to use for haptics: for example, a user may not want haptics playback near an injury.

[0267] In the airflow context, such controls may allow a user to choose one or more of the following:• Limiting the intensity of airflow:• Scaling the airflow intensity across content;• Limiting the temperature’ s upper and / or lower bounds.

[0268] Virtual or physical controls that allow a user to control such personalization features may be referred to herein as “knobs.” In some examples, knobs for controlling such features may be grouped together according to aspects of immersion, e.g., temporal, intensity, spatial. According to some examples, knobs for controlling such features may be grouped according to modality (e.g., light, haptic or air flow) or across one or more modalities. In some examples, knobs for controlling such features may be a continuous scale, whereas in other examples the knobs may have discrete steps representing levels of parameter modification.

[0269] Figure 28 shows an example of a GUI that includes personalization controls.According to this example, the GUI 2800 includes the following elements:• 2801 :A single immersion knob control, with which a user may interact in order to control the immersion level across all modalities on a continuous scale. For example, increasing the immersion level may increase the intensity or brightness of lighting effects and may also increase the amplitude of haptic effects;• 2802: A GUI area that includes knobs for controlling the immersion level across all modalities, with knobs grouped by aspect (temporal, spatial and intensity) on continuous scales;• 2803 : A GUI area that includes knobs grouped by modality, for controlling the immersion level of each modality on a continuous scale;• 2804: A GUI area that includes knobs grouped by modality and by aspect for each modality, for controlling the immersion level of each modality aspect on a continuous scale; andDolby Ref. D24098W001• 2805: A GUI area that includes knobs for controlling the immersion level across all modalities, with knobs grouped by aspect (temporal, spatial and intensity) on discrete scales.

[0270] Interactivity and immersion

[0271] The immersion level of some or all aspects of a sensory experience may also be modified by inputs corresponding to one or more types of interactivity. Such interactivity may include event-based triggers which cause dynamic changes to one or more aspects of sensory content. Following are some examples:• In a vehicle, when a user receives a call, the overall intensity of the experience may be reduced: lights may be dimmed, haptic intensity may be lowered and airflow intensity may be lowered (or the airflow may be turned off);• A volume control or other control of a vehicle, a television, etc., may be linked with sensory immersion such that where muting or pausing an experience causes some or all sensory content to be turned down, turned off or paused.

[0272] According to some examples, personalisation may allow a user to set one or more types of limits with which interactivity can affect a sensory experience and / or allow a user to set up new interactive features for themselves. For example, some personalization examples may allow a user to set upper and / or lower bounds to a sensory experience so that the user’s overall level of immersion and preferences remain consistent.

[0273] Zone-based personalisa tion

[0274] In some examples, personalization may also be modifiable on a zone basis. For example, particular actuators may be grouped together before applying user-controlled personalization control of immersion across types and modalities. For example, in a vehicle, the front and rear seats may have differing levels of immersion based on user preference or for safety. This could include:• No flashing or rapid changes in the front of the vehicle near the driver;• Reduced levels of haptics and airflow for the driver;• Different / selectable levels of haptics and airflow for rear seats, which often contain children, who may have sensitivities.Dolby Ref. D24098W001

[0275] Figures 29 A, 29B and 29C show three examples of zone-based personalization. The elements of Figures 29A, 29B and 29C are as follows:• 2901 : Top-down view of a room in which a single stream of sensory content is played back across two zones with different levels of immersion;• 2902: High-immersion zone;• 2903 : Low-immersion zone;• 2904: Top-down view of a room in which two different streams of sensory content are played back across two separate zones;• 2905 : High immersion zone of Content A, Content B not present;• 2906: High immersion zone of Content B, Content A not present;• 2907 : Hard transition between the two zones with separate streams of sensory content;• 2908: Top-down view of a room in which two streams of sensory content are played back across two separate zones with a low-immersion transition area between them;• 2909: High immersion zone of Content A, Content B not present;• 2910: High immersion zone of Content B, Content A not present;• 2911: Transition zone between two Content A and B. Here, both streams of sensory content arc played back in low-immersion to avoid a stark difference at the edges of zones 2909 and 2910.

[0276] According to some examples, zones may have specific types which may limit, or have default settings for, personalisation. For example:• In an ambient zone, the rate of change of movement and the intensity may be limited, so that these features are less distracting;• In a functional zone, no sensory objects other than those that are functional (informational) may be rendered, e.g. sensory objects that are on an informational overlay layer.

[0277] In some examples, the actuators which are included in a given zone may be specified and computed as part of configuration, particularly in setups (such as those of a vehicle) where actuator layout cannot be modified by a user. According to some examples, zones may be defined by a user, for example in an open room layout where the user may want to specify each space in the room as an individual zone. Zones are also a way for a user to exclude certain fixtures if, for example, a user has physical sensitivities to certain fixturesDolby Ref. D24098W001depending on where they are located. For example, someone with an injury may wish to exclude a particular haptics actuator.

[0278] According to some examples, zones may be applied using the masking matrix, as described in more detail elsewhere herein. In some examples, a zone activation matrix may be defined to activate the desired actuators within a given zone, which may be computed as part of configuration or may be dynamically updated by a user.

[0279] Layer-Based Personalization

[0280] Layer-based personalization is similar to the concept of using zones, but rather than grouping by actuator, instead layers are grouped together to apply differing levels of sensory immersion. For example, to lower sensory immersion a user may choose to only show functional (informational) lighting, which may correspond to a specific layer type in the sensory Tenderer, smoothing and or completely disabling any other layers lower in a layerbased hierarchy.

[0281] Dynamic Range Compression

[0282] Immersion control may involve reducing the dynamic range of sensory content by, for example, defining minimum and maximum permissible amplitude values. For the light modality, such minimum and maximum permissible amplitude values may map to brightness. When combined with an overall brightness bias, such dynamic range compression may support, for example, the following use cases:• Reducing high-brightness amplitude spikes that may be undesirable to some viewers;• Bringing up low-brightness objects so that they are still noticeable in the presence of bright ambient light (e.g., daytime sunlight in a vehicle);• Bringing up low brightness objects so they are as still noticeable as the brightness of the content as a whole is lowered (e.g., to move out of focus when higher-priority content is present).

[0283] The saturation of colors rendered in the light modality of sensory experiences may also be subject to dynamic range compression. Such dynamic range compression may be used for the purpose of immersion control, where the range of saturation could be limited significantly or simply set according to a user’s preference.Dolby Ref. D24098W001

[0284] For the haptics modality, dynamic range compression may be applied to the overall amplitude or to the envelope of the signals driving the haptic actuators. For the haptics modality, limiting the maximum actuation value is a way of controlling how intense or immersive the haptics experience is. Similarly, a particular user may be somewhat insensitive to haptics vibrations, or perhaps they arc wearing thick clothing between their body and the actuator. In this case, the minimum amplitude value may be raised, limiting the dynamic range but with the intent of increasing the overall perceived intensity.

[0285] For the airflow modality, dynamic range compression can be applied to both the speed of the airflow and the temperature.

[0286] Dynamic range compression can be applied by altering the minimum actuation value, the maximum actuation value, or both. Some sensory content may require that sensory objects fade out and may require the ability to turn an actuator all the way down to 0. For these sensory objects, dynamic range compression may be achieved by limiting the maximum actuation value.

[0287] Dynamic range compression can be implemented by applying a linear or a non-linear transformation on the actuator control data. The transformation may be based on a continuous function or may be constructed piecewise from multiple underlying functions.

[0288] Color Preferences for Lightscapes

[0289] According to some examples, personalization controls may allow a user to alter the perceived color tone of the rendered light modality. This can be implemented by transforming the colors of either the light objects themselves, potentially a subset of light objects (filtered by type, layer, etc.) or may be applied to the actuator commands themselves. Preference controls may be exposed (presented) by allowing the user to choose from a discrete set of color preferences (e.g., cool / wami / reference) or by allowing the user to parameterize transformation functions applied to the colors directly. The ability for the user to apply this transformation may, in some examples, be enabled or disabled by the sensory content itself.Dolby Ref. D24098W001

[0290] Time-Frequency / Spatial-Frequency Low Pass Filtering

[0291] In some examples, personalization controls may allow a user to control timefrequency limiting or filtering of sensory content attributes (such as color for lightscapes). Such personalization controls may enable one or more of the following:• Setting a maximum rate of change for amplitude A) rise and B) fall;• Slow and / or fast time control of dynamism in absolute brightness of rendered scene.

[0292] According to some examples, personalization controls may allow a user to control time-frequency limiting or filtering of sensory object positions. Such personalization controls may limit the spatial velocity at which a sensory object moves within the endpoint. In some examples, such controls may be implemented by modulating both the sensory object’s size and position in order to maintain the content creator’s intended trajectory for the sensory object.

[0293] In some examples, personalization controls may allow a user to control spatial-frequency limiting or filtering of the MS content positions. This may result in diffusing sensory content attributes (such as color for lightscapes or a haptics signal) over an area versus a point.

[0294] Preferred Frequency Profiles for Haptics

[0295] Applying time-frequency equalisation of haptics signals, similar in nature to how a user can have a preferred audio EQ, can be used to personalize sensory experiences in which a haptics modality is present. Such personalization controls may be implemented by allowing the user to control gains for multiple frequency bands.

[0296] In some examples, personalization controls may allow a user to control spacefrequency filtering of haptics signals. Such personalization controls would, for example, affect how “sharp” particular haptics objects would be perceived to be when rendered on an array of haptics actuators.

[0297] Formats for Scalable Creation and Sharing of Sensory Assets and Experiences

[0298] This disclosure describes schemas that facilitate storage and interchange of assets that may include multi-stream, multi-sensory and potentially interactive experiences. Also described are methods to interpret these schemas so that they can be represented in a human-readable and subsequently editable form.Dolby Ref. D24098W001

[0299] Motivation

[0300] When constructing a sensory experience, such as a lightscape, a content creator may wish to re-use previously-created elements. Suppose, for example, their current project needs an intricate fireworks light effect, an effect using several light-objects with a carefully coordinated sequence of movement and colour. If this is an effect the content creator had created for a previous lightscape project, rather than recreate the fireworks light effect from the elemental sensory objects, it would be highly desirable to re-use the earlier created effect, importing it into the new project as a packaged asset.

[0301] Figure 30 is a block diagram that illustrates assets being imported from, and exported to, an asset library. In this example, the system 3000 includes creation tools 3005a and 3005b, each of which is configured to import re-usable assets from the asset library 3010 and to export re-usable assets to the asset library 3010. In this example, re-usable assets A, B and C are shown in the asset library 3010 by way of example. The creation tool 3005a may, for example, be running on an instance of the apparatus 101 of Figure 1 (such as a laptop) controlled by the above-described content creator. The creation tool 3005b may, for example, be running on another instance of the apparatus 101 controlled by another member of a team of content creators that includes the above-described content creator. As with other figures provided herein, the types and numbers of elements shown in Figure 30 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. For example, the inventors contemplate that a much larger number of re-usable assets and re-usable asset versions could reside in an asset library.

[0302] Following are examples of desirable attributes of re-usable assets:• Information-Preserving: Once this packaged asset is imported / retrieved, the content creator may wish to adjust one or more effects, perhaps changing the sequence of light object movements. For creative flexibility it is therefore important to retain a maximal amount of information in the packaged asset.• Nestable: The content creator may also wish to add additional elements, for example a blue ambient layer over which the fireworks effect occurs. For this new asset to be re-usable, it would be desirable to nest the base fireworks asset, alongside this new layer, into a new asset.Dolby Ref. D24098W001• Standardized: It is desirable for exportable assets to be in a standardized format as this would allow different creation program instances to exchange files. Such standardization could help to promote a content creator ecosystem and could facilitate the development of dedicated tools that focus on particular tasks. One precedent to point to is the Universal Scene Description (USD) format, that would allow digital artists to create assets in Blender and for them to then be transferred across to a realtime rendering engine such as Unreal Engine or Unity.

[0303] Assets

[0304] Various asset structures are disclosed herein. Figures 31 and 32 are block diagrams that illustrate different types of asset structures.

[0305] Asset A of Figure 31 includes sensory objects 1, 2 and 3, which are described according to an asset timeline 3105a, and asset metadata 3110a, which is “global” asset metadata in this instance. Each of the sensory objects 1, 2 and 3 will generally include sensory object metadata such as sensory object size metadata, sensory object position metadata, sensory object metadata corresponding to the particular sensory object modality, etc., for example as described elsewhere herein. The asset metadata 3110a may, for example, include static information such as one or more of the following:• version number;• control hook mappings, which provide a mapping between a control in the playback environment (e.g., the presence of a person is detected) and a parameter in the asset (e.g., use a specified alternate color);• asset description;• author;• looping flag.

[0306] The asset metadata 3110a may include dynamic information such as information regarding overall (asset-wide) intensity changes over time, which in this example are scheduled according to the asset metadata timeline 3115a. In some instances, the dynamic information indicate changes that can be repeated or “looped” according to whether the looping flag is set. The asset metadata timeline 3115a may describe when a sensory object begins to exist and its duration, e.g., relative to the beginning of the asset timeline 3105a. In some instances there may be a delay, where the beginning of the asset timeline 3105a doesDolby Ref. D24098W001not correspond to the beginning of the first sensory object (here, sensory object 1). A more complex asset metadata timeline 3115a may include information regarding the modulation of one or more sensory object properties, such as color or brightness in the case of a light object, or intensity in the case of a vibrational haptic object. In some examples, these modulations may be represented via times tamped metadata.

[0307] Nestable Assets

[0308] Figure 32 shows an example of nested assets. In this example, asset B includes sensory objects 2 and 3, as well as asset A of Figure 31. Accordingly, asset A is “nested” within asset B. In this example, asset A and sensory objects 2 and 3 are all scheduled according to the asset timeline 3105a. Corresponding dynamic metadata is scheduled according to the asset metadata timeline 3115b.

[0309] According to some examples, a nested asset may itself be nested. For example, a third asset, asset C, may include asset B, as well as one or more other assets, sensory objects, etc.

[0310] Control Interface

[0311] In some examples, an asset metadata field may include an interface that links into properties of one or more of an asset’s sensory objects, assets, or both, allowing a single point of control. The link into the objects and / or assets may be defined by a mapping function. For example, a linear input control may range from 0 to 1, but may have an exponential effect on a sensory object’s property.

[0312] According to some examples, this interface of the asset metadata field may allow simultaneous control of more than one sensory object property. In some examples, this interface may extend control to more than one sensory object. According to some examples, each control may have a distinct mapping function, providing the ability to build complex control structures.

[0313] Figure 33 is a block diagram that shows examples of mapping a control to sensory object and asset properties. According to the example of control diagram 3300, a user may provide input to a single control, control 1, in order to control properties of sensory objects 1 and 2, as well as to provide input to control 2 of nested asset 1. The user input may, for example, be provided via a virtual control slider, a virtual control knob, etc. In someDolby Ref. D24098W001instances, the input control may range from 0 to 1, from 0 to 10, or another suitable range, that is selectable by a user via the virtual control. Accordingly, the control values 3305 provided to and by control 1 are the same in this example.

[0314] However, in this example, the control value 3305 is transformed via mapping 1 to mapped control values 3310a, arc transformed via mapping 2 to mapped control values 3310b and are transformed via mapping 3 to mapped control values 3310c. This may be appropriate because a content creator may wish to use a different scale or range for the hue control of sensory object 1 than for the brightness control of sensory object 2.

[0315] According to this example, the mapped control value 3310c is different from the control value 3305, because control 2 is different from control 1. However, in some instances, mapping 3 may not be used. Instead, the control value 3305 may be passed directly from control 1 to control 2.

[0316] In some examples, one or more portions of the control diagram 3300, or the entire the control diagram 3300, may be presented as a GUI and may be editable by a user. For example, the GUI may allow a user to input the control value 3305, to select which properties of which sensory objects to modify, etc.

[0317] According to some examples, a user may be able to interact with control 2, e.g., by clicking on “control 2’’ of the control diagram 3300, to access another nested control diagram that corresponds to control 2. The nested control diagram may, for example, the GUI may allow a user to input the control value 3305, to select which properties of which sensory objects of nested Asset 1 to modify, etc.

[0318] In some examples, the control diagram 3300, or a similar control diagram, may be presented as a GUI that allows the connection of one or more of the following types of modifiers:• Envelope signals;• Low frequency oscillator signals;• Manual controls such as a faders and knobs.Dolby Ref. D24098W001

[0319] Mezzanine Format

[0320] It can be desirable to unwrap a nested asset or set of sequenced assets into a single timeline that can be exported as a “mezzanine” format, suitable for encoding or direct ingestion by a sensory renderer / player. In a mezzanine format, the individual sensory objects of a nested asset arc exposed. Implementing a mezzanine format involves addressing a number of complications around combining sensory objects into a unified timeline and bringing all underlying sensory objects into a single namespace.

[0321] Timing

[0322] Figure 34 shows an example of sensory objects within an asset. In this example, asset A includes sensory objects a, b and c. According to this example, the timing of a sensory object within asset A is defined relative to the start of the A asset timeline 3105c defined within asset A.

[0323] Figure 35 shows an example of nested assets. In this example, asset Z includes two instances of asset A, as well as two instances of asset C and one instance of asset D.According to this example, the timing of a nested asset within asset Z, including the timing of all of the sensory objects within the nested asset, is defined relative to the start of the asset Z timeline 31O5d.

[0324] Figure 36 shows asset Z of Figure 35 in a mezzanine format. In this example, the individual sensory objects of nested assets A, C and D are exposed. In this example the individual sensory objects of nested assets A, C and D have been combined onto a single timeline, the asset Z timeline 3105d. Therefore, the individual sensory objects of nested assets A, C and D have their timing recalculated relative to the asset Z timeline 3105d. When presented in mezzanine format, it is desirable for the individual sensory objects of nested assets A, C and D to have the namespaces unified, in order to avoid duplication of sensory object names and the resulting potential ambiguity. In other words, when presented in mezzanine format, it is desirable for the individual sensory objects of nested asset A to have different names than the individual sensory objects of nested assets C and D, e.g., as shown in Figure 36. If, for example, the individual sensory objects of nested asset C were originally designated as sensory objects a, b and c, some disclosed methods involve automatically updating the sensory object names (here, to sensory objects d, e and f) in order to avoid duplication of sensory object names between different assets.Dolby Ref. D24098W001

[0325] Controls for Mezzanine Format

[0326] When assets are “flattened” into the Mezzanine format to expose their internal sensory objects, it is desirable that any controls exposed by the control interface will also be flattened so that there is a single layer of control, exposed at the top level of the mezzanine format. One way that the control interfaces may be defined is through an interactive API, e.g., as discussed in this disclosure with reference to Figures 15A and 15B. However, instead of the assets shown in Figures 15A and 15B, individual sensory objects of those assets may be exposed in a mezzanine format.

[0327] Figure 37 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein. The blocks of method 3700, like other methods described herein, are not necessarily performed in the order indicated. In some implementation, one or more of the blocks of method 3700 may be performed concurrently. Moreover, some implementations of method 3700 may include more or fewer blocks than shown and / or described. The blocks of method 3700 may be performed by one or more devices, which may be (or may include) one or more instances of control system such as the control system 110 that is shown in Figure 1 and described above.

[0328] In this example, method 3700 involves unwrapping a nested sensory object-based asset and arranging the sensory objects of the nested sensory object-based asset into a single timeline. According to this example, block 3705 involves receiving, by a control system, a first sensory object-based asset comprising a second sensory object-based asset, wherein the second sensory object-based asset is nested within the first sensory object-based asset. In this example, each of the first sensory object-based asset and the second sensory object-based asset include one or more sensory objects, corresponding asset metadata, and a sensory object timeline. According to some examples, the one or more sensory objects of the second sensory object-based asset are arranged according to a second sensory object timeline of the second sensory object-based asset. In some examples, the second sensory object timeline is different from a first sensory object timeline of the first sensory object-based asset. In some examples, the one or more sensory objects may include one or more light objects, one or more haptic objects, one or more air flow objects, or combinations thereof.

[0329] According to this example, block 3710 involves combining, by the control system, a first sensory object timeline of the first sensory object-based asset and a second sensory object timeline of the second sensory object-based asset to create a combined sensory objectDolby Ref. D24098W001timeline comprising the one or more sensory objects of both the first and second sensory object-based assets. In some examples, the combined sensory object timeline may correspond with the second sensory object timeline.

[0330] In this example, block 3715 involves recalculating, by the control system, timing metadata for each of the one or more sensory objects of the first sensory object-based asset or the second sensory object-based asset in reference to the combined sensory object timeline. In some examples in which the combined sensory object timeline corresponds with the second sensory object timeline, block 3715 may involve recalculating the timing metadata for each of the one or more sensory objects of the first sensory object-based asset with reference to the combined sensory object timeline.

[0331] Some disclosed methods involve creating, interchanging and / or packaging sensory object-based assets. Some disclosed methods involve receiving, by a control system, user input regarding the creation of a sensory object-based asset including one or more sensory objects, corresponding asset metadata, and an asset timeline. The asset timeline may include information corresponding to a start time and duration for each sensory object of the one or more sensory objects. According to some examples, the user input may be received by a control system. For example, the user input may be received via a graphical user interface of a sensory asset creation tool implemented by the control system. Some disclosed methods involve creating, by the control system, the sensory object-based asset according to the user input.

[0332] According to some examples, the asset metadata may include a control interface mapped to sensory object properties of the one or more sensory objects. In some examples, the sensory object-based asset may include one or more nested sensory object-based assets. According to some examples, the asset timeline may correspond the to one or more nested sensory object-based assets.

[0333] Assisted Sensory Experience Content Creation

[0334] This section of the disclosure outlines methods, devices and systems that may be used to assist in the creation of sensory content. Assistive methods, devices and systems may include trained artificial intelligence (Al) networks, classical algorithms, heuristics, or combinations thereof. These assistive methods, devices and systems can assist the content creator by automatically generating and tagging event markers, analysing content in someDolby Ref. D24098W001modalities to produce rough initial draft content in a different modality, extracting semantic information to aid and inform the content creator, or combinations thereof. In some examples, aspects of these assistive methods, devices and systems can aid in the content creation workflow by providing a first pass. Further, the ability to augment existing assets via assistive methods, devices and systems can enable the rc-usc of sensory content whilst maintaining stylistic continuity.

[0335] Some disclosed assistive methods may be based, at least in part, on implementing machine learning that may involve one or more of the following:• Regression method - training directly from existing multi-sensory content;• Diffusion model based - controllable through prompts such as text, video, audio etc.:• The use of differentiable digital signal processing (DSP) - utilising DSP techniques within the machine learning framework.

[0336] Some such machine learning based implementations may involve one or more of the following:• Creating content for previously-authored audio / video media based at least in part on existing metadata (e.g. position / size of objects, audio amplitude, colour gamut); • Mapping the above types of individual metadata to aspects of a scene such as colour, intensity, temporal and spatial;• Mapping combinations of metadata from standalone audio, standalone video, or both audio and video, to aspects of the scene such as colour, intensity, temporal and spatial.

[0337] Augmentation may involve spatial adaptation, temporal adaptation and / or gamut upscaling of an existing sensory asset. For example, an existing sensory “explosion asset” may be positioned at the front left of a scene and may last for 2 seconds, but may require augmentation to front right for 5 seconds to fit in a different section of the same piece of audio, visual or audiovisual content, or an entirely new piece of audiovisual content.

[0338] Similar techniques can be used to generate variants of an asset. For example, an existing sensory asset may have been crafted to simulate a thunderstorm, with co-ordinated light flashes and haptic rumbles. Suppose the existing sensory asset is 10 seconds long, but that the current project includes an audiovisual thunderstorm scene that lasts a few minutes. In this situation it would be convenient to continuously repeat the existing sensory asset for the duration of the thunderstorm scene. However, repeating the existing sensory asset wouldDolby Ref. D24098W001be perceived as unnatural due to the obvious repetition. Some disclosed examples involve generating many variants of the 10 second long existing sensory asset and presenting a sequence of these variants instead of merely repeating the unchanged existing sensory asset, thereby reducing repetition.

[0339] Multimodal analysis

[0340] Semantic analysis can be used to detect salient events in content to reduce manual tracking / identification by a creator. For example, finding the start and end of explosions in a piece of content which could later be mapped to lighting, haptic or airflow effects. Depending on the original content, analysis could be completed using a variety of methods such as:• Music-based, which may involve the detection of beats, transients, sections in a song (e.g., chorus, verse, bridge), etc.:• Film-based, which may involve event classification based on a text prompt (e.g., a user search such as “find gunshots’’), finding a general color palette based on “key scenes” in the movie, etc.;• Gaming-based, which may involve detecting game events such as level changes, scene changes, player health changes, filtering existing game metadata (e.g., game metadata corresponding to game actors, game environment changes, etc.) to decide which game metadata instances to represent through another medium, etc.

[0341] Further analysis beyond event detection may provide more specific information to map to sensory content. For example, some implementations may involve segmenting and tracking a particular object in a scene (such as a video scene or a game scene, or even an audio scene (e.g., a bird sound)), and representing this object as a sensory object. The object in the video scene or the game scene may be classified to provide semantic information / labels for the sensory object. This classification information may be used to generate an appropriate sensory effect and / or retrieve an appropriate sensory effect.

[0342] An example of an interface including semantic analysis results is shown in Figure 1. In this case, salient events have been tagged in a timeline, aligned with existing audio and video. Further analysis could allow for prompting to search for specific events, select objects to track in a scene or colors to extract from a section.Dolby Ref. D24098W001

[0343] Figure 38 shows an example of a GUI that may be presented as part of a semantic analysis tooling process. According to this example, the elements of the GUI 3800 include the following:• Window 3801 : Video editing / preview window, which shows a frame of video content;• Window 3802: Audio editing / preview window, which shows a representation of audio corresponding to the frame of video content in window 3801 ;• Detected event timeline 3803: timeline of events detected in the audio or video content, which include semantic labels in this example. In some examples, an automated process based on machine learning may have detected the events in the detected event timeline 3803; and• Window 3804: An example of a sensory content creation tooling 3D workspace.

[0344] According to this example, line 3810 indicates the current time depicted in the windows 3801 and 3804. In this example, the window 3804 shows an example of a reference environment 3805, within which are shown a display screen 3806 (e.g., a television) and a light object 3807 that is in the process of being authored. In some examples, an automated process based on machine learning may have created proposed sensory objects, potentially including the light object 3807, that correspond with the detected events in the detected event timeline 3803. A content creator may interact with the sensory object parameter subwindow 3808 in order to “fine-tune” properties of the light object 3807 and other sensory objects.

[0345] According to some implementations, metadata can be used as a base for creating scenes when authoring sensory content via machine learning techniques, particularly metadata for existing authored audio and / or video content. Some such implementations may be configured not only to provide an initial sensory scene, but also to reinforce effects from the original existing audio and / or video content, thereby potentially increasing the speed of the content creation workflow. Individual metadata, or combinations of metadata, from audio / video content may, according to some implementations, be mapped to various sensory dimensions in a scene such as intensity, temporal and spatial.

[0346] For example, if audio is spatial and includes metadata of audio objects, the relative positions of audio objects may be used by some automated creation tools to place light objects or haptic objects in the same or similar physical positions as the audio objects. The results of this process may strengthen perceptual cues. Similarly, the amplitude of an audioDolby Ref. D24098W001signal may, in some examples, be mapped to the brightness of light objects or the intensity of haptic and airflow objects. By using existing data for content for which sensory experiences are being authored there is also greater continuity within the entire experience, particularly for persistence across modalities. Some examples include:• Automatically creating haptics events based on low-frequency audio content (rather than directly routing low-frequency effects (LFEs) to haptics actuators);• Automatically creating global brightness peaks based on transients in video brightness and / or audio level;• Automatically rendering a light object either through activating fixtures or as blurred color animation on projector: for example, a spherical light object that is red could be denoted by a projector as a red circle;• Automatic object classification on video may identify semantic information that can be used to generate or retrieve appropriate haptic effects. For example, if a video depicts a swarm of bees, a random pattern of short haptic buzzes may be automatically retrieved or generated;• An attcntional model having a video as input may automatically identify and track the most salient spatial location on the screen, which may indicate where to place light objects (e.g., above the screen) and / or where to activate haptics (e.g., on the feet or under a seat). In some implementations, the results of an attentional model’s analysis of a video may be combined with analysis of the corresponding audio, to identify the most relevant time sections.

[0347] Creating Continuity in Sensory Experiences Containing Augmented and Generated Content

[0348] It is desirable to have continuity within sensory assets and for the overall experience — including the sensory object based experience and any associated audio and / or video — to be continuous. Therefore, it is desirable for any automatically-generated sensory assets to be conditioned to match the likeness of existing assets in the experience.

[0349] Figure 39 shows example blocks that may be involved in a process of augmenting previously-created assets. In this context, “augmenting” does not necessarily mean adding features to the previously-created assets, but may instead mean adjusting one or more features, removing one or more features, etc. Figure 40 shows example blocks that may be involved in a process of using previously-created assets to automatically generate sensoryDolby Ref. D24098W001assets. In both Figure 39 and Figure 40, previously-created assets 3905 are provided to a creation tool that is implemented by an instance of the control system 110 of Figure 1. The previously-created assets 3905 may include any type of previously-created sensory content, whether created according to instructions from a human being or via artificial intelligence. In both Figure 39 and Figure 40, the creation tool includes an encoder module 3915 — which in these examples is configured to determine and output embeddings 3917 based on the previously-created assets 3905 — and a manual conditioning module 3920. However, in the example shown in Figure 39, the creation tool 3910 includes a conditional augmentation module 3925, whereas in the example shown in Figure 40, the creation tool 4010 includes a conditional generation module 4025.

[0350] In the example shown in Figure 39, the conditional augmentation module 3925 is configured to augment the previously-created assets 3905 according to manual conditioning information 3920 to obtain generated assets 3930. The manual conditioning information 3920 may be, or may include, user input provided to the creation tool 3910, e.g., from a content creator, indicating what properties are desired to be changed. For example, suppose that the previously-created assets 3905 include a plurality of yellow light objects that were created to provide a sensory experience corresponding to a swarm of bees. In some examples, the manual conditioning information 3920 may indicate how a content creator would like for the previously-created assets 3905 to be changed, while preserving at least some aspects of the original creative intent. In one such example, the manual conditioning information 3920 may indicate that content creator would like to include additional yellow light objects, in order to create sensory effects corresponding to a larger swami of bees. Alternatively, or additionally, the manual conditioning information 3920 may indicate that content creator would like to increase the speed at which the yellow light objects are moving, in order to suggest that the swarm of bees is becoming more agitated.

[0351] According to the example shown in Figure 40, the conditional generation module 4025 is configured to generate one or more new sensory assets based on the previously-created assets 3905 according to manual conditioning information 3920 and generation prompt information 4007. The generation prompt information 4007 may be, or may include, user input provided to the creation tool 3910, e.g., from a content creator, indicating one or more characteristics of new sensory content the content creator would like the creation tool 4010 to automatically generate and output as the generated assets 4030. For example, suppose that the previously-created assets 3905 include the above-described plurality ofDolby Ref. D24098W001yellow light objects that are meant to correspond to a swarm of bees. In some examples, the generation prompt information 4007 may indicate that the content creator would like for the creation tool 4010 to generate one or more types of new sensory content, such as haptic content, based on properties of the previously-created assets 3905. In one such example, the generated assets 4030 would include newly-generated haptic content created based on the previously-created assets 3905 and responsive to the generation prompt information 4007. According to some examples, the newly-generated haptic content will have one or more features that correspond with features of the previously-created assets 3905 and / or with associated metadata, such as a distribution of localized vibrations across a haptic mesh, a vibration frequency that is consistent that of a buzzing bee, a vibration intensity that increases as a light approaches an assumed user position, etc.

[0352] Creator Autocomplete

[0353] According to some disclosed implementations, content creator choices and decisions are used as the bases for further prompts to “autocomplete” a scene by a creation tool implementing one or more of the disclosed machine learning methods. Such autocompletion may be done both at the event level, and across sections of sensory content that include multiple events, e.g., as follows:• Auto-completion at the event-level may involve proposing the remaining parameters of one or more sensory objects corresponding to an individual event as sensory content is being created for that event and implementing these proposed parameters, or modified versions of these parameters, according to input from the content creator.• Auto-generation of event sequences across sections of content may involve proposing subsequent similar events based on one or several existing events and implementing these proposed subsequent similar events, or modified versions of these proposed subsequent similar events, according to input from the content creator.

[0354] For example, a content creator may have chosen sensory asset A to represent a gunshot in the first scene of a movie. In some examples, sensory asset A may be automatically re-used for all gunshots of the movie. According to some examples, event characteristics may be derived from a combination of content creator choices and analyzed content. In some examples, a content creation tool may impart a level of randomization in order to increase variability and create less repetitive sensory effects.Dolby Ref. D24098W001

[0355] Figure 41 shows an example of auto-completion at the event level based on existing video content. According to this example, the GUI elements 4100 include light content GUI elements 4110 and video content GUI elements 4120.

[0356] In this example, the video content GUI elements 4120 include window 4125 and window 4130. In some examples, windows 4125 and 4130 may be presented at the same time, whereas in alternative examples windows 4125 and 4130 may be presented at different times. According to this example, the windows 4125 and 4130 show a character in the video that is causing a light beam 4127 to be emitted. The window 4125 shows the light beam 4127 at an earlier time and the window 4130 shows the light beam 4127 at a later time. A corresponding portion of the light beam is indicated in a circle 4128 in both of the windows 4125 and 4130: the changing position of the circle 4128 is intended to indicate the forward progression of the light beam 4127.

[0357] According to this example, the light content GUI elements 4110 include window 4112 and window 4114. The window 4112 indicates properties of light content that has already been designed and authored by a content creator, whereas the window 4114 indicates properties of light content that is currently being proposed for auto-completion by a disclosed sensory content authoring tool. In this example, the light content GUI elements 4110 show brightness, color and positional properties of a lighting effect. This lighting effect is intended to correspond to the light beam 4127 in the corresponding video. One may observe that the sensory content authoring tool is proposing to keep the brightness and the color the same, or substantially the same, but to increase the y coordinate in a linear fashion. The change in the y coordinate over time is consistent with the changing position of the light beam 4127, as indicated by the changing position of the circle 4128 in the video content GUI elements 4120.

[0358] According to some examples, the GUI elements 4100 may include one or more virtual buttons or similar virtual elements with which the content creator may interact in order to accept, reject or modify the proposed auto-completion parameters.

[0359] Figure 42 shows an example of auto-completion at the event level based on existing audio content. According to this example, the GUI elements 4200 include light content GUI elements 4210 and audio content GUI elements 4220.

[0360] In this example, the audio content GUI elements 4220 include window 4225, which shows a waveform representation 4226 of audio data and an envelope 4227 over a selectedDolby Ref. D24098W001portion of the waveform representation 4226. According to this example, the selected portion of the waveform representation 4226 corresponds with the first portion of a segment of audio data for which a content creator has already begun to create a lighting effect. In some alternative examples, the envelope 4227 may be formed over the waveform representation 4226 of a portion of audio data that a machine learning aspect of the creation tool has estimated to be a salient feature.

[0361] According to this example, the light content GUI elements 4210 include window 4212 and window 4214. The window 4212 indicates properties of light content that has already been designed and authored by a content creator, whereas the window 4214 indicates properties of light content that is currently being proposed for auto-completion by a disclosed sensory content authoring tool. In this example, the light content GUI elements 4210 show brightness and color lighting effects. The authored lighting effect that is shown in the window 4214 involves a red color that is modulated according to the amplitude of the waveform representation 4226 in the corresponding audio content. One may observe that the sensory content authoring tool is proposing to the color the same and to continue modulating the brightness according to the amplitude of envelope 4227 in the selected portion of the waveform representation 4226.

[0362] According to some examples, the GUI elements 4200 may include one or more virtual buttons or similar virtual elements with which the content creator may interact in order to accept, reject or modify the proposed auto-completion parameters.

[0363] Various features and aspects will be appreciated from the following enumerated example embodiments (“EEEs”):

[0364] EEE1 A. A method for dynamically adjusting a multisensory content, the method comprising: receiving input from a user device, wherein the input corresponds to an adjustment of a perceived immersion of the multisensory content.

[0365] EEE2A. The method of claim EEE1 A, wherein the multisensory content is flexibly rendered.

[0366] EEE3A. The method of claim EEE1A or claim EEE2A, wherein the adjustment of the perceived immersion of the multisensory content corresponds to a global adjustment of the multisensory content.Dolby Ref. D24098W001

[0367] EEE4A. The method of claim EEE1A or claim EEE2A, wherein the adjustment of the perceived immersion of the multisensory content corresponds to an adjustment of layers within the multisensory content.

[0368] EEE5A. The method of any one of claims EEE1A-EEE4A, wherein the multisensory content comprises static content or interactive content.

[0369] EEE6A. The method of any one of claims EEE1A-EEE5A, wherein the perceived immersion comprises at least one of a spatial resolution, a photosensitivity, a brightness, an intensity of vibration, a rate of change of vibration, a fixture limit, an airflow intensity, and a temperature of the multisensory content.

[0370] EEE7A. The method of any one of claims EEE1A-EEE6A, wherein the adjustment of the perceived immersion is limited to a range of adjustments.

[0371] EEE8A. The method of any one of claims EEE1 A-EEE7A, wherein the adjustment of the perceived immersion comprises at least one of dynamic range compression, tone mapping, time- frequency low pass filtering of luminaire brightness, and space-frequency low pass filtering for perceived object movement around a space.

[0372] EEE9A. The method of any one of claims EEE1 A-EEE8A, wherein the adjustment of the perceived immersion comprises an augmentation of an object of the multisensory content.

[0373] EEE10A. The method of any one of claims EEE1A-EEE9A, wherein the adjustment of the perceived immersion comprises processing light activations of the multisensory content.

[0374] EEE11A. The method of any one of claims EEE1A-EEE10A, wherein the adjustment of the perceived immersion comprises augmenting a color volume mapping scheme of the multisensory content.

[0375] EEE 12 A. The method of any one of claims EEE1A-EEE11 A, wherein receiving the input from the user device comprises receiving input from a knob of the user device.

[0376] EEE1B. A method for visualizing and augmenting sensory content, the method comprising: displaying, via a 3D visualizer implemented by a control system, a representation of an endpoint and sensory content projected into the endpoint, wherein the sensory content comprises one or more sensory objects of a parametric or non-parametric shape; and augmenting the displayed sensory content, according to augmentation user input received by the control system via an interface system, using a volumetric field or a volumetric texture.Dolby Ref. D24098W001

[0377] EEE2B. The method of claim EEE1B, wherein the sensory content comprises a first and a second sensory asset, wherein the first and second sensory assets are nested sensory assets.

[0378] EEE3B. The method of claim EEE1B or claim EEE2B, wherein the sensory content comprises image, video, or lightscapc content.

[0379] EEE4B. The method of any one of claims EEE1B-EEE3B, wherein the sensory content is configured for bed-based rendering.

[0380] EEE5B. The method of any one of claims EEE1B-EEE4B, further comprising creating, by the control system, at least one sensory object according to creation user input received by the control system via an interface system.

[0381] EEE1C. A method of generating editable multi-sensory assets using assistive modules, the method comprising: generating an editable multi-sensory asset via an assistive module, wherein the assistive module comprises at least one of a trained neural network and a classical technique.

[0382] EEE2C. The method of claim EEE1C, wherein the editable multi-sensory asset comprises a multi-sensory object and metadata.

[0383] EEE3C. The method of claim EEE1C or claim EEE2C, wherein, when the assistive module comprises the trained neural network, the neural network comprises at least one of a multi-modal model, a regression model, a diffusion model, and a differentiable digital signal processing-based model.

[0384] EEE4C. The method of claim EEE3C, when the neural network comprises a multimodal model trained to generate information related to the editable multi-sensory asset, the information comprising at least one of event or time markers, content, and semantic information.

[0385] EEE5C. The method of claim EEE4C, wherein the multi-modal model is configured to take, as input, audio data, video data, and / or text data.

[0386] EEE6C. The method of claim EEE4C, wherein the multi-modal model performs cross-modal analysis to generate the information related to the editable multi-sensory asset.

[0387] EEE7C. The method of any one of claims EEE1C-EEE6C, further comprising receiving, as input, content for controlling the generating of the editable multi-sensory asset.

[0388] EEE8C. The method of claim EEE7C, wherein the content comprises long and / or short time experiential congruency.Dolby Ref. D24098W001

[0389] EEE9C. The method of any one of claims EEE1C-EEE8C, further comprising receiving, as input, a creator choice for controlling the generating of the editable multi-sensory asset.

[0390] EEE1D. A method for interchanging and packaging sensory object-based assets, the method comprising: receiving, by a control system, user input regarding the creation of a sensory object-based asset comprising one or more sensory objects, corresponding asset metadata, and an asset timeline, wherein the asset timeline comprises information corresponding to a start time and duration for each sensory object of the one or more sensory objects; and creating, by the control system, the sensory object-based asset according to the user input.

[0391] EEE2D. The method of EEE1D, wherein the user input is received via a graphical user interface of a sensory asset creation tool implemented by the control system.

[0392] EEE3D. The method of EEE1D or EEE2D, wherein the asset metadata comprises a control interface mapped to sensory object properties of the one or more sensory objects.

[0393] EEE4D. The method of any one of EEEs 1D-3D, wherein the sensory object-based asset comprises one or more nested sensory object-based assets, wherein the asset timeline corresponds to the one or more sensory object-based assets.

[0394] EEE5D. A method for unwrapping a nested sensory object-based asset into a single timeline, the method comprising: receiving, by a control system, a first sensory object-based asset comprising a second sensory object-based asset, wherein the second sensory object-based asset is nested within the first sensory object-based asset, and wherein each of the first sensory object-based asset and the second sensory object-based asset comprise one or more sensory objects, corresponding asset metadata, and a sensory object timeline; combining, by the control system, a first sensory object timeline of the first sensory object-based asset and a second sensory object timeline of the second sensory object-based asset to create a combined sensory object timeline comprising the one or more sensory objects of both the first and second sensory object-based assets; and recalculating, by the control system, timing metadata for each of the one or more sensory objects of the first sensory object-based asset or the second sensory objectbased asset in reference to the combined sensory object timeline.

[0395] EEE6D. An apparatus configured to implement the method of any one of EEEs 1D-5D.

[0396] EEE7D. A system configured to implement the method of any one of EEEs 1D-5D.

[0397] EEE8D. One or more non-transitory media having instructions encoded thereon for controlling one or more devices to implement the method of any one of EEEs 1D-5D.Dolby Ref. D24098W001

[0398] The above description illustrates various embodiments of the present disclosure along with examples of how aspects of the present disclosure may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of the present disclosure as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents will be evident to those skilled in the art and may be employed without departing from the spirit and scope of the disclosure as defined by the claims.

Claims

1. Dolby Ref. D24098W0012.CLAIMS3.What Is Claimed Is:

1. A method for interchanging and packaging sensory object-based assets, the method comprising:5.receiving, by a control system, user input regarding the creation of a sensory objectbased asset comprising one or more sensory objects, corresponding asset metadata, and an asset timeline, wherein the asset timeline comprises information corresponding to a start time and duration for each sensory object of the one or more sensory objects; and6.creating, by the control system, the sensory object-based asset according to the user input.

2. The method of claim 1 , wherein the user input is received via a graphical user interface of a sensory asset creation tool implemented by the control system.

3. The method of claim 1 or claim 2, wherein the asset metadata comprises a control interface mapped to sensory object properties of the one or more sensory objects.

4. The method of any one of claims 1-3, wherein the sensory object-based asset comprises one or more nested sensory object-based assets, wherein the asset timeline corresponds to the one or more sensory object-based assets.

5. A method for unwrapping a nested sensory object-based asset into a single timeline, the method comprising:11.receiving, by a control system, a first sensory object-based asset comprising a second sensory object-based asset, wherein the second sensory object -based asset is nested within the first sensory object-based asset, and wherein each of the first sensory object-based asset and the second sensory object-based asset comprise one or more sensory objects, corresponding asset metadata, and a sensory object timeline;12.combining, by the control system, a first sensory object timeline of the first sensory object-based asset and a second sensory object timeline of the second sensory object-based asset to create a combined sensory object timeline comprising the one or more sensory objects of both the first and second sensory object-based assets; and Dolby Ref. D24098W00113.recalculating, by the control system, timing metadata for each of the one or more sensory objects of the first sensory object-based asset or the second sensory object-based asset in reference to the combined sensory object timeline.

6. An apparatus configured to implement the method of any one of claims 1-5.

7. A system configured to implement the method of any one of claims 1-5.

8. One or more non-transitory media having instructions encoded thereon for controlling one or more devices to implement the method of any one of claims 1-5.

Images